Bayesian Optimization

Bayesian Optimization is a sequential strategy for finding the global minimum of a black-box objective function, especially when function evaluations are expensive. It matters when tuning hyperparameters of machine learning models, optimizing experimental procedures, or configuring complex systems where each evaluation takes minutes to hours.

Selecting the Hyperparameters of a Machine Learning Model

When training a model with a large number of hyperparameters—such as learning rate, batch size, number of layers, dropout rate, and optimizer type—each configuration requires a full training run. Bayesian Optimization automates this process by selecting the most promising configurations to evaluate next, based on prior belief about the objective function and evidence from past evaluations.

The core of the method is a probabilistic surrogate model—typically a Gaussian Process—that encodes belief about the objective function. At each iteration, it computes an acquisition function to balance exploration (trying new regions) and exploitation (refining known good regions), and selects the hyperparameter configuration that maximizes the acquisition score.

To run a Bayesian Optimization loop, you define:

• A domain: a set of hyperparameters with their possible values,
• An objective function: a callable that trains and evaluates a model on a given configuration,
• An acquisition function: a rule for deciding the next point to evaluate.

Example: Optimizing a Neural Network with Scikit-learn and GPy

import numpy as np
from sklearn.model_selection import cross_val_score
from sklearn.neural_network import MLPClassifier
from gpyt import GaussianProcessRegressor
from gpyt.acquisition import ExpectedImprovement


param_space = {
    'learning_rate_init': (1e-4, 1e-1),       # continuous
    'batch_size': (16, 128),                   # discrete
    'hidden_layer_sizes': [(32,), (64,), (128,), (64, 32)],  # categorical
    'alpha': (1e-6, 1e-1),                     # continuous
    'max_iter': (50, 500),                     # continuous
}

# Objective function: returns validation score
def objective(params):
    model = MLPClassifier(
        learning_rate_init=params['learning_rate_init'],
        batch_size=params['batch_size'],
        hidden_layer_sizes=params['hidden_layer_sizes'],
        alpha=params['alpha'],
        max_iter=params['max_iter'],
        random_state=42
    )
    scores = cross_val_score(model, X_train, y_train, cv=5, scoring='accuracy')
    return -np.mean(scores)  # minimize negative accuracy

# Initialize Bayesian Optimization loop
gp = GaussianProcessRegressor(kernel='rbf', alpha=1e-3)
acq_func = ExpectedImprovement(gp)

# Start with a few random configurations
initial_configs = [
    {'learning_rate_init': 0.01, 'batch_size': 32, 'hidden_layer_sizes': (64,), 'alpha': 0.01, 'max_iter': 200},
    {'learning_rate_init': 0.001, 'batch_size': 64, 'hidden_layer_sizes': (128,), 'alpha': 0.001, 'max_iter': 300},
    {'learning_rate_init': 0.005, 'batch_size': 16, 'hidden_layer_sizes': (32,), 'alpha': 0.1, 'max_iter': 100},
]

# Evaluate initial points
results = []
for config in initial_configs:
    loss = objective(config)
    results.append((config, loss))

# Iteratively suggest and evaluate new configurations
for _ in range(20):
    # Fit GP to current results
    X = np.array([list(r[0].values()) for r in results])
    y = np.array([r[1] for r in results])
    gp.fit(X, y)

    # Suggest next point
    next_config = acq_func.suggest(X, y, param_space)
    next_loss = objective(next_config)
    results.append((next_config, next_loss))

The param_space defines the search domain. Continuous parameters are specified as intervals, discrete ones as lists, and categorical ones as choices. The objective function must accept a dictionary of hyperparameters and return a scalar loss.

Common Pitfalls

• Misaligned parameter types: A batch_size parameter defined as a continuous interval [16, 128] is treated as real-valued, but the model expects integers. Without rounding, the GP suggests batch_size=64.7, which fails at model training. The solution: define batch_size as a discrete range and round during evaluation.
• Improper scaling: When parameters span different scales—e.g., learning_rate_init in [1e-4, 1e-1] and alpha in [1e-6, 1e-1]—the Gaussian Process assumes uniform covariance across all dimensions. Without feature scaling (e.g., standardization), the model overweights the larger-scale parameters. Always normalize the input space.
• Acquisition function confusion: Expected Improvement (EI) is the default, but it often under-explores when the objective is noisy. Lowering the xi (exploration rate) in EI can help, but too low leads to over-exploration. The kappa parameter in Upper Confidence Bound (UCB) is often misinterpreted: a high kappa increases exploration, but it must be tuned with the kernel’s length scale.

Failure Modes in Practice

• Cold start: The initial configurations are poor because they are chosen uniformly at random, but the surrogate model is undertrained. The first few evaluations are noisy, and the GP overfits to early data. The solution: use Latin Hypercube Sampling instead of random sampling for initial points.
• Overfitting the surrogate: With many iterations, the GP becomes overconfident in regions with many evaluations. But if the true objective function changes (e.g., due to data drift), the model underperforms in new configurations. The fix: periodically retrain the GP with all data, or use a moving window of recent evaluations.
• Stale acquisition: The acquisition function is computed once per iteration, but the next configuration is evaluated in parallel. During evaluation, the surrogate model may have been updated, leading to a mismatch between suggestion and actual evaluation. The solution: use a queue of suggested configurations and evaluate them in order, updating the model after each completion.

Optimizing Experimental Procedures

In physical or biological experiments—e.g., optimizing the growth conditions of a cell culture or the settings of a chemical reactor—Bayesian Optimization automates the process of adjusting multiple control parameters to maximize yield, purity, or throughput.

The objective is the outcome of a full experiment, which may take hours or days. Each configuration is a vector of parameters: temperature, pressure, flow rate, nutrient concentration, and duration.

Example: Optimizing a Protein Expression Experiment

The experiment is controlled by a lab automation platform that accepts a JSON configuration and returns a result object.

{
  "temperature": 37.5,
  "pressure": 1.2,
  "flow_rate": 0.8,
  "nutrient_concentration": 5.0,
  "duration": 24.0
}

The platform exposes an HTTP API:

curl -X POST https://lab-api.example.com/experiments \
  -H 'Content-Type: application/json' \
  -d @experiment.json \
  -o experiment-result.json

The response includes:

{
  "experiment_id": "exp-123",
  "status": "completed",
  "yield": 83.2,
  "purity": 0.91,
  "timestamp": "2024-04-05T12:00:00Z"
}

The objective function extracts the yield or a composite score.

import requests
import json

def run_experiment(config):
    url = "https://lab-api.example.com/experiments"
    response = requests.post(url, json=config, timeout=3600)
    result = response.json()
    return -result["yield"]  # minimize negative yield

The Bayesian Optimization loop runs as before, using a GP surrogate and Expected Improvement acquisition.

Sharp Edges

• Parameter constraints: Not all parameter combinations are valid. For example, pressure cannot exceed 2.0 when temperature is above 40°C. These constraints must be encoded in the search space. A configuration that violates constraints is rejected, and the acquisition function must account for feasibility.
• Noisy observations: Experimental results vary due to batch effects, equipment drift, or human error. The GP can model this by adding a noise term to the kernel. The noise hyperparameter in the GP must be estimated from the data, and the acquisition function should account for the variance in predictions.
• Batch evaluations: Multiple experiments can run in parallel. The acquisition function can be adapted to suggest a batch of configurations at once. The Expected Improvement for Batch Acquisition selects a set of configurations that jointly improve the expected objective, accounting for correlation between experiments.

Configuration of Complex Systems

For systems with many tunable parameters—e.g., a web service with caching, load balancing, database indexing, and API routing—Bayesian Optimization can find the configuration that minimizes latency, maximizes throughput, or reduces cost.

The objective is measured by running a stress test or by monitoring a production deployment over time.

Example: Optimizing a Web Service with a Custom Load Test

A system is deployed with a set of configuration keys:

# config.yaml
server:
  max_connections: 1024
  worker_threads: 8
  cache_ttl: 300
  buffer_size: 1024
  compression: true
  load_balancer:
    strategy: "round_robin"
    health_check_interval: 60
    timeout: 120

A load test is run via a command-line tool:

./run-stress-test \
  --config config.yaml \
  --duration 300 \
  --concurrency 100 \
  --output results.json

The output includes:

{
  "avg_latency_ms": 124.3,
  "throughput_rps": 450.1,
  "error_rate": 0.023,
  "total_cost": 142.8
}

The objective function computes a composite score:

def objective(config):
    # Write config to file
    with open("config.yaml", "w") as f:
        yaml.dump(config, f)

    # Run stress test
    subprocess.run([
        "./run-stress-test",
        "--config", "config.yaml",
        "--duration", "300",
        "--concurrency", "100",
        "--output", "results.json"
    ], check=True)

    # Read results
    with open("results.json") as f:
        results = json.load(f)

    # Compute objective: minimize latency + 2 * error rate + 0.1 * cost
    score = (
        results["avg_latency_ms"]
        + 2 * results["error_rate"]
        + 0.1 * results["total_cost"]
    )
    return score

The hyperparameter space includes:

• Continuous: server.max_connections, server.buffer_size, server.load_balancer.timeout
• Discrete: server.worker_threads in [4, 8, 16, 32]
• Categorical: server.load_balancer.strategy in ["round_robin", "least_connections", "weighted_round_robin"]
• Boolean: server.compression

The optimization loop selects configurations to evaluate, based on the GP and acquisition function.

Critical Considerations

• Cold start with default configuration: The initial evaluation often uses a default config, but the GP may suggest configurations that are hard to deploy. The solution: define a deployment pipeline that applies a config and waits for stability before measuring the objective.
• Configuration drift: The system evolves between evaluations. The optimal configuration at time t may degrade by time t+1. The GP must model this temporal drift, or the objective function must include a time-weighted average.
• Acquisition function choice: When the cost of evaluation is high, the acquisition function must account for the trade-off between the expected gain and the cost of evaluation. The Cost-Aware Acquisition Function can be defined as acquisition = expected_improvement / (1 + cost_per_evaluation).

Summary

Bayesian Optimization is not a one-size-fits-all tool. It requires careful design of the objective function, precise definition of the hyperparameter space, and attention to the details of the acquisition and surrogate models. Its power lies in its ability to learn from each evaluation and guide future choices, but its success depends on avoiding common pitfalls: misaligned parameter types, poor scaling, noisy observations, and stale acquisition. With attention to these details, it becomes a robust engine for automating the tuning of complex systems.

This page was rewritten on 17 October 2026. It replaced a templated version that was not specific to its own title. How these pages are written and checked. If anything here is wrong, tell us at [email protected] and we will correct it.