Search Space

Continuous parameters are numerical hyperparameters that can take any real value within a specified interval. They are defined by a lower bound, an upper bound, and often a prior distribution (e.g., uniform, log-uniform).

• Examples: Learning rate, weight decay coefficient, dropout rate.
• Optimization Challenge: The search space is infinite and uncountable, requiring algorithms like Bayesian Optimization that can model smooth relationships between the parameter value and the objective function.
• Key Consideration: The scale matters. A learning rate is often searched in log-space (e.g., from 1e-5 to 1e-1) to give equal consideration to orders of magnitude.

Discrete parameters are hyperparameters that can only take integer values within a defined range. They represent countable quantities.

• Examples: Number of layers in a neural network, batch size, number of epochs, k in a k-nearest neighbors algorithm.
• Optimization Nuance: While the set of values is finite, the parameter often controls a structural property of the model. Some optimization frameworks treat them as ordered categorical variables.
• Practical Note: Batch size is often constrained by GPU memory, making its feasible range a discrete set of powers of two (e.g., 32, 64, 128, 256).

Categorical parameters are hyperparameters that can take one value from a finite set of unordered options. There is no intrinsic numerical relationship between the choices.

• Examples: Optimizer type (e.g., Adam, SGD, RMSprop), activation function (e.g., ReLU, GELU, Sigmoid), model architecture variant (e.g., ResNet50, EfficientNetB0).
• Encoding Requirement: Optimization algorithms require these to be encoded, typically via one-hot encoding, so the surrogate model can reason about them.
• Search Implication: The performance landscape across categories can be non-smooth, making this a challenging dimension for sequential model-based optimization.

Conditional parameters are hyperparameters whose existence or valid range depends on the value of another 'parent' hyperparameter. They create a hierarchical search space.

• Example: The learning_rate for a specific optimizer is only relevant if that optimizer is chosen. The n_estimators parameter for a Random Forest is only active if the model type is set to RandomForest.
• Framework Support: Advanced tuning libraries like Optuna and Ray Tune provide APIs (e.g., trial.suggest_categorical) to define these dependencies, allowing the search algorithm to efficiently prune invalid branches.
• Complexity: Conditional spaces significantly increase the complexity of the optimization problem, as the effective dimensionality changes across trials.

For continuous parameters like learning rates or regularization strengths that span orders of magnitude, a log-uniform or log-normal distribution is the appropriate prior. This ensures samples are drawn evenly across the logarithmic scale.

• Mechanism: Instead of sampling x uniformly between low and high, the algorithm samples log(x) uniformly. For a range of [1e-5, 1e-1], a log-uniform sampler is as likely to pick a value between 1e-5 and 1e-4 as it is to pick one between 1e-2 and 1e-1.
• Why it's Critical: Hyperparameter sensitivity is often multiplicative, not additive. A grid search over a linear scale would waste most trials on overly large values.
• Implementation: In Optuna, this is trial.suggest_float('lr', 1e-5, 1e-1, log=True). In a search space definition, it's specified as {'distribution': 'LogUniform', 'lower': 1e-5, 'upper': 1e-1}.

Search spaces are programmatically defined using the APIs of tuning frameworks. Here are canonical examples:

• Optuna (Define-by-Run):

  pythonCopy
  def objective(trial):
      lr = trial.suggest_float('lr', 1e-5, 1e-1, log=True)
      n_layers = trial.suggest_int('n_layers', 1, 5)
      optimizer = trial.suggest_categorical('optimizer', ['Adam', 'SGD'])

• Ray Tune (Declarative):

  pythonCopy
  config = {
      'lr': tune.loguniform(1e-5, 1e-1),
      'batch_size': tune.choice([32, 64, 128]),
      'optimizer': tune.choice(['Adam', 'SGD'])
  }

• Key Difference: Define-by-run allows dynamic, conditional spaces. Declarative configs are static but easier to serialize.

The overarching process of finding the optimal configuration for a machine learning model by systematically exploring a search space. It involves:

• Defining the parameters to optimize and their ranges.
• Selecting a search strategy (e.g., grid, random, Bayesian).
• Running training trials and evaluating performance on a validation set. The goal is to maximize an objective function, such as validation accuracy or minimize loss.

An exhaustive search strategy that evaluates every possible combination of hyperparameters within a predefined, discrete search space. For example, for learning rates [0.01, 0.001] and batch sizes [32, 64], it runs 4 trials: (0.01, 32), (0.01, 64), (0.001, 32), (0.001, 64).

• Guarantees finding the best combination within the defined grid.
• Becomes computationally intractable as the number of parameters or their possible values grows (the "curse of dimensionality").

A search strategy that randomly samples hyperparameter combinations from defined distributions (e.g., uniform, log-uniform) over the search space.

• Often more efficient than grid search, especially in high-dimensional spaces, as it doesn't waste resources on evaluating every granular combination.
• Proven to find good configurations with fewer trials because it has a better chance of hitting important parameter regions.

A sequential model-based optimization (SMBO) strategy for hyperparameter tuning. It builds a probabilistic surrogate model (often a Gaussian Process) to predict model performance across the search space.

• Balances exploration (trying uncertain areas) and exploitation (refining known good areas).
• Highly sample-efficient, making it ideal for expensive-to-evaluate functions, like training large neural networks.
• Frameworks like Optuna and Ray Tune implement advanced Bayesian methods.

The automated execution of multiple training runs (trials), each with a different hyperparameter configuration drawn from the search space. A sweep is defined by:

• The search space specification.
• The search algorithm (e.g., random, Bayesian).
• A budget (number of trials or total time).
• An objective metric to optimize. Platforms like Weights & Biases and MLflow provide tools to launch, manage, and visualize sweeps.

The specific, quantitative metric that a hyperparameter tuning process aims to optimize. It formally defines the goal of searching the space.

• Also called the target metric or loss function.
• Common examples: Validation Accuracy (maximize), Log Loss (minimize), F1 Score (maximize).
• The tuning algorithm's purpose is to find the hyperparameter set that yields the best (max or min) value for this function.