Combined Algorithm Selection and Hyperparameter Optimization (CASH)

The CASH problem is defined by a composite search space that is the Cartesian product of a discrete set of machine learning algorithms and their respective, often continuous, hyperparameter ranges. This creates a vast, heterogeneous, and often hierarchical configuration space. For example, the hyperparameters for a Support Vector Machine (kernel type, C, gamma) are entirely different from those for a Random Forest (number of trees, max depth). The search algorithm must navigate this complex space efficiently.

Hyperparameters are often conditionally dependent on the algorithm choice. This creates a hierarchical structure where some parameters only become relevant (or active) when a specific algorithm or component is selected.

• If algorithm = SVM, then parameters kernel_type and C are active.
• If kernel_type = 'rbf', then parameter gamma becomes active.
• If algorithm = DecisionTree, then SVM parameters are irrelevant. This conditional structure invalidates simple grid or random search across the flattened space and requires search methods that understand these dependencies.

Evaluating a single (algorithm, hyperparameter) configuration is a black-box optimization problem. The function mapping a configuration to a performance metric (e.g., validation accuracy) is:

• Expensive to evaluate: Requires full model training and validation.
• Noisy: Performance can vary due to random seeds, data shuffling, or hardware non-determinism.
• Has no tractable gradient: The relationship between hyperparameters and final loss is not directly differentiable in a standard way. This necessitates the use of sample-efficient optimization strategies like Bayesian optimization or multi-fidelity methods (e.g., Hyperband) to minimize total computational cost.

While often framed as maximizing a single performance metric (e.g., accuracy), the true CASH problem is inherently multi-objective. Practical solutions must balance:

• Predictive Performance (Accuracy, F1-Score, AUC)
• Computational Cost (Training time, inference latency)
• Model Complexity (Memory footprint, model size)
• Robustness & Fairness The optimal solution is often not a single point but a Pareto front of configurations representing the best possible trade-offs between these competing objectives.

CASH is formally defined as selecting a learning algorithm A from a set A and its hyperparameters λ from domain Λ_A that minimize a loss function L evaluated via cross-validation on a dataset D. The objective is: A*, λ* = argmin_{A in A, λ in Λ_A} E_{(D_train, D_val)} [ L( A_λ, D_train, D_val ) ] This formulation reduces the traditionally manual, multi-step ML pipeline (choose algorithm, then tune it) into a single, automated, stochastic optimization problem. Popular solvers include Auto-Sklearn (using Bayesian optimization with meta-learning) and TPOT (using genetic programming).

CASH is a specific, foundational problem within AutoML that differs from related concepts:

• Hyperparameter Optimization (HPO): Assumes the algorithm is fixed; CASH is a generalization that includes algorithm choice.
• Neural Architecture Search (NAS): A specialized sub-problem of CASH where the algorithm space A consists exclusively of neural network architectures and their hyperparameters.
• Full AutoML Pipelines: CASH is typically the core component, but full pipelines also automate upstream steps like feature engineering and data preprocessing, which can be integrated as additional hyperparameters in an extended CASH formulation.

CASH is extended to multi-objective optimization when deploying models to resource-constrained devices. The goal isn't just accuracy, but finding the best trade-off on the Pareto front between performance and metrics like:

• Inference Latency (< 100 ms)
• Model Size (< 10 MB)
• Energy Consumption Frameworks like Google's Model Search or Hardware-Aware NAS methods treat the choice of model architecture (e.g., MobileNetV3, EfficientNet-Lite) and its hyperparameters (width multiplier, resolution) as a CASH problem, directly searching for configurations that meet specific deployment service level agreements (SLAs) on target hardware.

While full retraining of LLMs is prohibitively expensive, CASH is crucial for efficient fine-tuning. The problem involves selecting:

• The fine-tuning algorithm (e.g., full fine-tuning, LoRA, QLoRA, prefix tuning).
• Its hyperparameters (e.g., LoRA rank r, alpha, dropout).
• Optimizer-specific settings (learning rate, scheduler). Platforms like Weights & Biases Sweeps or Optuna are used to orchestrate large-scale CASH runs across hundreds of GPU trials to find the optimal adaptation recipe for a specific downstream task (e.g., legal document analysis) within a fixed compute budget, maximizing performance gains per GPU hour.

Time series forecasting introduces unique challenges, requiring the joint selection of:

• Forecasting model (e.g., Prophet, ARIMA, DeepAR, N-BEATS).
• Preprocessing steps (differencing, outlier handling, calendar feature engineering).
• Model-specific hyperparameters (seasonality periods, trend flexibility). Libraries like AutoGluon-Timeseries or PyCaret frame this as a CASH problem. The optimizer evaluates thousands of pipeline combinations, often using multi-fidelity optimization (e.g., evaluating on shorter series first) to quickly identify the best method for a given series' characteristics (seasonality, noise, trend), automating a highly specialized and manual domain.

The automated search for the optimal configuration of a model's hyperparameters—settings that govern the training process itself, such as learning rate, batch size, or regularization strength. HPO is a core sub-problem within CASH, focused on tuning a single selected algorithm.

• Methods: Grid search, random search, Bayesian optimization.
• Goal: Maximize a performance metric (e.g., validation accuracy) by finding the best point in a continuous or discrete hyperparameter space.

The automated design of neural network architectures. NAS searches over operations (e.g., convolution type, activation functions) and their connectivity to discover high-performing structures for a given dataset and task. It can be viewed as a specialized form of algorithm selection within CASH, where the "algorithm" is the network topology.

• Search Strategies: Reinforcement learning, evolutionary algorithms, differentiable search (DARTS).
• Key Challenge: Balancing search efficacy with the enormous computational cost of training each candidate architecture.

A sequential model-based optimization (SMBO) strategy for globally optimizing expensive black-box functions, like a model's validation loss. It is the dominant methodology for solving HPO and, by extension, CASH problems.

• Mechanism: Builds a probabilistic surrogate model (e.g., Gaussian Process) to approximate the objective function. An acquisition function (e.g., Expected Improvement) uses this model to decide the most promising hyperparameter configuration to evaluate next.
• Advantage: Extremely sample-efficient, making it ideal when each model training run is costly.

The paradigm of 'learning to learn'. A meta-learner is trained on a distribution of related tasks to acquire knowledge that enables rapid adaptation to new tasks with minimal data. In AutoML, meta-learning can warm-start CASH by suggesting promising algorithms/hyperparameters based on dataset meta-features.

• Example: Model-Agnostic Meta-Learning (MAML) learns a model initialization that allows fast fine-tuning.
• AutoML Application: Using prior experiment histories (meta-data) to build recommendation systems for algorithm selection.

An optimization strategy that uses cheaper, lower-quality approximations to guide the search for optimal high-fidelity configurations. This is critical for making CASH tractable, as evaluating a full model on the entire dataset is expensive.

• Low-Fidelity Proxies: Training on a data subset, for fewer epochs, or at lower resolution.
• Algorithms: Hyperband is a prominent example, using successive halving to dynamically allocate resources to the most promising configurations.

The formal definition of the set of all possible configurations an AutoML system can explore. For CASH, this is the Cartesian product of the algorithm portfolio and each algorithm's hyperparameter space. The design of this space is a major determinant of search efficiency and final performance.

• Components: Includes choice of algorithms, hyperparameter types (continuous, categorical), and their ranges or values.
• Constraint: Poorly designed spaces (too large, uninformative) can make the CASH problem intractable, regardless of the optimization method used.