Automated Machine Learning (AutoML)

These are code-first libraries that provide AutoML capabilities within a developer's existing Python or R environment. They automate feature engineering, model selection, and hyperparameter tuning (HPO).

• H2O AutoML: Provides an automated interface for training and tuning a wide selection of models, including stacked ensembles, with strong scalability.
• AutoKeras: Specializes in neural architecture search (NAS) for deep learning models, built on top of Keras/TensorFlow.
• AutoGluon: Focuses on ease of use and strong out-of-the-box performance for tabular, text, and image tasks with minimal code.

Fully managed, end-to-end platforms that abstract infrastructure management. They provide automated data preprocessing, built-in NAS and HPO, and one-click deployment.

• Google Cloud Vertex AI: Offers a unified platform with AutoML for vision, video, tabular data, and documents, alongside custom training and pipeline tools.
• Azure Machine Learning: Features automated ML for classification, regression, and forecasting, with robust MLOps capabilities for model lifecycle management.
• Amazon SageMaker Autopilot: Automatically explores data, applies feature processors, and trains multiple models to generate candidate pipelines with explainability reports.

Frameworks built on top of traditional ML libraries that use advanced Bayesian optimization and genetic programming to automate pipeline construction.

• Auto-Sklearn: An extension of scikit-learn that uses Bayesian optimization, meta-learning, and ensemble construction to automate algorithm selection and hyperparameter tuning.
• TPOT (Tree-based Pipeline Optimization Tool): Uses genetic programming to evolve entire ML pipelines (preprocessors, feature selectors, models) represented as trees of operators. It optimizes for pipeline structure as well as hyperparameters.

Frameworks dedicated to the automated search for optimal neural network architectures, a core subfield of AutoML. They often employ weight sharing and differentiable NAS (DNAS).

• DARTS (Differentiable Architecture Search): A seminal DNAS method that formulates the search space as a continuous supernet, allowing architecture parameters to be optimized via gradient descent.
• ENAS (Efficient Neural Architecture Search): Uses a controller RNN and weight sharing across child models to achieve search efficiency.
• ProxylessNAS: Conducts architecture search directly on the target task and hardware without using proxy datasets or networks, making it hardware-aware.

Core libraries that provide the algorithmic backbone for AutoML, focusing specifically on the efficient search of hyperparameters.

• Optuna: A define-by-run HPO framework that supports Bayesian optimization, population-based training (PBT), and pruning of unpromising trials.
• Ray Tune: A scalable library for distributed HPO and NAS, supporting most search algorithms (HyperOpt, Bayesian, PBT) and integrating with major ML frameworks.
• Hyperopt: A Python library for serial and parallel Bayesian optimization over awkward search spaces, using a Tree-structured Parzen Estimator (TPE) algorithm.

A core AutoML subfield focused on algorithmically discovering high-performing neural network architectures. In the context of PEFT, NAS can be used to automate the design of adapter modules or the search for optimal low-rank dimensions in methods like LoRA. It optimizes for multiple objectives:

• Task performance (accuracy, F1-score)
• Parameter efficiency (number of trainable parameters)
• Inference latency Techniques like Differentiable NAS (DNAS) and One-Shot NAS are particularly relevant for efficiently searching over PEFT structural hyperparameters.

The automated process of searching for the optimal set of hyperparameters that control a model's learning process. For PEFT, critical hyperparameters include:

• Adapter size (bottleneck dimension)
• LoRA rank (r)
• Learning rate for the new parameters
• Dropout within adapter layers Methods like Bayesian Optimization and Population-Based Training (PBT) are used to efficiently navigate this search space, balancing adaptation quality with computational cost. HPO is essential for realizing the full potential of any PEFT method.

A neural network that generates the weights for another primary network (the target model). This provides a powerful, dynamic mechanism for parameter-efficient adaptation:

• The hypernetwork takes a task or condition embedding as input.
• It outputs the weights for a small set of task-specific layers or modules in the frozen base model.
• Only the hypernetwork's weights are trained, making it a highly flexible PEFT method. This approach enables conditional computation and is closely related to the concept of delta tuning, where the hypernetwork generates the parameter change (delta) applied to the base model.

A variant of NAS that optimizes for multiple, often competing objectives simultaneously. This is critical for enterprise PEFT deployment, where trade-offs must be managed:

• Primary Objective: Task accuracy or performance.
• Efficiency Objectives: Number of added parameters, inference latency, memory footprint. The output is not a single architecture but a Pareto-optimal frontier, a set of solutions where improving one objective worsens another. This allows engineers to select the optimal PEFT configuration based on specific deployment constraints (e.g., edge device memory).

A specialized form of NAS that incorporates direct hardware performance metrics into the search objective. For PEFT, this means finding adaptation architectures optimized for a specific deployment chip (e.g., GPU, NPU, microcontroller). The search optimizes for:

• Measured latency on target hardware.
• Peak memory usage during inference.
• Energy consumption. This is achieved by integrating hardware performance predictors or on-device profiling into the search loop, ensuring the discovered PEFT configuration is not just parameter-efficient but also runtime-efficient in production.

The process of algorithmically determining the optimal sparsity pattern for a neural network. In PEFT, this can be applied to the trainable parameters themselves to achieve extreme efficiency. Techniques include:

• Automated pruning of adapter weights or LoRA matrices.
• Searching for sparse activation patterns in mixture-of-experts (MoE) gating networks.
• Learning which neurons or attention heads in the base model should be selectively fine-tuned. The goal is to maximize task performance while minimizing the active parameter count per forward pass, which directly impacts inference cost. This intersects with sparse fine-tuning PEFT methods.