Search Space

A search space is formally defined by a set of architectural decisions (e.g., number of layers, operation types, connection patterns) and their associated allowable values or choices. It is often represented as a directed acyclic graph (DAG) for computational graphs or a hierarchical structure for complex networks. Key structural paradigms include:

• Cell-based search spaces: Search for a optimal computational cell (e.g., a Normal or Reduction cell) that is then stacked to form the full network, as used in NASNet and ENAS.
• Hierarchical spaces: Define macro-architectures (overall network skeleton) and micro-architectures (operations within each block), enabling search at multiple levels of granularity.
• Continuous relaxation spaces: Used in Differentiable NAS (DNAS), where the choice between discrete operations is relaxed into a continuous, weighted mixture, allowing gradient-based optimization.

The design of a search space involves a fundamental trade-off between expressivity and tractability. A highly expressive space contains many high-performing candidate solutions but is vast and difficult to search efficiently. A constrained space is easier to search but may exclude optimal architectures.

• Expressivity: The capacity of the space to represent a diverse set of high-quality, novel architectures. An overly restrictive space can inductive bias the search towards known, sub-optimal designs.
• Tractability: The feasibility of navigating the space within practical computational budgets. The size of the search space grows combinatorially with the number of decisions; for example, a space with 10 layers and 5 operation choices per layer results in 5^10 (~9.7 million) possible architectures. Effective search space design incorporates prior knowledge (e.g., known effective motifs like residual connections) to constrain the space intelligently without sacrificing potential for discovery.

The search space is not independent; its definition is intrinsically linked to the choice of search algorithm. Different algorithms require the space to be structured in specific ways:

• Reinforcement Learning (RL) & Evolutionary Methods: Treat the space as a discrete set of actions or genes. They require a method to encode an architecture into a string or sequence (the architecture encoding).
• Differentiable NAS (e.g., DARTS): Requires the space to be continuously relaxable. Each discrete choice (e.g., 'conv3x3' or 'skip_connect') is represented as a continuous variable (architecture weight) within a supernet.
• Bayesian Optimization: Uses a surrogate model (e.g., Gaussian Process) to model the performance landscape of the space. It requires an efficient kernel function to measure similarity between two different architecture encodings.
• One-Shot / Weight-Sharing NAS: Depends on a supernet that subsumes all architectures in the search space. The space must allow for weight sharing across all candidate sub-networks (child models) during the supernet training phase.

Modern search spaces are increasingly designed to be hardware-aware and multi-objective. Instead of solely maximizing accuracy, the space includes decisions that directly impact deployment metrics.

• Hardware-Aware Decisions: The search space can include parameters that affect inference latency, memory footprint, or energy consumption. Examples include:
  • Kernel size choices (3x3 vs 5x5).
  • Channel width multipliers.
  • Choice between different operator implementations (e.g., standard conv vs. depthwise-separable conv).
• Multi-Objective Optimization: The search space is evaluated against a Pareto frontier balancing competing objectives like accuracy vs. latency. This requires the search algorithm to navigate the space to find optimal trade-offs, not a single best point.
• Efficiency Dimensions: Spaces can be explicitly structured along efficiency axes, such as in the Once-For-All (OFA) network, which defines sub-spaces for different depth, width, and resolution settings all contained within one supernet.

Concrete examples illustrate how search spaces are implemented:

• NASNet Search Space: A cell-based space where the algorithm searches for two types of cells. Each cell is a small directed graph with nodes representing feature maps and edges representing candidate operations (conv, pooling, identity, zero).
• DARTS Search Space: A continuous relaxation of a cell-based space. Each edge between nodes holds a mixture of all possible operations, weighted by learnable architecture parameters (alpha).
• EfficientNet Search Space (MBConv): The space is built around Mobile Inverted Bottleneck Conv (MBConv) blocks with searchable parameters: expansion ratio, kernel size (3,5,7), and squeeze-and-excitation ratio. The search also determines the number of layers per stage and channel widths.
• Transformer Architecture Search: Spaces for NLP models may include decisions on the number of attention heads, feed-forward network dimension, attention mechanism type (e.g., full, sparse, linear), and the arrangement of encoder/decoder layers.
• Hyperparameter Search Space: For HPO, the space is typically a bounded Cartesian product of ranges for each hyperparameter (e.g., learning rate: log-uniform [1e-5, 1e-1], dropout rate: uniform [0.0, 0.5]).

Search spaces are not always static; they can be dynamically pruned or evolved during the search process to improve efficiency.

• Pruning: Low-performing or irrelevant regions of the space can be eliminated early. This uses zero-cost proxies (e.g., gradient norm, synaptic saliency) or a neural predictor to score architectures without full training, allowing the search to focus on promising subspaces.
• Progressive Search: Starts the search in a smaller, simpler space and gradually expands its complexity (e.g., increasing network depth) as the search progresses, a technique used in Progressive NAS.
• Evolutionary Expansion: In architecture evolution methods, the space itself can be morphed by applying mutation and crossover operations to high-performing parent architectures, effectively exploring a dynamically defined space centered on good solutions. The goal is to reduce the effective search space the algorithm must exhaustively evaluate, transforming an intractable combinatorial problem into a manageable one.

Modern search spaces are structured hierarchically to manage complexity. This design mirrors how human experts build networks.

• Macro-Architecture: Defines the high-level skeleton, such as the number of stages or blocks (e.g., ResNet stages, Transformer encoder blocks).
• Micro-Architecture: Defines the operations within each block (e.g., choice between a 3x3 convolution, 5x5 depthwise convolution, or identity operation in a cell).
• Hyperparameters: Defines continuous or discrete settings like layer width (channels), expansion ratios, dropout rates, or learning rates.

This separation allows the search algorithm to reason about structure at different levels of abstraction, making the exploration tractable.

A dominant paradigm in NAS defines a repeatable computational cell as the fundamental building block. The search discovers the optimal internal graph of this cell, which is then stacked to form the final network.

• Normal and Reduction Cells: Often, two cell types are searched: one for processing data at the same resolution and one for downsampling.
• Operation Set: The candidate operations for each node in the cell graph (e.g., conv 3x3, sep_conv 5x5, max_pool 3x3, skip_connect, zero).
• Differentiable Formulation: In Differentiable NAS (DNAS), the discrete choice between operations is relaxed using a continuous mixture (e.g., Softmax over architecture parameters), allowing gradient-based optimization of the cell structure.

To make searching over vast spaces feasible, the One-Shot NAS paradigm uses a supernet—an over-parameterized network that contains all possible architectures in the search space as subgraphs.

• Weight-Sharing: All candidate sub-architectures share weights from the supernet. Evaluating a candidate involves activating its specific path and using the shared weights for inference, avoiding training from scratch.

Critical Challenge: Ranking Consistency. The performance ranking of architectures using shared weights must correlate highly with their performance if trained independently. Poor weight-sharing leads to misleading search results.

For production deployment, search spaces are constrained by target hardware metrics from the outset. This moves beyond just accuracy to multi-objective optimization.

Space dimensions can include:

• Kernel Sizes (3x3 vs. 5x5 vs. 7x7)
• Channel Widths (e.g., choices from [16, 32, 64, 128])
• Network Depth (number of blocks)
• Attention Heads (in Transformers)
• Precision (FP16, INT8)

Each candidate architecture is evaluated against a latency/energy lookup table or an on-device profiler during the search, ensuring the final model meets real-world constraints.

In Parameter-Efficient Fine-Tuning, the search space defines how and where to insert adaptable parameters into a frozen base model.

Key dimensions include:

• Adapter Placement: Which transformer layers receive adapter modules (e.g., every layer, every other layer, only last N layers).
• Adapter Configuration: Bottleneck dimension, activation function, and placement (after attention or after FFN).
• LoRA Rank: The intrinsic rank (r) of the low-rank update matrices.
• Prefix/Prompt Length: The number of continuous token vectors to prepend.
• Sparse Mask: Which subset of model parameters (e.g., attention heads, FFN layers) are eligible for fine-tuning.

Automated search here aims to find the optimal efficiency-performance trade-off for a given downstream task.

A well-designed search space balances expressivity and size.

Principles:

• Expressivity: Must contain high-performing architectures for the target task.
• Size: Must be small enough to search efficiently, but not so small that the optimal solution is excluded.
• Smoothness: Neighboring architectures in the space should have similar performance, enabling guided search.

Common Pitfalls:

• Overly Large Spaces: Exponentially increase search cost; may require billions of evaluations.
• Poor Inductive Bias: Spaces lacking basic proven design principles (e.g., skip connections) waste time rediscovering them.
• Search-Validation Mismatch: An architecture found via weight-sharing may not train as well from scratch (performance collapse).

Neural architecture search (NAS) is the automated process of discovering high-performing neural network architectures within a predefined search space. It treats architecture design as an optimization problem, using algorithms like reinforcement learning, evolutionary strategies, or gradient-based methods to evaluate candidates. The search space defines the set of all possible layer types, connections, and hyperparameters that the NAS algorithm can explore.

• Core Objective: Automate the design of network topologies for specific tasks and hardware constraints.
• Key Challenge: Balancing the expressiveness of the search space (its ability to contain good solutions) with its size (which impacts search cost).
• Example: Searching for the optimal combination of convolutional filter sizes, number of layers, and skip connections for a mobile vision model.

Hyperparameter optimization (HPO) is the systematic search for the optimal set of hyperparameters that govern a model's training process. The hyperparameter search space includes variables like learning rate, batch size, dropout rate, and regularization strength. Unlike NAS, which searches over model architecture, HPO searches over model training configuration.

• Common Algorithms: Grid search, random search, and Bayesian optimization.
• Search Space Design: Can be categorical (optimizer type), continuous (learning rate), or integer (number of epochs).
• Critical Consideration: The search space must be carefully bounded based on domain knowledge to avoid wasteful exploration of ineffective regions.

Bayesian optimization is a sample-efficient, sequential strategy for optimizing black-box functions, making it ideal for navigating complex search spaces where each evaluation (e.g., training a model) is expensive. It builds a probabilistic surrogate model (typically a Gaussian process) to predict the performance of unexplored configurations and uses an acquisition function to decide the most promising point to evaluate next.

• Key Advantage: Intelligently balances exploration (trying uncertain areas of the space) and exploitation (focusing on known good areas).
• Role in Search: Provides a principled framework for guiding the search through a high-dimensional space with a limited budget of evaluations.
• Use Case: Optimizing the hyperparameters of a large language model's fine-tuning process.

Differentiable neural architecture search (DNAS) is a gradient-based NAS method that transforms the discrete search space into a continuous, differentiable one. It constructs a supernet that contains all possible candidate operations as parallel paths. Architecture parameters (e.g., mixing weights for operations) are optimized via gradient descent alongside the model weights, making the search highly efficient.

• Mechanism: Uses continuous relaxation (e.g., Gumbel-Softmax) to make architectural choices differentiable.
• Contrast with RL-based NAS: Avoids the need for a separate controller network and is significantly faster.
• Outcome: After optimization, a discrete architecture is derived from the continuous parameters (e.g., by selecting the operation with the highest weight).

Multi-objective neural architecture search extends NAS to optimize for several, often competing, objectives simultaneously. The search space is explored to find architectures that represent the best trade-offs, forming a Pareto front. Common objectives include model accuracy, inference latency, memory footprint, and energy consumption.

• Key Driver: Practical deployment requires balancing performance with efficiency.
• Hardware-Aware NAS: A subset that directly incorporates hardware metrics (e.g., measured latency on a target device) into the multi-objective search.
• Result: Not a single "best" model, but a set of optimal models from which to choose based on deployment constraints.

A hypernetwork is a neural network that generates the weights for a primary target network. In the context of automated configuration, a hypernetwork can be used to dynamically produce parameters for architectures within a search space. This allows a single hypernetwork to compactly represent a vast, continuous family of models.

• Link to Search Space: The input to the hypernetwork (e.g., an architecture encoding) specifies a point within the search space, and its output is the corresponding model weights.
• Efficiency Benefit: Enables rapid evaluation of many architectures without training each from scratch, facilitating efficient search.
• Application in PEFT: Can generate parameters for adapter modules or other PEFT components conditioned on task or resource constraints.