Kernel Size Search

In kernel size search, the algorithm explores a discrete set of possible filter dimensions (e.g., 1x1, 3x3, 5x5, 7x7) for each convolutional layer. This is a fundamental axis of the overall Neural Architecture Search (NAS) search space, alongside operator search and channel search. The choice of kernel size directly controls the receptive field—the area of the input a neuron sees—and the number of parameters and multiply-accumulate (MAC) operations. For microcontroller targets, the search space is often aggressively pruned to exclude large, computationally expensive kernels like 7x7.

A larger kernel size increases the receptive field, allowing a single neuron to integrate information from a wider area of the input feature map. This is crucial for tasks like image classification where capturing long-range spatial dependencies can improve model accuracy. However, the relationship is non-linear. A 5x5 kernel has a receptive field of 5, but can often be approximated by two stacked 3x3 kernels (combined receptive field of 5) with fewer parameters and potentially better non-linearity. The search algorithm must evaluate this trade-off to find the optimal configuration for the target dataset.

Kernel size is a primary driver of computational cost. The number of operations in a standard convolution scales quadratically with kernel width/height. For a layer with C_in input channels, C_out output channels, and an H x W feature map:

• A 3x3 kernel requires ~9 * C_in * C_out * H * W MACs.
• A 5x5 kernel requires ~25 * C_in * C_out * H * W MACs. For memory-constrained microcontrollers, larger kernels also increase the size of the weight tensor stored in flash. A hardware cost model is used during search to predict the latency and energy impact of each candidate kernel size on the target MCU.

Kernel size search is rarely performed in isolation. It is tightly coupled with:

• Operator Search: Deciding between a standard conv, depthwise separable convolution, or pooling.
• Channel Search: Determining the width (number of filters) for each layer. For example, a search might substitute a costly 5x5 standard convolution with a more efficient combination of a 3x3 depthwise convolution followed by a 1x1 pointwise convolution (a MobileNet-like block). The NAS algorithm must evaluate these compound architectural decisions to find globally optimal designs.

Evaluating every possible kernel size via full training is prohibitively expensive. Kernel size search employs efficient NAS strategies:

• Weight Sharing (Supernet): A supernet is constructed where each layer has parallel branches for different kernel sizes (3x3, 5x5). The weights for these branches are trained concurrently, allowing rapid evaluation of sub-networks.
• Differentiable Search (DARTS): The choice between kernel sizes is relaxed into a continuous, learnable architectural parameter. The search optimizes these parameters via gradient descent.
• Zero-Cost Proxies: Simple metrics (e.g., gradient norm at initialization) can rank kernel size choices without any training, providing a fast, low-fidelity signal to guide the search.

For microcontroller NAS (MCU-NAS), kernel size search is heavily biased by hardware limits:

• Memory Constraint: Large kernels increase parameter count, which can exceed the few hundred KB of SRAM/Flash available. Search is constrained to keep model size under budget.
• Latency Constraint: The search objective includes a latency constraint measured via a hardware-in-the-loop profiler or a pre-characterized hardware cost model.
• Fixed-Point Arithmetic: Kernels must perform well under post-training quantization. Some kernel sizes may be less robust to quantization error, a factor considered in quantization-aware NAS (QA-NAS). The goal is to find the smallest kernel that delivers the necessary receptive field without violating strict hardware ceilings.

Kernel size is defined as a discrete, searchable parameter within the NAS search space. Common implementations allow the algorithm to select from a candidate set, such as {1x1, 3x3, 5x5, 7x7}, for each convolutional layer. This parameter is often searched jointly with other dimensions like operator type (e.g., regular vs. depthwise convolution) and channel count. In Differentiable Architecture Search (DARTS), kernel size choices are represented as continuous architecture parameters in a supernet, which are optimized via gradient descent.

In Hardware-Aware NAS, kernel size search is directly coupled with a hardware cost model. Larger kernels (e.g., 5x5) increase multiply-accumulate (MAC) operations and on-chip memory traffic. Frameworks like ProxylessNAS and MNasNet incorporate real or estimated latency and energy costs for each kernel size option on the target hardware (e.g., a specific microcontroller or NPU). The search objective becomes a multi-term loss function: Loss = Task_Loss + λ * Hardware_Cost(Kernel_Size, ...), steering the search toward efficient configurations.

To avoid the prohibitive cost of training every kernel-size variant from scratch, most modern NAS frameworks use weight sharing within a supernet. A single, over-parameterized network is trained where each layer contains filters for all candidate kernel sizes. During search, evaluating a sub-network with a specific kernel size configuration involves using the corresponding weights from the supernet. This One-Shot NAS approach, used by frameworks like Once-For-All (OFA), allows for rapid estimation of how kernel size impacts accuracy without full retraining.

For TinyML deployment, kernel size search must account for extreme constraints. Strategies include:

• Severely constrained candidate sets: Often limited to {1x1, 3x3} to minimize SRAM usage for feature maps.
• Depthwise convolution preference: Searching for kernel size within depthwise separable layers, where a 5x5 depthwise conv is far cheaper than a 5x5 standard conv.
• Direct hardware profiling: Hardware-in-the-loop search may compile and benchmark candidate layers on the actual MCU (e.g., an Arm Cortex-M4) to get cycle-accurate latency data for the cost model.
• Co-search with quantization: Quantization-Aware NAS (QA-NAS) may evaluate kernel size choices under simulated 8-bit or 4-bit integer arithmetic.

NAS frameworks implement kernel size search through different core strategies:

• Gradient-Based (e.g., DARTS): Treats the choice as a continuous relaxation. The supernet has parallel convolution paths with different kernel sizes; architecture parameters (alphas) control the weight of each path. The final discrete kernel size is selected by argmax on these parameters.
• Sampling-Based (e.g., RL-NAS, Evolutionary): A controller (RL agent) or evolutionary algorithm samples a discrete kernel size for each layer. The performance (reward/fitness) of the full sampled architecture is evaluated, often via the weight-sharing supernet, and used to update the sampler.
• Predictor-Based: A learned performance estimator or zero-cost proxy predicts the accuracy of an architecture based on its kernel size configuration, guiding the search without full evaluation.

Kernel size search does not occur in isolation; it interacts with other NAS decisions:

• Operator Search: The cost/benefit of a 5x5 kernel differs drastically between a standard convolution and a depthwise separable convolution.
• Channel Search: A larger kernel in a wide layer (high channel count) is more expensive than in a narrow layer. Joint optimization is key.
• Skip Connection Search: The presence of a residual bypass can change the optimal kernel size in a layer.
• Network Depth: In deeper networks, smaller kernels (3x3) are often sufficient to build a large effective receptive field, making large kernel searches less beneficial. Advanced frameworks model these interactions within the supernet structure.

The search space defines the universe of all possible neural network architectures a NAS algorithm can explore. It is parameterized by discrete choices, including:

• Layer types (e.g., convolution, pooling)
• Connectivity patterns (e.g., skip connections)
• Hyperparameters like kernel size, channel count, and number of layers. A well-designed search space balances expressiveness (to find good models) with tractability (to make search feasible). Kernel size search operates within this defined space.

A hardware cost model is a predictive function that estimates the real-world efficiency metrics—latency, energy consumption, memory usage—of a candidate neural network architecture on specific target hardware. In hardware-aware NAS, this model is integrated into the search objective. Instead of measuring on real devices (which is slow), the search algorithm uses this fast, approximate model to steer the search toward architectures that meet microcontroller constraints, making kernel size selections based on predicted computational cost.

Operator search is the process of selecting the fundamental mathematical operation applied at each layer of a neural network. While kernel size search selects the spatial dimensions of a filter, operator search selects the type of operation. Common choices in TinyML include:

• Standard Convolution
• Depthwise Separable Convolution (for extreme efficiency)
• Pooling Layers (Max, Average)
• Skip Connections or Identity The chosen operator directly interacts with the kernel size to define the layer's computational footprint and representational capacity.

Channel search optimizes the width of a neural network by selecting the number of output channels (i.e., filters) for each convolutional layer. This is a primary lever for controlling model capacity and parameter count. It works in tandem with kernel size search:

• A larger kernel (e.g., 5x5) has more parameters per filter.
• More channels increase the number of such filters. The search algorithm must balance these dimensions—often choosing smaller kernels and prudent channel counts—to fit within a microcontroller's severe memory budget (often < 512KB RAM).

A memory constraint is a hard upper bound on the peak RAM (activation memory) and flash (model storage) usage of a candidate neural network. This is the dominant constraint in Microcontroller NAS (MCU-NAS). The search algorithm, when performing kernel size search, must account for this:

• Larger kernels increase the size of convolutional weight tensors.
• The memory required for intermediate feature maps is also influenced by kernel size through padding and stride. The search is forced to discover architectures where the total memory footprint, calculated as Model Weights + Activation Memory, fits within the target's kilobyte-scale budget.

Differentiable Architecture Search (DARTS) is a gradient-based NAS method that makes the search space continuous. Instead of evaluating discrete candidates, it represents choices (like which operator or what kernel size to use) as a mixture of all options. During search, architecture parameters are optimized via gradient descent alongside model weights. For kernel size search, DARTS might learn a continuous distribution over sizes like 3x3, 5x5, and 7x7, eventually discretizing to the strongest candidate. This approach is more efficient than reinforcement learning or evolutionary-based searches.