Stochastic Depth

During each training iteration, residual blocks are randomly deactivated with a probability that increases linearly with block depth. When a block is dropped, its transformation is skipped entirely, and the input passes directly through the identity connection. This creates a network with an effectively random depth for each mini-batch, preventing co-adaptation between specific layers.

• Survival probability decreases from 1.0 for early blocks to 0.5 for the final block
• Dropped blocks are bypassed entirely, not just zeroed out
• During inference, all blocks are active with no stochasticity

Training with Stochastic Depth simulates training an exponentially large ensemble of networks with different depths. For a network with L residual blocks, there are 2^L possible configurations. The final trained model represents an approximate geometric mean of all these sub-networks, providing ensemble-like regularization without the inference cost.

• Each mini-batch trains a different sub-network configuration
• The ensemble effect reduces overfitting on small datasets
• Comparable to dropout but operates at the block level rather than individual neurons

The survival probability for each block follows a linear decay rule: p_l = 1 - (l/L) * (1 - p_L), where l is the block index, L is the total number of blocks, and p_L is the survival probability of the final block (typically 0.5). This design principle ensures that early layers, which extract fundamental low-level features, are preserved more often than deeper, task-specific layers.

• Early blocks have near-certain survival (p ≈ 1.0)
• Final blocks have a 50% chance of being dropped
• The linear schedule balances feature extraction stability with deep layer regularization

By randomly dropping entire blocks, Stochastic Depth reduces the effective depth of the network during each forward and backward pass. This directly decreases computational cost and memory footprint, enabling faster training iterations. The reduction is proportional to the expected number of dropped blocks.

• Reduces FLOPs by approximately 25% during training
• Decreases GPU memory usage by avoiding activations from dropped blocks
• Enables training of deeper networks that would otherwise be prohibitively expensive

Stochastic Depth improves gradient propagation through very deep networks. When blocks are dropped, the identity connection provides a direct, unimpeded path for gradients to flow from the loss to earlier layers. This mitigates the vanishing gradient problem and enables effective training of networks with hundreds of layers.

• Identity connections act as gradient highways
• Reduces the effective path length for backpropagation
• Complements Batch Normalization and other stabilization techniques

In Vision Transformer architectures, Stochastic Depth is applied to the residual connections within each Transformer encoder block, dropping the entire multi-head self-attention and MLP sub-layers. This is critical for training deep ViTs like ViT-Large and Swin Transformer, which can have 24 or more encoder blocks.

• Applied independently to the attention and MLP sub-layers
• Essential for training data-efficient ViT variants on smaller datasets
• Often combined with Layer Scale for additional training stability

The foundational stochastic regularization technique that randomly zeros individual neurons during training with probability p. Unlike stochastic depth which drops entire residual blocks, dropout operates at the fine-grained neuron level, forcing the network to learn redundant representations that do not co-adapt. At inference, all neurons are active but their outputs are scaled by p to maintain expected values. Variants include SpatialDropout for convolutional feature maps and DropConnect which drops individual weights rather than activations.

A memory-optimization technique that trades compute for memory by discarding intermediate activations during the forward pass and recomputing them on demand during backpropagation. When combined with stochastic depth, the effective depth of the network varies per sample, making checkpointing strategies more complex. Checkpoint segments must align with residual block boundaries to maximize memory savings without disrupting the stochastic depth schedule.

An optimization algorithm that seeks parameters in neighborhoods of uniformly low loss rather than just low-loss points. SAM and stochastic depth are complementary: stochastic depth acts as an implicit regularizer by preventing co-adaptation, while SAM explicitly optimizes for flat minima that generalize better. When used together, they produce models with superior robustness to distribution shift and adversarial perturbations.

A per-channel multiplicative parameter initialized to a small value (typically ε < 1e-5) that scales the output of each residual branch before addition. Layer Scale stabilizes training in very deep Vision Transformers by dampening the contribution of each sub-layer initially. When combined with stochastic depth, Layer Scale provides an additional mechanism to control the effective contribution of each block, enabling more aggressive depth dropping schedules without training instability.

A training-free inference acceleration technique that reduces the number of tokens in a Vision Transformer by gradually merging similar redundant tokens based on bipartite soft matching. While stochastic depth reduces training cost by dropping blocks, ToMe reduces inference cost by pruning tokens. Both exploit the observation that not all computational elements contribute equally—blocks in training, tokens in inference.

A training paradigm that uses 16-bit floating-point (FP16 or BF16) for forward and backward passes while maintaining a 32-bit master copy of weights. When combined with stochastic depth, the reduced memory footprint from both techniques enables training deeper networks on the same hardware. The stochastic nature of depth dropping also helps mask the quantization noise introduced by lower precision, as dropped blocks contribute zero gradient regardless of precision format.