Rectified Linear Unit (ReLU)

ReLU is defined as f(x) = max(0, x) . For any positive input, the function outputs the input directly, acting as a linear identity function. For any negative input, the output is clamped to exactly zero.

• Derivative: 1 for x > 0, 0 for x < 0, undefined at x = 0 (handled by subgradient 0 in practice).
• Range: [0, ∞)
• This piecewise linearity is the source of its computational efficiency.

Unlike sigmoid or tanh, ReLU does not saturate in the positive region. The gradient is a constant 1 for all positive activations.

• Sigmoid Issue: Gradients approach zero at extreme values, stalling learning in early layers.
• ReLU Solution: The identity mapping for positive inputs allows gradients to flow backward unimpeded through deep networks.
• This property was critical in enabling the training of very deep architectures used in modern CTR models like Deep Interest Networks.

ReLU introduces sparse representations by outputting a true zero for all negative inputs. This is not merely a small value, but a hard zero.

• Biological Plausibility: Mimics the all-or-nothing firing rate of biological neurons.
• Computational Benefit: Sparse activations mean only a fraction of neurons are active at any time, reducing the effective computation in forward and backward passes.
• Regularization Effect: The sparsity acts as an implicit regularizer, often improving generalization in high-dimensional sparse CTR feature spaces.

A significant failure mode occurs when neurons become permanently inactive. If a large negative gradient flows through a ReLU neuron, it can push the weights such that the neuron's output is always negative for all inputs.

• Mechanism: Once dead, the gradient is permanently zero, and the neuron stops learning entirely.
• Common Cause: Excessively high learning rates or large negative bias terms.
• Mitigations: Using variants like Leaky ReLU (f(x) = max(0.01x, x)) or Parametric ReLU (PReLU) ensures a small, non-zero gradient for negative inputs, preventing permanent death.

ReLU is the default activation for deep CTR models not just for its mathematical properties, but for its extreme hardware efficiency.

• Simple Thresholding: The function requires only a comparison operation (x > 0) and a conditional assignment.
• No Exponentials: Unlike sigmoid or tanh, there are no expensive exponential calculations.
• Inference Speed: This simplicity translates directly to lower latency during online inference, a critical requirement for real-time bidding systems operating under 100ms deadlines.

While piecewise linear, ReLU is globally non-linear. Stacking multiple ReLU layers allows a deep network to model highly complex, non-linear feature interactions.

• Universal Approximator: A network with at least one hidden layer of ReLU neurons can approximate any continuous function.
• Interaction Modeling: In CTR prediction, this allows the model to automatically learn complex cross-feature relationships, such as the interaction between a user's time-of-day pattern and a specific item category, without manual feature crossing.

The critical training failure mode that ReLU was designed to solve. In deep networks using saturating activation functions like sigmoid or tanh, gradients become exponentially smaller as they are backpropagated through layers. This prevents weights in earlier layers from updating, stalling learning entirely. ReLU's constant gradient of 1 for positive inputs allows unimpeded gradient flow, enabling the training of networks with dozens or hundreds of layers that power modern CTR models.

A variant designed to solve the dying ReLU problem where neurons become permanently inactive and output only zero. Instead of outputting zero for negative inputs, Leaky ReLU outputs a small linear function:

• Formula: f(x) = max(0.01x, x)
• Benefit: Maintains a small, non-zero gradient during negative activations
• Trade-off: The slope coefficient (0.01) becomes a hyperparameter requiring tuning
• Use case: Often deployed when monitoring reveals a high percentage of dead units in standard ReLU layers

An adaptive improvement over Leaky ReLU where the negative slope coefficient is learned during training rather than fixed. Each neuron can learn its own optimal slope parameter through backpropagation:

• Formula: f(x) = max(αx, x), where α is a learnable parameter
• Advantage: Eliminates the need for manual tuning of the negative slope
• Cost: Negligible increase in parameter count (one per neuron)
• Performance: Consistently outperforms standard ReLU on large-scale image and CTR benchmarks when overfitting is not a concern

A smooth alternative that produces negative outputs for negative inputs, pushing mean activations closer to zero and reducing bias shift:

• Formula: f(x) = x if x > 0 else α(e^x - 1)
• Key property: Continuously differentiable, unlike ReLU's non-differentiable point at zero
• Normalization effect: Negative outputs naturally center activations, acting as a built-in batch normalization mechanism
• Computational cost: Higher than ReLU due to the exponential calculation, making it less common in latency-sensitive CTR serving environments

A high-performance activation function used in Transformer architectures and advanced CTR models like BERT-based recommenders. GELU weights inputs by their value rather than gating them by sign:

• Formula: f(x) = x * Φ(x), where Φ is the standard Gaussian CDF
• Behavior: Applies a stochastic, smooth regularization effect by multiplying the input by a value between 0 and 1
• Adoption: Standard in GPT, BERT, and Vision Transformer architectures
• CTR relevance: Increasingly used in deep interest networks that leverage multi-head self-attention for user behavior modeling

A failure condition where ReLU neurons become permanently trapped in the inactive state, outputting only zeros for all inputs. This occurs when:

• A large gradient update pushes weights into a configuration where the neuron never activates
• The learning rate is set too high, causing catastrophic weight updates
• Consequence: The neuron stops learning entirely and contributes nothing to the network
• Detection: Monitor the fraction of zero activations per layer during training
• Mitigation: Use Leaky ReLU, PReLU, or careful learning rate scheduling with gradient clipping