Binary Cross-Entropy

The loss for a single prediction is calculated as: L = -(y * log(p) + (1 - y) * log(1 - p))

• y: The true label (0 or 1).
• p: The predicted probability of the positive class. When the true label is 1, the loss simplifies to -log(p), driving the model to push p towards 1. When the label is 0, the loss is -log(1-p), driving p towards 0.

BCE applies a logarithmic penalty that grows exponentially as the prediction diverges from the truth.

• A confident wrong prediction (e.g., predicting 0.99 when the true label is 0) generates an extremely high loss value approaching infinity.
• A slightly wrong prediction (e.g., predicting 0.51 when the true label is 0) generates a much smaller, manageable loss. This property forces models to be well-calibrated and avoid overconfidence in incorrect classifications.

BCE naturally handles soft labels (probabilistic ground truth), not just hard 0/1 labels.

• In Click-Through Rate (CTR) prediction, an impression with a 30% empirical CTR can be used as y=0.3.
• This allows the model to learn from aggregated statistics rather than noisy individual binary outcomes.
• This is a key advantage over accuracy-based metrics, which require thresholded hard decisions.

Direct computation of log(p) can cause numerical instability when p approaches 0 or 1, leading to -inf or NaN values.

• Solution: Deep learning frameworks use a fused operation that clips predicted logits or applies the log-sum-exp trick.
• In practice, p is typically clamped to a small epsilon range like [1e-7, 1 - 1e-7] to prevent undefined gradients.
• This ensures stable backpropagation even with highly confident predictions.

Minimizing Binary Cross-Entropy is mathematically equivalent to Maximum Likelihood Estimation (MLE) for a Bernoulli distribution.

• The model's predicted probability p parameterizes a Bernoulli distribution over the binary outcome.
• Minimizing BCE finds the parameters that maximize the probability of observing the actual training labels.
• This connection provides BCE with strong statistical foundations and desirable properties like asymptotic consistency.

Unlike Hinge Loss used in Support Vector Machines, BCE produces continuous gradients even for correctly classified examples.

• Hinge Loss: Zero gradient once the margin is satisfied, leading to sparse updates.
• BCE: Always provides a non-zero gradient, continuously pushing probabilities closer to 0 or 1.
• This makes BCE preferable for calibration-sensitive applications like CTR prediction, where the exact probability value matters, not just the classification decision.