Label Smoothing

Instead of using a hard target vector [0, 0, 1, 0], label smoothing replaces it with a soft target like [ε/4, ε/4, 1-ε, ε/4], where ε (epsilon) is a small constant, typically 0.1.

• The correct class is assigned a probability of 1 - ε.
• The remaining ε mass is distributed uniformly among the K-1 incorrect classes.
• This prevents the model from being forced to predict exactly 1.0 for the true class, which would require infinite logits.

Label smoothing directly combats overconfidence, a primary driver of miscalibration. By training on targets that are never 1.0, the model's output probabilities are naturally softened.

• The model learns to assign non-zero probability to plausible alternatives.
• This results in a tighter Expected Calibration Error (ECE) without requiring a post-hoc step like temperature scaling.
• It encourages the logits of the correct class to be closer to the logits of other classes, creating a more diffuse, better-calibrated distribution.

Label smoothing with a uniform distribution is a special case of knowledge distillation where the teacher model provides a uniform, uninformative prior.

• In full distillation, a complex teacher model provides a rich, non-uniform soft target distribution that captures inter-class similarities.
• Uniform label smoothing is a simpler, teacher-free baseline that still provides the regularization benefits of soft targets.
• Both methods increase the entropy of the training targets, discouraging peaked output distributions.

Label smoothing regularizes the penultimate layer's representation space by preventing the model from collapsing intra-class samples into extremely tight, low-variance clusters.

• Without smoothing, the model is incentivized to maximize the logit gap between the correct class and all others, pushing same-class embeddings arbitrarily close together.
• By softening targets, the model tolerates a wider spread of embeddings within a class.
• This wider spread improves generalization and makes the model more robust to adversarial perturbations.

Label smoothing modifies the standard cross-entropy loss by adding a penalty term proportional to the KL divergence between a uniform distribution and the model's predictions.

• The effective loss becomes: L = (1-ε) * CrossEntropy(y, p) + ε * KL(uniform || p).
• This second term penalizes the model for being too confident in any single class.
• It smooths the loss landscape, leading to flatter minima that are associated with better calibration and improved out-of-distribution detection.

Label smoothing is not universally beneficial and can degrade performance in specific scenarios.

• Distillation teachers: Applying smoothing to a teacher model can erase valuable inter-class similarity information, producing a less informative soft target for the student.
• Zero-shot and few-shot learning: Smoothing can wash out the sharp decision boundaries needed for recognizing novel classes with limited examples.
• Beam search in language models: Smoothed probabilities can flatten the distribution, reducing the discriminative power needed for selecting high-quality sequences.

The primary metric for diagnosing miscalibration. ECE partitions predictions into M equally-spaced bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: ECE = Σ (|B_m|/n) * |acc(B_m) - conf(B_m)|
• A perfectly calibrated model has an ECE of 0.
• Label smoothing directly minimizes ECE by preventing the model from pushing confidence to 1.0.

A post-hoc calibration method that uses a single scalar parameter T to soften the softmax distribution. It divides all logits by T before applying softmax.

• T > 1: Softens probabilities, reducing overconfidence.
• T < 1: Sharpens probabilities, increasing peakiness.
• The optimal T is learned on a held-out validation set by minimizing NLL.
• Unlike label smoothing, this is applied after training and does not affect the learned feature representations.

A dynamically scaled cross-entropy loss that adds a modulating factor (1 - p_t)^γ to standard cross-entropy.

• Down-weights the loss contribution from well-classified examples (where p_t is high).
• Forces the model to focus on hard, misclassified samples.
• The γ parameter controls the down-weighting strength; γ=0 recovers standard cross-entropy.
• Like label smoothing, it acts as an implicit regularizer that prevents overconfident predictions on easy examples.

A data augmentation strategy that trains on convex combinations of random input pairs and their corresponding labels.

• x̃ = λx_i + (1-λ)x_j, ỹ = λy_i + (1-λ)y_j
• λ is sampled from a Beta(α, α) distribution.
• Encourages the model to behave linearly between training samples, reducing oscillatory, overconfident decision boundaries.
• Empirically improves both generalization and calibration, complementing the effect of label smoothing.

A strictly proper scoring rule that evaluates the quality of a probabilistic classifier. NLL = -Σ log(p̂_i[y_i]).

• Heavily penalizes confident misclassifications: assigning 0.99 probability to the wrong class incurs a massive loss.
• Minimized only when the predicted distribution matches the true data-generating distribution.
• Label smoothing increases the raw NLL on the training set (since it prevents perfect 1.0 predictions) but typically improves NLL on held-out test data.

A method for uncertainty quantification that trains M independent neural networks with different random initializations and data shuffles.

• At inference, predictions are averaged: p(y|x) = (1/M) Σ p_m(y|x).
• The ensemble's aggregate distribution is typically better calibrated than any single member.
• Combines naturally with label smoothing; smoothing each member independently produces a robust, well-calibrated mixture model.