Brier Score

The Brier Score calculates the mean squared error between the predicted probability vector and the actual one-hot outcome. For a binary event, it is computed as:

• Formula: ( \frac{1}{N} \sum_{t=1}^{N} (f_t - o_t)^2 )
• Range: 0 (perfect prediction) to 1 (worst prediction).
• Interpretation: It penalizes both overconfidence (predicting 0.99 when wrong) and underconfidence (predicting 0.6 when correct) heavily.

A key property is its decomposition into Calibration (reliability) and Refinement (resolution + uncertainty).

• Calibration: Measures how closely predicted probabilities match observed frequencies. A well-calibrated model predicts rain with 70% probability, and it actually rains 70% of the time.
• Refinement: Measures the model's ability to separate different outcomes, independent of calibration. It rewards predictions that are confidently close to 0 or 1, provided they are correct.

The Brier Score is a strictly proper scoring rule, meaning the expected score is uniquely minimized when the forecaster reports their true belief.

• Honesty Incentive: It mathematically prevents strategic manipulation. A forecaster cannot 'game' the score by reporting a probability different from their actual estimate.
• Comparison: Unlike accuracy, which encourages hedging with 0.5 predictions, the Brier Score rewards decisive, honest probability assessments.

For problems with more than two classes, the Brier Score generalizes to the mean squared difference across all classes.

• Formula: ( \frac{1}{N} \sum_{t=1}^{N} \sum_{i=1}^{R} (f_{ti} - o_{ti})^2 ), where R is the number of classes.
• Interpretation: It evaluates the entire predicted probability vector, not just the top-1 prediction. This makes it sensitive to how the model distributes probability mass among all incorrect classes.

While both are proper scoring rules, they penalize errors differently.

• Brier Score: Penalty is quadratic. A prediction of 0.99 for the wrong class adds 0.9801 to the score.
• Log Loss (Cross-Entropy): Penalty is logarithmic. The same error adds 4.605 to the score.
• Sensitivity: Log Loss is far more sensitive to extreme errors (tail risk), while the Brier Score provides a more bounded, linear view of miscalibration.

Engineers use the Brier Score to benchmark model confidence.

• Brier Skill Score: Often reported relative to a reference forecast (e.g., climatology) to show added value: ( 1 - \frac{BS_{model}}{BS_{ref}} ).
• Diagnostic Tool: A high Brier Score combined with a low ECE suggests the issue is poor refinement (inability to separate classes), not just miscalibration.
• Threshold: A Brier Score below 0.01 is generally considered excellent, while above 0.25 indicates performance worse than a constant baseline.

The primary companion metric to the Brier Score for diagnosing miscalibration. ECE partitions predictions into M bins and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Lower ECE indicates better calibration.
• Unlike the Brier Score, ECE isolates the calibration component from refinement.
• Often visualized using a Reliability Diagram.

The theoretical family to which the Brier Score belongs. A scoring rule is strictly proper if the expected score is minimized only when the forecaster reports their true belief distribution.

• Encourages honest probability assessments.
• The Brier Score is the canonical example for binary outcomes.
• Logarithmic Score (NLL) is another strictly proper rule that penalizes overconfidence more heavily.

An alternative proper scoring rule that evaluates the negative log of the predicted probability assigned to the true class. NLL is a local scoring rule, meaning it only considers the probability of the observed event.

• Heavily penalizes confident misclassifications.
• Unlike the Brier Score, NLL can approach infinity for a single wrong prediction with 100% confidence.
• Often used as the loss function during neural network training.

The standard visual diagnostic for calibration. It plots the predicted probability on the x-axis against the observed empirical frequency on the y-axis.

• A perfectly calibrated model follows the identity diagonal.
• Deviations above the diagonal indicate underconfidence.
• Deviations below the diagonal indicate overconfidence.
• The Brier Score can be decomposed into a calibration loss term directly visible on this plot.

A post-hoc calibration method that fixes overconfidence. It divides the model's output logits by a single scalar parameter T before applying the softmax function.

• T > 1 softens probabilities, reducing overconfidence.
• The optimal T is found by minimizing NLL on a validation set.
• This simple extension often dramatically improves both ECE and the Brier Score without changing the model's accuracy ranking.

The broader discipline of measuring a model's confidence in its predictions. The Brier Score is a key UQ metric that captures the total predictive error.

• Decomposes into Aleatoric Uncertainty (data noise) and Epistemic Uncertainty (model ignorance).
• A low Brier Score implies effective management of both uncertainty types.
• Critical for high-stakes applications like medical diagnosis and autonomous driving.