Confidence Calibration

A model is perfectly calibrated if, for all samples where it predicts a confidence of p, the empirical accuracy is exactly p. For example, if a model assigns 80% confidence to 100 predictions, exactly 80 of those predictions should be correct.

• Reliability Diagram: A plot of expected sample accuracy against confidence, where a perfectly calibrated model follows the identity line.
• Expected Calibration Error (ECE): The primary metric that bins predictions by confidence and computes the weighted average of the absolute difference between accuracy and confidence.
• Overconfidence: When predicted probability systematically exceeds actual accuracy, a common failure mode in modern neural networks using ReLU activations and batch normalization.

Platt Scaling fits a logistic regression model to the raw classifier scores, transforming them into calibrated probabilities. It works well when the distortion in predicted probabilities is sigmoidal.

• Isotonic Regression: A non-parametric method that learns a piecewise constant, monotonically increasing mapping from scores to probabilities. It is more flexible than Platt scaling but prone to overfitting on small datasets.
• Use Case: Platt scaling is preferred for small calibration sets, while isotonic regression excels when abundant calibration data is available and the distortion is non-sigmoidal.
• Open Set Relevance: Both methods calibrate the known-class probabilities, enabling a more statistically sound threshold for rejecting unknown emitter classes.

A simple non-parametric calibration method that divides predictions into equal-width or equal-frequency bins and replaces each predicted probability with the observed accuracy within that bin.

• Equal-Width Binning: Divides the [0,1] probability interval into fixed segments, but can result in empty bins in sparse regions.
• Equal-Frequency Binning: Ensures each bin contains the same number of samples, providing more stable accuracy estimates across the full confidence range.
• Trade-off: Fewer bins reduce variance but increase bias; more bins reduce bias but increase variance. Bayesian binning priors can optimize this trade-off automatically.

In open set emitter recognition, calibration serves a dual purpose: it ensures that known-class probabilities are trustworthy and that the rejection threshold for unknowns is statistically grounded.

• Threshold Selection: A well-calibrated model allows operators to set a rejection threshold (e.g., < 70% confidence) with predictable false-positive and false-negative rates.
• OpenMax Integration: The OpenMax algorithm uses Extreme Value Theory to recalibrate SoftMax probabilities, but the underlying model must still be calibrated for the Weibull fitting to be valid.
• Joint Calibration: Advanced systems jointly calibrate the closed-set SoftMax output and the open-set uncertainty score to produce a unified, interpretable confidence metric.

Calibration in multi-class settings is more complex than binary calibration because the predicted probability vector must sum to 1, introducing interdependencies between class confidences.

• Classwise ECE: Computes calibration error independently for each class, revealing that a model may be well-calibrated globally but poorly calibrated for specific rare emitter types.
• Confidence Calibration: The standard approach that examines only the maximum predicted probability, suitable when the primary concern is whether the top prediction is trustworthy.
• Adaptive Calibration: For RF fingerprinting with many emitter classes, classwise calibration often reveals that rare or similar emitters require separate temperature parameters for optimal reliability.

A visual tool for diagnosing calibration quality. The diagram plots expected sample accuracy against confidence (or predicted probability).

• Perfect Calibration: The plot follows the identity line (y = x).
• Overconfidence: The curve falls below the identity line (high confidence, low accuracy).
• Underconfidence: The curve rises above the identity line (low confidence, high accuracy). These plots often include a histogram of prediction densities to show where most samples fall.

A strictly proper scoring rule that measures the mean squared error between predicted probabilities and the actual binary outcomes. It decomposes into refinement (resolution + uncertainty) and calibration components.

• Range: 0 (perfect) to 1 (worst).
• Unlike ECE, the Brier score is sensitive to both calibration and discrimination, penalizing confident wrong answers more heavily than hesitant wrong answers.

A non-parametric calibration method that partitions uncalibrated predictions into K bins and assigns a calibrated probability based on the empirical accuracy within each bin.

• Equal-width: Bins have uniform intervals.
• Equal-frequency: Bins contain the same number of samples.
• Requires sufficient data per bin to avoid overfitting, making it less suitable for extreme tail probabilities compared to parametric methods like Platt scaling.

A parametric method that fits a logistic regression model to the raw logits of a binary classifier to produce calibrated probabilities. It learns two scalar parameters A and B such that P(y=1|x) = 1 / (1 + exp(A * logit + B)).

• Originally designed for SVMs.
• Prone to overfitting if the calibration set is small, but highly effective for binary open set rejection thresholds.