Confidence Score

In neural network classifiers, the final layer is typically a softmax function that squashes raw logits into a probability distribution over all classes. The confidence score is the maximum value in this vector.

• Values range from 0.0 to 1.0, with all class probabilities summing to 1.
• A score of 0.95 suggests the model assigns 95% probability mass to the predicted class.
• Critical caveat: Modern deep networks are often poorly calibrated, meaning a score of 0.9 does not imply a true 90% likelihood of correctness.

A model is perfectly calibrated if its confidence scores align with empirical accuracy. For example, among all predictions with a score of 0.8, exactly 80% should be correct.

• Overconfidence: The model outputs high scores but actual accuracy is lower. Common in standard neural networks.
• Underconfidence: The model outputs conservative scores despite high accuracy.
• Expected Calibration Error (ECE) quantifies the mismatch by binning predictions and computing the weighted average of the difference between accuracy and confidence.

Confidence scores are mapped to binary decisions (accept/reject) via an operating threshold. Adjusting this threshold trades off security against convenience.

• High threshold (e.g., 0.99): Minimizes false accepts but increases false rejects. Suitable for high-security authentication.
• Low threshold (e.g., 0.5): Maximizes accessibility but risks unauthorized access.
• The Receiver Operating Characteristic (ROC) curve and Detection Error Trade-off (DET) curve visualize this trade-off across all possible thresholds.

A raw softmax score is a poor indicator for out-of-distribution (OOD) inputs. A model can assign a high confidence score (e.g., 0.99) to a noise pattern or an unknown emitter class it has never seen.

• This occurs because softmax forces a relative comparison among known classes, not an absolute measure of familiarity.
• Temperature scaling and energy-based models are techniques used to recalibrate scores to better reflect epistemic uncertainty.
• For open-set recognition, a separate OOD detector or an explicit 'unknown' class is often required.

In metric-based few-shot learning architectures like Prototypical Networks, the confidence score is derived from the distance between a query embedding and class prototypes in the embedding space.

• A softmax over negative Euclidean distances or cosine similarities produces the probability distribution.
• A low confidence score indicates the query sample is far from all known prototypes, which can signal an imposter device or an open-set emitter.
• This geometric interpretation provides a more intuitive basis for rejection than standard classifier logits.

A single confidence score conflates different types of uncertainty. Advanced systems decompose this into:

• Aleatoric uncertainty: Inherent noise in the data, such as low SNR in an RF fingerprint. This is irreducible.
• Epistemic uncertainty: Model uncertainty due to lack of knowledge or training data. This is reducible with more data.
• Bayesian neural networks and Monte Carlo Dropout estimate epistemic uncertainty by sampling multiple forward passes and measuring prediction variance, providing a more robust rejection signal than a point-estimate confidence score.