Outcome Confidence Calibration

The gold standard where predicted probabilities exactly match observed frequencies across all confidence levels. For every prediction bucket (e.g., all cases assigned a 70% win probability), the actual win rate equals that bucket's value.

• Reliability Diagram: A plot of predicted probability vs. observed frequency forms a perfect diagonal line
• Brier Score: Approaches zero as calibration improves; decomposes into refinement and calibration components
• Practical Reality: Perfect calibration is asymptotically approached but rarely achieved in legal domains due to small sample sizes and distribution shift

The concentration of predictions toward 0% or 100%, reflecting the model's decisiveness independent of accuracy. A sharp model confidently distinguishes clear winners from clear losers rather than hedging with middling probabilities.

• Trade-off with Calibration: Overly sharp models may sacrifice calibration by overstating confidence
• Variance of Predictions: Measured by the spread of output probabilities across a case portfolio
• Information Content: Sharp, well-calibrated predictions provide more actionable intelligence for litigation risk stratification

The model's ability to maintain calibration as judicial behavior shifts over time. A static calibration layer degrades when legal outcome drift occurs due to changes in bench composition or evolving precedent.

• Recalibration Frequency: How often the calibration mapping is updated using recent case outcomes
• Drift Detection Triggers: Automated monitoring that flags when the Expected Calibration Error (ECE) exceeds a threshold
• Bayesian Updating: Incorporating new verdicts as they arrive to continuously refine probability estimates without full retraining

A parametric calibration method that fits a logistic regression model on top of raw classifier scores to produce calibrated probabilities. Particularly effective for binary legal outcomes like motion grants or denials.

• Mechanism: Learns two parameters (A and B) to transform raw scores: P(y=1|x) = 1 / (1 + exp(A × score + B))
• Data Efficiency: Requires relatively few held-out calibration cases compared to non-parametric methods
• Limitation: Assumes a sigmoidal relationship between scores and true probabilities, which may not hold for complex multi-document legal reasoning outputs

A non-parametric calibration technique that learns a monotonically increasing step function mapping raw scores to calibrated probabilities. Makes no assumptions about the functional form of the distortion.

• Flexibility: Can correct arbitrary monotonic miscalibrations that Platt Scaling cannot capture
• Overfitting Risk: Requires more calibration data than parametric methods; prone to fitting noise in small jurisdiction-specific datasets
• Piecewise Constant Output: Produces discrete probability steps rather than smooth curves, which may be desirable for litigation risk stratification tiers

The primary metric for measuring miscalibration, computed by partitioning predictions into M equal-width bins, then averaging the absolute difference between accuracy and confidence within each bin.

• Calculation: ECE = Σ (|B_m|/n) × |acc(B_m) - conf(B_m)| across all bins
• Bin Selection: Typical choices are 10 or 15 bins; too few masks miscalibration, too many creates noisy estimates
• Complementary Metrics: Use alongside Maximum Calibration Error (MCE) to identify worst-case reliability failures in high-stakes legal predictions

The primary empirical metric for measuring miscalibration. ECE partitions predictions into M equally-spaced bins based on confidence and computes the weighted average of the absolute difference between accuracy and confidence within each bin.

• Formula: A lower ECE indicates better calibration.
• Limitation: ECE is sensitive to the chosen number of bins and can be gamed by models that predict extreme probabilities.

A visual diagnostic tool for calibration. It plots observed frequency against predicted probability.

• Perfect Calibration: The plot follows the identity line (y = x).
• Under-Confidence: The curve sits above the identity line for low probabilities and below it for high probabilities.
• Over-Confidence: The curve sits below the identity line for low probabilities and above it for high probabilities, a common failure mode in modern neural networks.

A strictly proper scoring rule that measures the mean squared difference between the predicted probability and the actual binary outcome.

• Decomposition: The Brier Score can be mathematically decomposed into refinement and calibration loss components.
• Comparison: Unlike ECE, the Brier Score is a continuous, bin-free metric that jointly assesses discrimination and calibration, penalizing confident wrong answers more heavily.

A parametric post-hoc calibration method that fits a logistic regression model to the raw outputs (logits) of a classifier.

• Mechanism: It learns two scalar parameters (A and B) to transform logits into calibrated probabilities.
• Best Use Case: Works well for small datasets and is effective at correcting sigmoidal distortions in the output probability curve without overfitting.

A non-parametric calibration technique that fits a piecewise constant, monotonically increasing function to map model scores to empirical probabilities.

• Advantage: More flexible than Platt Scaling as it makes no assumptions about the functional form of the distortion.
• Risk: Prone to overfitting on small datasets, which can increase variance and degrade calibration on hold-out data.

A single-parameter extension of Platt Scaling designed for multi-class classification. It divides all logits by a scalar temperature parameter (T) before applying the softmax function.

• Effect: T > 1 softens the probability distribution (reducing over-confidence), while T < 1 sharpens it.
• Key Benefit: It preserves the rank order of predictions (accuracy is unchanged) while significantly improving the Expected Calibration Error.