Precision and Recall

Precision (Positive Predictive Value) answers: "Of all the instances the model labeled as positive, how many were actually positive?" It's calculated as True Positives / (True Positives + False Positives).

Recall (Sensitivity, True Positive Rate) answers: "Of all the actual positive instances, how many did the model correctly identify?" It's calculated as True Positives / (True Positives + False Negatives).

These metrics are inherently in tension. Adjusting a model's classification threshold directly impacts the balance:

• Increasing the threshold (making predictions more conservative) typically increases precision (fewer false positives) but decreases recall (more false negatives).
• Decreasing the threshold (making predictions more liberal) typically increases recall (fewer false negatives) but decreases precision (more false positives). This trade-off is visualized with a Precision-Recall Curve.

The relative importance of precision vs. recall is dictated by the business or operational cost of different error types:

• High Precision Critical: Spam detection (cost of false positive: missing an important email), legal document review, diagnostic tests with risky follow-up procedures.
• High Recall Critical: Disease screening (cost of false negative: missing a sick patient), fraud detection in transactions, search engine retrieval (missing a relevant result is worse than returning some irrelevant ones).

Single scores that combine precision and recall to simplify model comparison:

• F1 Score: The harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). It equally weights both metrics.
• Fβ Score: A generalized F score where the β parameter controls the weight given to recall. (β > 1 weights recall more, β < 1 weights precision more).
• Average Precision (AP): The weighted mean of precisions at each threshold, with the increase in recall from the previous threshold used as the weight. Common in information retrieval.

Precision and Recall are derived directly from the four core counts in a confusion matrix:

• True Positives (TP): Correctly identified positives.
• False Positives (FP): Incorrectly identified positives (Type I error).
• True Negatives (TN): Correctly identified negatives.
• False Negatives (FN): Incorrectly identified negatives (Type II error). Precision = TP / (TP + FP). Recall = TP / (TP + FN). The matrix provides the complete picture these summary metrics distill.

For problems beyond binary classification:

• Multi-Class: Metrics are computed per-class (treating that class as "positive" and all others as "negative") and then aggregated via macro-average (simple mean across classes) or micro-average (pooling all class counts first).
• Multi-Label: Each instance can have multiple true labels. Precision and recall are calculated for each label independently and then averaged, or computed globally by examining the set of predicted vs. true labels for each instance.

A confusion matrix is the foundational table used to calculate precision, recall, and other classification metrics. It provides a complete breakdown of a model's predictions versus the true labels.

• Structure: Rows represent true classes, columns represent predicted classes.
• Core Cells: True Positives (TP), False Positives (FP), True Negatives (TN), False Negatives (FN).
• Use Case: Essential for moving beyond simple accuracy, especially with imbalanced datasets, by revealing the specific types of errors a model makes.

The F1 Score is the harmonic mean of precision and recall, providing a single metric that balances the trade-off between the two.

• Calculation: F1 = 2 * (Precision * Recall) / (Precision + Recall).
• Interpretation: It is most useful when you need a single number to compare models and when false positives and false negatives are of similar importance.
• Context: In error detection, a high F1 score indicates a system that is both accurate in its alerts (high precision) and comprehensive in catching errors (high recall).

Sensitivity is synonymous with recall (True Positive Rate). Specificity is the True Negative Rate, measuring a model's ability to correctly identify negative cases.

• Sensitivity (Recall): TP / (TP + FN). Proportion of actual positives correctly identified.
• Specificity: TN / (TN + FP). Proportion of actual negatives correctly identified.
• Trade-off: In medical diagnostics or fraud detection, the cost of a false negative (low sensitivity) versus a false positive (low specificity) dictates the optimal operating point on an ROC curve.

The Receiver Operating Characteristic (ROC) curve visualizes the trade-off between sensitivity (recall) and specificity across all classification thresholds. The Area Under the ROC Curve (AUC-ROC) summarizes this performance.

• ROC Curve: Plots True Positive Rate (Sensitivity) vs. False Positive Rate (1 - Specificity).
• AUC-ROC Interpretation: A value of 1.0 represents a perfect classifier; 0.5 represents a classifier with no discriminative power (random guessing).
• Application: Used to select an optimal threshold that balances precision and recall based on operational costs.

These are the fundamental statistical error types that precision and recall directly measure in a binary classification context.

• Type I Error (False Positive): Incorrectly rejecting a true null hypothesis. Precision measures the inverse of the Type I error rate among positive predictions.
• Type II Error (False Negative): Failing to reject a false null hypothesis. Recall measures the inverse of the Type II error rate among actual positives.
• Implication: In autonomous systems, a Type I error might be a false alarm, while a Type II error is a missed critical failure. The cost of each dictates whether to optimize for high precision or high recall.

Calibration error assesses the reliability of a model's predicted probabilities. A well-calibrated model's confidence scores (e.g., "90% sure this is an error") match the true likelihood of correctness.

• Problem: A model can have high precision/recall but be poorly calibrated, making its confidence scores unreliable for decision-making.
• Measurement: Often calculated via Expected Calibration Error (ECE) or Brier Score, which compares predicted probabilities to empirical outcomes.
• Importance for Agents: For recursive error correction, an agent must trust its own confidence scores to decide when to trigger a refinement loop or rollback.