F1 Score

Precision measures the correctness of a model's positive predictions. It answers the question: "Of all the instances the model labeled as positive, how many were actually positive?"

• Formula: Precision = True Positives / (True Positives + False Positives)
• High Precision indicates a low rate of false alarms. This is critical in scenarios where the cost of a false positive is high, such as spam detection (labeling a legitimate email as spam) or fraud alerting.
• A model with 95% precision that identified 100 transactions as fraudulent means approximately 95 of those were actual fraud.

Recall (or Sensitivity) measures the completeness of a model's positive identifications. It answers the question: "Of all the actual positive instances in the dataset, how many did the model correctly find?"

• Formula: Recall = True Positives / (True Positives + False Negatives)
• High Recall indicates the model misses few actual positives. This is paramount in medical diagnostics (e.g., cancer screening) or safety-critical systems, where failing to detect a real threat (a false negative) is unacceptable.
• A model with 90% recall for a disease present in 100 patients correctly identified 90 of them.

The F1 Score is calculated as the harmonic mean of Precision and Recall, not the simple arithmetic average. The harmonic mean penalizes extreme imbalances between the two component metrics.

• Formula: F1 Score = 2 * (Precision * Recall) / (Precision + Recall)
• Why Harmonic Mean? An arithmetic mean can be high even if one metric is very poor (e.g., Precision=1.0, Recall=0.1 has an arithmetic mean of 0.55). The harmonic mean for this case is ~0.18, accurately reflecting the poor overall performance.
• This property makes the F1 Score a robust single metric for imbalanced datasets, where one class (e.g., 'fraud') is much rarer than another ('not fraud').

In most classification models, increasing precision typically reduces recall, and vice-versa. This is a fundamental trade-off governed by the model's decision threshold.

• High Threshold: Model only makes very confident positive predictions. This increases Precision (fewer false positives) but decreases Recall (more false negatives).
• Low Threshold: Model makes more positive predictions. This increases Recall (fewer false negatives) but decreases Precision (more false positives).
• The F1 Score finds the optimal balance point for a given threshold. Analyzing the Precision-Recall Curve provides a complete view of this trade-off across all possible thresholds.

For multi-class classification, the method of averaging per-class F1 scores changes the metric's interpretation.

• Macro-F1: Computes F1 for each class independently, then takes the arithmetic mean. Treats all classes equally, which can be harsh if classes are imbalanced.
• Micro-F1: Aggregates all classes' contributions to the global True Positives, False Positives, and False Negatives first, then calculates one F1 score. This is dominated by the performance on the majority class.
• Weighted-F1: Calculates Macro-F1 but weights each class's contribution by its support (number of true instances), providing a balance between the two.
• Choice depends on business objective: Equal class importance (Macro), overall document-level accuracy (Micro), or a balanced view of imbalanced data (Weighted).

For autonomous AI agents, the F1 Score is adapted to evaluate task-oriented classification, such as intent recognition, tool selection accuracy, or success/failure in multi-step reasoning.

• Example - Tool Call Validation: An agent must decide which API to call. Precision measures how often a selected tool was the correct one. Recall measures how often the agent successfully invoked the correct tool when it was needed.
• Example - Success State Classification: In evaluating an agent's completed task, Precision measures how often a self-reported 'success' was truly successful. Recall measures how many actual successes were correctly identified by the agent.
• It provides a more nuanced view than simple accuracy, especially when agent failures (the 'positive' class in an error analysis) are rare but critical events.

Precision measures the correctness of a model's positive predictions. It answers the question: "Of all the instances the model labeled as positive, what percentage were actually correct?"

• Formula: Precision = True Positives / (True Positives + False Positives)
• High Precision indicates a low rate of false alarms. It is critical in scenarios where the cost of a false positive is high, such as spam detection (labeling a legitimate email as spam) or fraud screening.
• Trade-off: Optimizing for precision alone can lead to a model that is overly conservative, missing many true positives (low recall).

Recall (also called Sensitivity or True Positive Rate) measures the completeness of a model's positive predictions. It answers: "Of all the actual positive instances, what percentage did the model successfully find?"

• Formula: Recall = True Positives / (True Positives + False Negatives)
• High Recall indicates the model misses few positive cases. It is paramount in applications like medical diagnosis (missing a disease is costly) or search and retrieval.
• Trade-off: A model with very high recall may also generate many false positives, sacrificing precision.

A Confusion Matrix is a tabular visualization of a classification model's performance, breaking down predictions into four fundamental categories:

• True Positives (TP): Correctly predicted positive cases.
• True Negatives (TN): Correctly predicted negative cases.
• False Positives (FP): Incorrectly predicted positive cases (Type I error).
• False Negatives (FN): Incorrectly predicted negative cases (Type II error).

This matrix is the foundational data source for calculating precision, recall, accuracy, and the F1 Score. It provides a more nuanced view than a single accuracy metric, especially for imbalanced datasets.

Accuracy is the most intuitive classification metric, representing the proportion of total correct predictions (both positive and negative) out of all predictions.

• Formula: Accuracy = (True Positives + True Negatives) / Total Predictions
• While simple, accuracy can be highly misleading for imbalanced datasets. For example, a model that always predicts the majority class (e.g., 'not fraud' in a dataset with 99% non-fraud) will have 99% accuracy but be useless for detecting the critical minority class (fraud).
• The F1 Score is often preferred over accuracy when class distribution is skewed and the positive class is of primary interest.

The Receiver Operating Characteristic (ROC) Curve plots the True Positive Rate (Recall) against the False Positive Rate at various classification thresholds. The Area Under the Curve (AUC) provides a single scalar value summarizing the model's ability to discriminate between classes across all thresholds.

• An AUC of 1.0 represents a perfect classifier; 0.5 represents a classifier no better than random chance.
• While the F1 Score is threshold-dependent (requires choosing a single operating point), the AUC evaluates model performance across all possible thresholds. They are complementary: AUC gives an overall performance picture, while F1 reports performance at a chosen business-relevant threshold.

The Fβ Score is a generalization of the F1 Score that allows you to assign different weights to precision and recall using a factor β.

• Formula: Fβ = (1 + β²) * (Precision * Recall) / (β² * Precision + Recall)
• β > 1 weights recall more heavily than precision (e.g., F2 Score).
• β < 1 weights precision more heavily than recall (e.g., F0.5 Score).
• The standard F1 Score is the specific case where β = 1, treating precision and recall with equal importance. The Fβ Score provides flexibility when the business cost of false positives and false negatives is not symmetrical.