Confusion Matrix

Accuracy is the proportion of total predictions that the model got correct. It is calculated as (True Positives + True Negatives) / Total Predictions.

• Use Case: Provides a high-level overview of model performance when the class distribution is balanced.
• Limitation: Can be misleading for imbalanced datasets. For example, a model that always predicts the majority class in a dataset with 95% negative examples will have 95% accuracy but fails completely on the positive class.
• Formula: (TP + TN) / (TP + TN + FP + FN)

Precision (or Positive Predictive Value) measures the model's exactness when it makes a positive prediction. It answers: "Of all the records the model labeled as a match, what fraction were actually matches?"

• Focus: Quality of positive predictions. High precision means the model's matches are trustworthy.
• Critical For: Entity resolution tasks where a false match (merging two different entities) has high downstream costs, such as in customer data integration or financial compliance.
• Formula: TP / (TP + FP)

Recall (or Sensitivity, True Positive Rate) measures the model's completeness in finding all actual positives. It answers: "Of all the true matches that exist, what fraction did the model successfully retrieve?"

• Focus: Coverage of the positive class. High recall means the model misses few true matches.
• Critical For: Tasks where missing a true match is costly, such as fraud detection (missing a fraudulent transaction) or medical diagnosis.
• Formula: TP / (TP + FN)

The F1 Score is the harmonic mean of Precision and Recall. It provides a single metric that balances the trade-off between them.

• Use Case: The preferred metric when you need a single number to compare models, especially with imbalanced class distributions. It penalizes models that are strong in one metric (e.g., high precision) but weak in the other (e.g., low recall).
• Interpretation: An F1 score of 1 represents perfect precision and recall, while 0 represents the worst.
• Formula: 2 * (Precision * Recall) / (Precision + Recall)

Specificity (True Negative Rate) measures the proportion of actual negatives correctly identified. False Positive Rate (FPR) is its complement (1 - Specificity).

• Specificity: Answers "Of all the true non-matches, what fraction did the model correctly leave unlinked?" Crucial for understanding how well the model avoids creating false links.
• False Positive Rate: The probability that a true negative is incorrectly classified as positive. This is critical for evaluating the operational burden of a system—a high FPR means many record pairs require manual review.
• Formulas:
  • Specificity: TN / (TN + FP)
  • FPR: FP / (FP + TN)

ROC-AUC (Receiver Operating Characteristic - Area Under Curve) evaluates the model's performance across all classification thresholds. It plots the True Positive Rate (Recall) against the False Positive Rate.

• ROC-AUC Interpretation: A value of 0.5 indicates a random classifier; 1.0 indicates a perfect classifier. Useful for comparing models on balanced datasets.

Precision-Recall AUC plots Precision against Recall and is often more informative for imbalanced datasets common in entity resolution (where true matches are rare). It focuses directly on the performance of the positive class.

Precision measures the correctness of a system's positive predictions. For entity resolution, it answers: 'Of all the record pairs the system linked as matches, what proportion were actually matches?'

• Formula: Precision = True Positives / (True Positives + False Positives)
• High Precision indicates a low rate of false merges, meaning the system is conservative and makes few incorrect linkage decisions. This is critical in domains like healthcare or finance where merging incorrect records has severe consequences.
• Trade-off: Often inversely related to Recall. A system can achieve 100% precision by only linking pairs it is absolutely certain about, but this would miss many true matches.

Recall (or Sensitivity) measures the system's ability to find all the actual matches. It answers: 'Of all the true matching record pairs that exist, what proportion did the system successfully retrieve?'

• Formula: Recall = True Positives / (True Positives + False Negatives)
• High Recall indicates a low rate of missed matches, meaning the system is comprehensive. This is vital for tasks like customer deduplication where missing a duplicate can lead to operational inefficiencies.
• Trade-off: A system can achieve 100% recall by linking every possible pair, but this would create a massive number of false merges, destroying precision.

The F1-Score is the harmonic mean of Precision and Recall, providing a single metric that balances both concerns. It is especially useful when you need a single number to compare models and when there is an uneven class distribution.

• Formula: F1-Score = 2 * (Precision * Recall) / (Precision + Recall)
• Interpretation: The F1-Score ranges from 0 to 1, where 1 represents perfect precision and recall. It penalizes extreme values in either metric. A model with high precision but low recall (or vice versa) will have a middling F1-Score.
• Use Case: The standard F1-Score weighs precision and recall equally. Variants like Fβ-Score allow you to weight recall (β > 1) or precision (β < 1) as more important for your specific business objective.

Accuracy is the proportion of all predictions (both positive and negative) that are correct. It is the most intuitive but often misleading metric for imbalanced classification tasks like entity resolution.

• Formula: Accuracy = (True Positives + True Negatives) / Total Predictions
• Pitfall in ER: In a typical entity resolution task, the number of non-matching pairs (True Negatives) is astronomically larger than the number of matching pairs. A naive system that simply declares everything a non-match could achieve 99.99% accuracy while being completely useless, as it would have 0% recall.
• When to Use: Accuracy is a valid metric only when the class distribution (matches vs. non-matches) is roughly balanced, which is rare in real-world ER scenarios.

The False Positive Rate measures how often the system incorrectly labels a non-match as a match. It is a key metric for understanding the cost of errors in systems where false merges are expensive.

• Formula: FPR = False Positives / (False Positives + True Negatives)
• Context: In the confusion matrix, it is the proportion of all actual negatives that were incorrectly classified as positive. A low FPR is synonymous with high specificity.
• Business Impact: A high FPR in entity resolution means operational clean-up is required to undo incorrect merges, which can be labor-intensive and damage data integrity.

A ROC Curve is a graphical plot that illustrates the diagnostic ability of a binary classifier system (like an entity resolution matcher) as its discrimination threshold is varied. It plots the True Positive Rate (Recall) against the False Positive Rate at various threshold settings.

• Key Element: The Area Under the Curve (AUC) summarizes the curve's information into a single value between 0 and 1. An AUC of 1.0 represents a perfect classifier; 0.5 represents a classifier no better than random guessing.
• Use in ER: By analyzing the ROC curve, engineers can select the optimal threshold for their matching model based on the business's tolerance for false positives versus false negatives. It shows the full trade-off space between recall and FPR.