Confusion Matrix

Precision answers: Of all products flagged as defective, how many were actually defective? It is the ratio of True Positives (TP) to the sum of True Positives and False Positives (FP).

• Formula: TP / (TP + FP)
• Business Impact: Low precision means a high False Reject Rate (FRR). The system is scrapping or reworking good products, directly eroding yield and increasing manufacturing costs.
• Optimization Goal: Maximize precision when the cost of a false alarm (scrapping a good part) is high.

Recall answers: Of all actually defective products, how many did the system catch? It is the ratio of True Positives (TP) to the sum of True Positives and False Negatives (FN).

• Formula: TP / (TP + FN)
• Business Impact: Low recall means a high Escape Rate. Defective products are reaching customers, creating warranty liabilities, brand damage, and potential safety recalls.
• Optimization Goal: Maximize recall when the cost of missing a defect (shipping a bad part) is catastrophic.

The F1 Score is the harmonic mean of Precision and Recall, providing a single balanced metric when you need to optimize both simultaneously. It penalizes extreme trade-offs more severely than an arithmetic mean.

• Formula: 2 * (Precision * Recall) / (Precision + Recall)
• Use Case: Essential for evaluating models on imbalanced datasets where the number of non-defective products vastly outnumbers defective ones. A high F1 score indicates the model performs well on both catching defects and avoiding false alarms.
• Limitation: The F1 score treats precision and recall as equally important, which may not align with specific business cost functions.

Specificity answers: Of all good products, how many did the system correctly pass? It is the ratio of True Negatives (TN) to the sum of True Negatives and False Positives (FP).

• Formula: TN / (TN + FP)
• Business Impact: High specificity means the system is efficient and does not create unnecessary rework loops. It is the direct complement to the False Positive Rate (FPR), which is 1 - Specificity.
• Optimization Goal: Critical for high-volume production lines where even a 1% false positive rate can overwhelm human rework stations with thousands of incorrectly rejected units per shift.

Accuracy is the ratio of all correct predictions (both defective and non-defective) to the total number of inspections. It answers: What percentage of all decisions did the system get right?

• Formula: (TP + TN) / (TP + TN + FP + FN)
• Critical Warning: Accuracy is a dangerously misleading metric for defect inspection. In a process with a 0.1% defect rate, a model that blindly classifies every product as 'good' achieves 99.9% accuracy while catching zero defects.
• Best Practice: Never use accuracy in isolation for imbalanced manufacturing datasets. Always pair it with Precision, Recall, and the F1 Score.

The MCC is a balanced measure of binary classification quality that considers all four confusion matrix quadrants. It produces a score between -1 (total disagreement) and +1 (perfect prediction), with 0 representing random guessing.

• Formula: ((TP * TN) - (FP * FN)) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))
• Advantage over F1: Unlike the F1 score, the MCC accounts for True Negatives, making it a more holistic and reliable statistical measure for evaluating models on imbalanced datasets.
• Interpretation: A high MCC score is only achieved if the model performs well across all four categories (TP, TN, FP, FN), making it the gold standard for single-number evaluation.

The ratio of true positives to total predicted positives (TP / (TP + FP)). In manufacturing, precision answers: Of all the parts flagged as defective, how many were actually bad? Low precision leads to a high False Reject Rate (FRR), causing unnecessary scrap and rework of perfectly conforming products. A precision-focused system is critical when the cost of investigating a false alarm is extremely high.

The ratio of true positives to total actual positives (TP / (TP + FN)). Recall answers: Of all the truly defective parts, how many did we catch? Low recall indicates a high Escape Rate, meaning defective products are reaching customers. A recall-focused system is non-negotiable in safety-critical industries like automotive or medical device manufacturing where a missed defect can be catastrophic.

The harmonic mean of precision and recall, providing a single balanced metric when you need to optimize both simultaneously. The F1 score penalizes extreme imbalances between the two. It is the preferred evaluation metric when you have an uneven class distribution—common in defect detection where defective parts are rare—and you cannot easily assign a dollar value to false positives versus false negatives.

The ratio of true negatives to total actual negatives (TN / (TN + FP)). Specificity answers: Of all the good parts, how many did we correctly pass? This metric is the direct counterpart to the False Reject Rate (FRR = 1 - Specificity). High specificity is the primary economic driver in high-volume production, ensuring that the inspection system does not become a bottleneck by rejecting an excessive number of conforming units.

The probability that a conforming part is incorrectly flagged as defective. Calculated as FP / (FP + TN). This metric directly quantifies the Type I Error of the inspection system. A high FPR erodes trust in the automated system, forcing operators to manually re-inspect flagged parts and undermining the return on investment of the computer vision deployment.

The probability that a true defective part is missed by the inspection system. Calculated as FN / (FN + TP). This is the Type II Error or miss rate, and it represents the most dangerous failure mode in quality assurance. The FNR directly defines the Escape Rate, which is the key performance indicator for consumer safety and regulatory compliance in industries like pharmaceutical manufacturing.