F1 Score

The F1 Score is calculated as 2 * (Precision * Recall) / (Precision + Recall) . Unlike a simple arithmetic mean, the harmonic mean severely penalizes extreme disparities between precision and recall. A model that achieves high precision by finding only one easy entity will have a very low F1 Score, forcing a balanced optimization.

Accuracy is a misleading metric for clinical NER because the vast majority of tokens in a medical record are 'O' (Outside any entity). A model that predicts 'O' for every token would have high accuracy but an F1 Score of zero. The F1 Score ignores true negatives, focusing exclusively on the model's ability to correctly identify the rare, critical clinical entities like diseases, medications, and procedures.

The F1 Score quantifies the trade-off between two competing errors:

• Precision: Of all the entities the model predicted, how many were correct? High precision means fewer false alarms.
• Recall: Of all the actual entities in the text, how many did the model find? High recall means fewer missed diagnoses. A high F1 Score confirms the model is both accurate in its predictions and thorough in its detection.

For multi-class NER problems, F1 must be aggregated across all entity types:

• Micro-F1: Calculates metrics globally by counting total true positives, false negatives, and false positives. It favors performance on frequent entities.
• Macro-F1: Calculates the F1 for each entity type independently and takes the unweighted mean. It treats rare entities like 'Adverse Event' as equally important as common ones like 'Drug'.
• Weighted-F1: A mean weighted by each class's support (number of true instances), balancing the two approaches.

The F1 Score's value depends heavily on the evaluation script's boundary matching rules:

• Strict/Exact Match: The predicted entity span must perfectly align with the gold-standard annotation. A single token offset counts as a false positive and a false negative.
• Lenient/Overlap Match: Any overlap between the predicted and true span is considered a true positive. This is often used in medication extraction where capturing the core drug name is more critical than exact dosage boundaries.

While powerful, the F1 Score is not a complete picture of model performance. It treats precision and recall as equally important, which may not be true for all clinical use cases. A screening application might prioritize recall (finding every possible case) over precision, while an automated billing system might demand extremely high precision to avoid claim denials. Always evaluate F1 alongside raw precision and recall values.

Measures the exactness of a model. It answers: Of all the entities the model predicted as 'Medication', how many were actually correct? High precision is critical in clinical workflows to prevent alert fatigue caused by false positives.

• Formula: True Positives / (True Positives + False Positives)
• Clinical Context: A low precision in medication extraction means the system is flagging too many non-drug terms, wasting pharmacist review time.

Measures the completeness of a model. It answers: Of all the actual 'Disease' entities in the text, how many did the model successfully find? High recall is vital for safety to ensure no critical finding is missed.

• Formula: True Positives / (True Positives + False Negatives)
• Clinical Context: A low recall in pathology report extraction might miss a malignant diagnosis, posing a direct risk to patient safety.

In clinical Named Entity Recognition, 'O' (Outside) tags vastly outnumber entity tags like 'B-Drug'. Accuracy is a misleading metric here because a model that predicts 'O' for everything scores highly but is clinically useless. The F1 score is the standard metric because it ignores true negatives and focuses solely on the model's ability to correctly identify the rare positive entity classes.

When evaluating multi-class NER (e.g., drugs, diseases, procedures), the aggregation method matters.

• Micro-F1: Aggregates contributions of all classes globally. It is heavily influenced by the performance on frequent classes.
• Macro-F1: Calculates the F1 for each class independently and takes the unweighted mean. This penalizes the model more severely for poor performance on rare entities like 'Lab Value'.

The raw data structure from which the F1 score is derived. For sequence labeling, it visualizes the True Positives, False Positives, True Negatives, and False Negatives for each entity type. Analyzing the confusion matrix reveals specific systematic errors, such as the model frequently confusing 'Procedure' with 'Lab Test'.

The strictness of evaluation drastically changes the F1 score.

• Exact Match: Requires the predicted span boundaries to perfectly align with the gold standard annotation.
• Relaxed Match: Often allows partial overlap. A prediction of 'severe chest pain' for a gold label of 'chest pain' would be a false positive in exact match but a true positive in relaxed match. Clinical NLP typically reports strict F1 to ensure boundary integrity.