F1 Score

The F1 score is calculated as the harmonic mean of precision and recall, not a simple arithmetic average. This mathematical property ensures the score is disproportionately penalized when either precision or recall is extremely low, making it ideal for imbalanced clinical datasets where true medication discrepancies are rare events. A model that achieves 100% recall by flagging every drug as a discrepancy would have near-zero precision and thus a near-zero F1 score.

Precision answers: Of all the discrepancies the model flagged, how many were actual errors?

• High precision means fewer false positives, directly reducing alert fatigue in clinical decision support systems.
• A model with 0.95 precision generates only 5 false alarms per 100 flags, preserving clinician trust.
• Low precision causes pharmacists to override or ignore alerts, a dangerous phenomenon linked to missed adverse drug events (ADEs).

Recall answers: Of all the true medication discrepancies that exist, how many did the model successfully identify?

• High recall minimizes omission errors, the most common type of unintentional discrepancy during care transitions.
• A model with 0.98 recall catches 98% of clinically significant drug omissions, directly preventing potential adverse drug events.
• Missing a true discrepancy (false negative) can result in a patient not receiving a critical medication, making recall a patient safety imperative.

In medication reconciliation, the F1 score is typically calculated at the medication entry level rather than the document level:

• Each extracted drug name, dosage, frequency, and route is compared against a gold-standard annotated dataset.
• Cohen's Kappa is often reported alongside F1 to measure inter-annotator agreement on the ground truth labels.
• A clinically acceptable F1 threshold is typically ≥ 0.90, reflecting the high-stakes nature of medication safety.

The F1 score is directly influenced by the model's confidence threshold—the probability cutoff above which a prediction is considered positive:

• Lowering the threshold increases recall (fewer missed discrepancies) but decreases precision (more false positives).
• Raising the threshold increases precision (fewer nuisance alerts) but decreases recall (more missed errors).
• Human-in-the-loop (HITL) systems use confidence thresholding to route low-confidence predictions for pharmacist review, optimizing the operational F1 score.

While the F1 score is the standard metric, it has limitations in medication reconciliation:

• It treats precision and recall as equally important, but in some clinical scenarios, recall (patient safety) may be weighted more heavily than precision (efficiency).
• The F-beta score generalizes F1 by introducing a beta parameter; an F2 score weights recall twice as heavily as precision.
• F1 does not capture the clinical severity of a missed discrepancy—missing a warfarin interaction is far more consequential than missing a multivitamin omission.

The fraction of AI-flagged medication discrepancies that are true positives—actual errors requiring clinical intervention. High precision minimizes alert fatigue by ensuring pharmacists are not wasting time on false alarms.

• Formula: True Positives / (True Positives + False Positives)
• Clinical Impact: A low precision means the system is crying wolf, flagging intentional therapeutic substitutions as errors.
• Example: If the AI flags 100 discrepancies but only 80 are real, the precision is 0.80.

The fraction of actual medication errors present in the chart that the AI successfully identified. High recall is critical for patient safety, ensuring no dangerous omission or duplication slips through.

• Formula: True Positives / (True Positives + False Negatives)
• Clinical Impact: A low recall means the system is missing real, potentially fatal, adverse drug events.
• Trade-off: Maximizing recall often decreases precision, generating more false positives.

A robust statistical measure that quantifies the level of agreement between two clinical pharmacists labeling the same medication list. Unlike raw percentage agreement, it corrects for the probability of random chance.

• Scale: -1 (perfect disagreement) to +1 (perfect agreement).
• Ground Truth Quality: A low Kappa score indicates ambiguous annotation guidelines, making it impossible to train a reliable F1-optimized model.
• Benchmark: A Kappa > 0.80 is generally considered strong agreement for clinical data.

A probabilistic gate that routes predictions for human review based on the model's certainty. Only discrepancies where the model's F1-optimized confidence score falls below a set boundary are sent to a pharmacist.

• High Threshold: Automates only the most obvious matches, maximizing precision but requiring heavy manual labor.
• Low Threshold: Automates more tasks but risks introducing errors into the Best Possible Medication History (BPMH).
• Goal: Find the threshold that maximizes the F1 score in the production workflow.

Deterministic post-processing rules that prevent the model from generating clinically impossible outputs, regardless of the F1 score on validation data. These are safety nets, not statistical metrics.

• RxNorm Validation: Rejects any generated drug name that does not exist in the RxNorm database.
• Dose Sanity Checks: Flags a dosage of '5000 mg' of a drug typically dosed in micrograms.
• Temporal Logic: Prevents a 'discontinued' date from occurring before a 'started' date.

A regular expression-based NLP algorithm that determines if a clinical finding is negated. Without this, an F1 score becomes meaningless because the system confuses 'patient denies chest pain' with an active symptom.

• Mechanism: Identifies negation phrases ('no evidence of', 'denies') within a window of the target term.
• Relevance: Crucial for distinguishing 'no medication changes' from an active list of changes.
• Evolution: Modern systems use contextual embeddings (like ClinicalBERT) that outperform rigid regex rules.