Error Taxonomy

Each error category must define a distinct, non-overlapping failure mode. Ambiguity between classes forces reviewers into subjective judgment, degrading Inter-Annotator Agreement (IAA) and introducing noise into the training data.

• Boundary Clarity: A 'Wrong Drug' error must be unambiguously distinct from a 'Wrong Dosage' error.
• Decision Trees: Complex distinctions often require a brief decision tree in reviewer guidelines to ensure consistent classification.
• Negative Impact: Overlapping categories are the primary cause of Reviewer Drift and unreliable performance dashboards.

The taxonomy must account for every possible failure the model can generate, including a critical 'Other' or 'Null' category. An unclassifiable error is a lost learning opportunity and a source of reviewer frustration.

• The 'Other' Trap: The 'Other' category must be monitored closely; if it grows beyond 5% of total volume, it signals the need for a taxonomy update to capture a new, systematic failure mode.
• Edge Cases: Ensure coverage for rare but critical failures, such as Negation errors ('no evidence of' vs. 'evidence of') or Temporality errors (confusing historical and current conditions).

Categories must be specific enough to direct a concrete model improvement, not just a vague observation. A tag like 'Extraction Error' is useless; 'Incorrect Span Detection' or 'Hallucinated Lab Value' provides a direct path to remediation.

• Retraining Directives: Each tag should map to a specific data augmentation or fine-tuning strategy. A 'Laterality Confusion' tag (left vs. right) directly informs the creation of synthetic counter-examples.
• Granularity Balance: Avoid overly atomized categories that overwhelm reviewers. A taxonomy with 100+ classes often suffers from low IAA and high cognitive load.

Organize error types in a parent-child hierarchy to support both high-level trend analysis and granular debugging. A reviewer can quickly select a broad category, then drill down to a specific sub-type.

• Top-Level Classes: Broad domains like 'Clinical Entity Error', 'Relational Error', 'Assertion Error'.
• Sub-Classes: Under 'Clinical Entity Error', specific types like 'Medication Name', 'Dosage', 'Frequency', 'Route'.
• Dashboard Utility: This structure allows a CTO to view a high-level 'Relational Error' rate while an NLP engineer drills into the specific 'Incorrect Subject-Object Linking' sub-type.

An error's classification should depend solely on the model's output relative to the ground truth, not on the reviewer's speculation about the cause of the error. Do not confuse error type with root cause.

• Error vs. Cause: The error is 'Incorrect Laterality' (right vs. left). The root cause might be a poor attention mechanism or biased training data. The taxonomy captures the what, not the why.
• Reviewer Focus: This constraint simplifies the reviewer's task from a complex diagnostic exercise to a direct comparison, reducing Cognitive Load and speeding up the Review Cadence.

The taxonomy itself is a living artifact that must be version-controlled. Any change to the schema—adding, merging, or splitting categories—must be tracked to ensure historical performance comparisons remain valid.

• Schema as Code: Treat the taxonomy definition as a configuration file in a git repository, with semantic versioning (e.g., v2.1.0).
• Backwards Compatibility: When a category is deprecated, map it to its successor to prevent data loss in long-running longitudinal studies.
• Retrospective Re-Labeling: A major version change may require a one-time re-labeling of a Golden Dataset to establish a new performance baseline.