Skill-Based Routing

Tasks are dynamically routed to reviewers based on their demonstrated competency in specific clinical domains or error categories. The system maintains a skills matrix that maps each reviewer's certified specialties—such as cardiology, oncology, or radiology—and their historical performance metrics.

• Reviewers receive cases aligned with their board certifications and subspecialties
• The system tracks per-category accuracy rates to refine future assignments
• New reviewers are initially restricted to low-complexity tasks until proficiency thresholds are met
• Example: A complex chemotherapy regimen extraction is routed only to oncology-certified reviewers with >95% historical accuracy on medication entities

Every potential model failure mode is categorized within a structured error taxonomy, and reviewers are evaluated against their ability to detect and correct specific error classes. This granular mapping enables precise matching between task requirements and reviewer capabilities.

• Error classes include: boundary errors, negation misclassification, entity normalization failures, and temporal relation errors
• Reviewers develop specialized expertise in correcting recurring failure patterns
• The taxonomy evolves as new model weaknesses are identified through discrepancy analysis
• Example: A reviewer with high accuracy on negation detection is preferentially assigned tasks flagged for uncertainty in affirmed vs. negated findings

The routing engine continuously monitors queue depth, reviewer availability, and service level agreements to optimize task distribution. It prevents bottlenecks by redistributing tasks when reviewers are overloaded or unavailable.

• Real-time capacity tracking prevents any single reviewer from becoming a critical path blocker
• Urgent cases with approaching SLA deadlines are escalated and rerouted to available qualified reviewers
• The system applies fair queuing algorithms to distribute work evenly across equally skilled reviewers
• Example: If a cardiology specialist logs off, their pending high-priority tasks are immediately reassigned to the next available cardiology-certified reviewer

Reviewer proficiency is continuously measured against a golden dataset—a meticulously curated set of ground truth clinical data. Performance on these benchmark tasks determines skill levels and identifies reviewers who require recalibration.

• Golden tasks are silently injected into production queues to monitor ongoing accuracy
• Inter-annotator agreement scores against the gold standard are tracked per error category
• Reviewers whose accuracy drifts below threshold are automatically flagged for targeted retraining
• Example: A reviewer whose medication dosage extraction accuracy drops below 90% is temporarily restricted from medication-related tasks until they complete a norming session

Complex or ambiguous cases that exceed a reviewer's skill level are automatically escalated through a predefined adjudication hierarchy. This ensures that difficult clinical judgments reach the most senior and specialized experts.

• Escalation paths are defined per clinical domain and error severity
• Adjudication workflows resolve conflicts when two reviewers disagree on a finding
• Senior reviewers can override routing rules to accept challenging cases outside their primary specialty
• Example: A disputed pathology finding escalates from a general reviewer to a pathology specialist, then to a medical director for final adjudication

Every correction a reviewer makes feeds back into the system to refine both the AI model and the reviewer's skill profile. This closed-loop system ensures continuous improvement in routing precision and clinical accuracy.

• Corrections tagged with error taxonomy codes inform targeted model retraining via active learning loops
• Reviewer skill profiles are updated in real-time based on correction patterns and peer review outcomes
• The system identifies emerging error patterns and proactively routes similar cases to reviewers with proven correction accuracy
• Example: A spike in LOINC code mapping errors triggers retraining of the normalization model while routing future LOINC tasks to reviewers with the highest mapping accuracy