Confidence Scoring

A well-calibrated confidence score reflects the true likelihood of correctness. If a model assigns a confidence of 0.9 to 100 predictions, exactly 90 should be correct. Platt scaling and isotonic regression are common post-hoc calibration methods that correct overconfident neural networks. Without calibration, a raw softmax probability is often an unreliable measure of true uncertainty.

Confidence scores enable decision triage in clinical pipelines:

• High confidence (>0.95): Auto-commit the negation or assertion to the structured record.
• Low confidence (<0.70): Route to a human abstractor for review.
• Ambiguous band (0.70–0.95): Flag for secondary algorithmic review or ensemble voting. This architecture balances automation rates against clinical risk tolerance.

Beyond a single scalar, rigorous confidence scoring distinguishes between:

• Aleatoric uncertainty: Inherent noise in the data, such as genuinely ambiguous hedging language.
• Epistemic uncertainty: Model uncertainty due to lack of training data, reducible with more examples. Monte Carlo Dropout and Deep Ensembles approximate Bayesian inference to estimate both types, providing richer signals than a point probability.

In negation and uncertainty detection, confidence can be assigned at different granularities:

• Token-level: Each word receives a probability of being a negation cue or within scope.
• Span-level: The entire clinical entity span receives a single factuality score. Span-level scoring is often more actionable for downstream FHIR mapping, as it directly answers: 'Is this specific diagnosis negated?'

A robust confidence scoring system must recognize when an input falls outside its training distribution. A model fine-tuned on radiology reports may assign high confidence to an unfamiliar surgical note phrasing, a phenomenon known as overconfidence in the unknown. Techniques like Mahalanobis distance on feature embeddings or energy-based models help flag these inputs for mandatory human review, preventing silent failures.

Confidence scores should be dynamic, not static. When a human reviewer corrects a low-confidence prediction, that correction becomes a supervised signal for fine-tuning. This closed loop:

• Improves calibration on edge cases over time.
• Identifies systematic overconfidence in specific linguistic patterns, such as pseudo-negation ('not only pneumonia but...').
• Builds a growing dataset of high-value ambiguous cases.

The user experience layer that consumes confidence scores to prioritize clinical review queues. When a model assigns a low confidence score to a negation or uncertainty prediction, the interface surfaces that specific span for manual adjudication.

• Threshold routing: Cases below 0.85 confidence are flagged for review
• Pre-annotation: High-confidence predictions are auto-accepted, reducing reviewer fatigue
• Feedback loops: Reviewer corrections are captured to refine future confidence calibration

The classification label assigned to a clinical named entity indicating whether the concept is present, absent, or uncertain. Confidence scoring provides the probabilistic weight behind each assertion status assignment.

• Present: High confidence that the finding is affirmed
• Absent: High confidence that negation cues are active
• Uncertain: Low confidence or hedging detected, requiring review

The assertion status is the downstream consumer of the confidence score, determining how structured data populates the patient record.

The evaluation metric measuring the proportion of correctly identified negated findings out of all findings flagged as negated. Confidence thresholds directly tune precision-recall tradeoffs.

• Raising the confidence threshold increases precision but risks missing true negations
• Lowering the threshold improves recall but introduces false positives
• Clinical safety often demands high precision to avoid incorrectly ruling out conditions

Deterministic and probabilistic logic systems that consume confidence scores alongside extracted entities to verify data accuracy. These engines apply domain-specific rules to catch contradictions.

• Cross-referencing: A low-confidence negation of 'pneumonia' is cross-checked against a high-confidence affirmative mention elsewhere
• Temporal consistency: Confidence decay models flag findings where certainty drops over time without explicit resolution
• Hard stops: Rules can block low-confidence extractions from populating problem lists until human review occurs

The proportion of actual negated or uncertain findings that the system fails to detect. This is the critical safety metric that confidence scoring calibration aims to minimize.

• A missed negation can cause a condition to be incorrectly attributed to a patient
• Confidence score distributions reveal model blind spots where uncertainty is systematically underestimated
• Calibration curves plot predicted confidence against observed accuracy to identify overconfidence in error-prone contexts

A deep learning approach where a contiguous sequence of tokens is classified as negated or uncertain as a single unit. Confidence scores are assigned to the entire span, not individual tokens.

• Contrasts with token-level labeling where each word gets an independent score
• Span-level confidence provides a more clinically meaningful signal: 'the entire phrase no evidence of malignancy is negated with 0.94 confidence'
• Enables more coherent human review by highlighting complete semantic units rather than fragmented tokens