Confidence Score

Confidence scores are continuous values (e.g., 0.0 to 1.0) that must be converted into discrete actions. Thresholding is the process of setting a cutoff.

• High Threshold (e.g., 0.95): Maximizes precision but sends more mappings for manual review.
• Low Threshold (e.g., 0.70): Maximizes recall but risks introducing incorrect automated mappings.
• Deterministic logic applies a hard cutoff, while probabilistic systems may factor in the cost of an error.

A well-calibrated model outputs scores that match observed frequencies. If a model assigns a score of 0.9 to 100 mappings, exactly 90 should be correct.

• Sharpness refers to how concentrated scores are near 0 or 1.
• Overconfidence is a common failure mode in deep neural networks, where scores cluster near 1.0 even for wrong answers.
• Temperature Scaling is a post-hoc technique used to recalibrate these raw logits.

The source of a score drastically changes its meaning in ontology alignment.

• Lexical Confidence: Derived from string similarity (e.g., edit distance, TF-IDF). High score means the words match closely.
• Semantic Confidence: Derived from graph embeddings or BERT-based models. High score means the contextual meaning aligns.
• A mapping can have high lexical confidence but low semantic confidence if terms are spelled similarly but mean different things (e.g., 'cold' as temperature vs. illness).

When mapping a complex clinical phrase to a single code, the score aggregates uncertainty from multiple sub-tasks.

• Named Entity Recognition (NER) confidence: How sure is the model that 'myocardial infarction' is a condition?
• Negation Detection confidence: How sure is the model that the condition is absent?
• Linking confidence: How sure is the model that the extracted entity maps to SNOMED CT code 22298006?
• The final score is often a product or minimum of these sub-scores.

Confidence scores are the primary trigger for human-in-the-loop validation workflows.

• Auto-accept: Scores above a high threshold bypass human review to maximize throughput.
• Review Queue: Scores in a middle band are sent to clinical informaticists for adjudication.
• Auto-reject: Scores below a low threshold are discarded to prevent noise.
• This triage ensures that expert effort is focused only on ambiguous cases, optimizing operational cost.

A confidence score without provenance is a liability in regulated environments. The score must be stored alongside its metadata.

• Model Version: Which specific model generated the score?
• Timestamp: When was the mapping created?
• Justification: Was the score based on lexical match, semantic similarity, or a hybrid?
• This audit trail allows for retrospective analysis if a specific model version is found to have a systematic bias or calibration drift.