Dictionary-Based NER

Unlike statistical models that predict entity boundaries probabilistically, dictionary-based NER performs exact string matching against a pre-compiled knowledge base. This guarantees that every extracted entity is a verified clinical concept.

• Zero hallucination risk: Only terms present in the gazetteer are extracted
• Auditable logic: Every match can be traced to a specific dictionary entry
• No training data required: Operates immediately upon dictionary compilation

Large clinical gazetteers containing millions of terms require efficient data structures to avoid linear scanning. Prefix trees (tries) organize terms by shared character sequences, enabling O(n) lookup where n is the length of the input text.

• Aho-Corasick automaton: Simultaneously matches multiple dictionary terms in a single pass
• Memory-efficient variants: Radix trees and compressed tries reduce storage overhead
• Token-level indexing: Pre-tokenized tries align with clinical NLP pipeline outputs

The quality of dictionary-based NER depends entirely on the comprehensiveness and accuracy of the underlying gazetteer. Curating clinical dictionaries requires domain expertise and ongoing maintenance.

• Source vocabularies: RxNorm for medications, SNOMED CT for disorders, LOINC for lab tests
• Synonym expansion: Each concept includes brand names, abbreviations, and common misspellings
• Version drift management: Dictionaries must synchronize with ontology release cycles

Dictionary matching must handle tokenization mismatches between the gazetteer entries and the raw clinical text. Multi-word terms like 'congestive heart failure' require span-level matching.

• Longest-match priority: Resolves overlapping matches by selecting the most specific term
• Case-insensitive matching: Normalizes text to handle inconsistent clinical documentation
• Punctuation normalization: Strips or standardizes hyphens, slashes, and apostrophes before matching

Raw dictionary matching produces high recall but can generate false positives. Post-processing filters apply contextual rules to suppress incorrect matches.

• Negation detection: Suppresses entities in negated contexts using algorithms like NegEx
• Section scoping: Limits certain entity types to relevant clinical note sections
• Temporal constraint: Filters historical conditions from active problem lists

Dictionary-based NER excels at precision but misses novel or misspelled terms. Hybrid architectures combine dictionary matching with machine learning models to balance precision and recall.

• Dictionary features: Gazetteer matches serve as input features for statistical models
• Candidate pre-filtering: Dictionary lookup narrows the search space before ML classification
• Ensemble voting: Dictionary and model predictions are merged with configurable confidence thresholds

A classic string-searching algorithm that efficiently matches a large dictionary of terms against input text in a single pass. Unlike naive sequential scanning, it constructs a finite state machine from the dictionary, enabling linear-time complexity relative to text length. This makes it ideal for matching thousands of clinical terms simultaneously without performance degradation. The algorithm outputs all matches, including overlapping spans, which must be resolved through longest-match or leftmost-match heuristics in a clinical NER pipeline.

A tree-like data structure where each node represents a character, and paths from root to leaf spell out dictionary terms. Tries enable O(k) lookup time where k is the term length, independent of dictionary size. In clinical NER, tries support incremental matching—as the tokenizer advances through text, the trie state is updated character-by-character, emitting entity spans when terminal nodes are reached. This structure naturally supports prefix-based autocompletion and fuzzy matching extensions.

The process of constructing, cleaning, and maintaining the term list that drives dictionary-based NER. Raw clinical vocabularies contain noisy synonyms—abbreviations, historical terms, and overly broad strings that cause false positives. Curation involves:

• Stop-phrase filtering to remove generic terms like 'patient' or 'report'
• Abbreviation expansion to add common clinical shorthand
• Ambiguity resolution to flag terms that map to multiple concepts
• Frequency-based pruning to remove rarely used synonyms that degrade precision

A disambiguation rule applied when dictionary matching produces overlapping entity spans. The principle selects the longest matching term, assuming it carries the most specific clinical meaning. For example, 'type 2 diabetes mellitus' is preferred over 'diabetes' when both match. This heuristic reduces fragmentation and improves semantic accuracy, but can fail with nested entities where both the broader and narrower terms are clinically significant. Advanced implementations use priority scores or ontology depth to resolve conflicts.