Knowledge Base Distillation

A large, high-fidelity teacher model (e.g., a full cross-encoder over the UMLS Metathesaurus) generates soft probability distributions over candidate entities. A smaller student model is trained to mimic these distributions, learning a dense, compressed mapping from mention to concept without storing the full ontology graph. This transfers the teacher's disambiguation capability into a lightweight architecture suitable for production inference.

Rather than traversing a graph or executing expensive lexical lookups, the distilled knowledge base is stored as a set of fixed-dimensional dense vectors in an approximate nearest neighbor (ANN) index. A clinical mention is encoded once by a bi-encoder, and the top-k candidate entities are retrieved via cosine similarity in microseconds. This collapses the multi-stage retrieval and ranking pipeline into a single, highly optimized vector search operation.

Standard contrastive learning treats all non-matching entities as equally negative. Distillation incorporates ontology structure into the loss function by weighting negatives based on semantic type distance or hierarchical proximity in SNOMED CT. A 'pneumonia' mention penalizes confusion with 'myocardial infarction' more heavily than confusion with 'bronchitis', preserving fine-grained clinical distinctions in the compressed representation.

A critical requirement is that the distilled model retains the calibrated probability estimates of the teacher. Post-distillation, temperature scaling or Platt scaling is applied to ensure that a predicted linking probability of 0.9 genuinely reflects a 90% chance of correctness. This is essential for downstream human-in-the-loop workflows where low-confidence predictions are routed for clinical review.

The compressed model must preserve the teacher's ability to predict NIL (no match) when a mention has no valid concept in the target ontology. A dedicated NIL embedding vector is learned during distillation, positioned in the embedding space to represent the decision boundary for 'out-of-knowledge-base' mentions. This prevents the dangerous error of forcibly grounding an unrecognized term to a false positive concept.

A full cross-encoder reranking pipeline over 100 candidates might require 50-100ms per mention. A distilled bi-encoder with an ANN index reduces this to < 1ms per mention on CPU, a 50-100x speedup. This makes real-time entity linking feasible for streaming clinical NLP workloads, such as processing live HL7 v2 messages or powering autocomplete in EHR documentation interfaces.

The foundational paradigm where a large, high-capacity teacher model (often a cross-encoder or full ontology index) generates soft probability distributions over all concepts. A compact student model is trained to mimic this distribution, capturing nuanced inter-concept relationships that hard labels miss.

• Soft labels transfer dark knowledge about concept similarity
• Student learns to replicate teacher confidence, not just the top-1 answer
• Enables a 10-50x reduction in model size with minimal accuracy degradation

Distillation compresses a sparse, high-dimensional ontology into a dense vector space where semantically similar concepts cluster together. Each concept is represented by a compact embedding (e.g., 256-768 dimensions) rather than a one-hot encoding over millions of classes.

• Reduces memory footprint from gigabytes to megabytes
• Enables approximate nearest neighbor (ANN) search for sub-millisecond retrieval
• Preserves ontological distance: similar diseases remain close in latent space

A technique adapted from image generation where the gradient of the teacher's output distribution guides the student's learning. Instead of matching final predictions, the student learns to match the score function—the gradient of the log-probability with respect to the input.

• Captures the teacher's decision boundary geometry
• More robust to calibration errors in the teacher
• Particularly effective for distilling graph-based ontology reasoners into feed-forward networks

Standard distillation can produce embeddings that collapse distinct but confusable concepts. Ontology-aware negative sampling uses the source knowledge graph's hierarchical structure to select hard negatives—concepts that are siblings or share a parent in the ontology tree.

• Prevents embedding collapse between SNOMED CT siblings like 'Type I Diabetes' and 'Type II Diabetes'
• Uses graph distance to weight negative sample difficulty
• Improves disambiguation accuracy by 3-7% on fine-grained clinical distinctions

A structural distillation method where the student model is initialized as a subset of the teacher's transformer layers. During training, intermediate layers are progressively removed while layer-wise distillation losses ensure the remaining layers compensate for the lost capacity.

• Starts with a full teacher, ends with a 2-4 layer student
• Preserves the teacher's attention patterns in compressed form
• Often combined with hidden-state matching for deeper knowledge transfer

A joint optimization that simultaneously distills knowledge and quantizes weights to 8-bit or 4-bit integers. The student model is trained with simulated quantization noise, ensuring that post-training compression does not degrade the distilled concept representations.

• Produces models deployable on CPU-only inference servers
• Maintains < 0.5% accuracy loss at INT8 precision
• Critical for edge deployment in HIPAA-compliant on-premise environments