Supervised Fine-Tuning (SFT)

SFT requires a high-quality labeled dataset of clinical documents where each example is paired with a ground-truth category. For medical document classification, this means radiologists or HIM professionals must annotate thousands of reports with labels like 'Discharge Summary', 'Pathology Report', or 'Operative Note'. The model learns to map input text patterns to these specific labels through supervised learning. The quality and representativeness of this labeled data directly determines downstream accuracy.

The pre-trained model's generic output layer is replaced with a new classification head initialized with random weights. For a document classification task with 15 clinical categories, this head is a linear layer with 15 output neurons. During SFT, the entire model—or just this head—is trained to minimize the cross-entropy loss between predicted class probabilities and the true labels. This architectural modification transforms a general-purpose language model into a specialized medical document router.

A critical challenge in SFT is catastrophic forgetting—the model's tendency to overwrite general linguistic knowledge learned during pre-training when adapting to a narrow clinical domain. Techniques to mitigate this include:

• Learning rate annealing: Using a smaller learning rate to make gradual weight updates
• Mixed-task training: Interleaving general language examples with clinical data
• Elastic weight consolidation: Penalizing large changes to parameters important for prior tasks Without these safeguards, a model fine-tuned on pathology reports may lose its ability to understand general medical terminology.

Full fine-tuning updates all model parameters, which is computationally expensive for large language models. Parameter-efficient fine-tuning (PEFT) methods like LoRA (Low-Rank Adaptation) freeze the pre-trained weights and inject small, trainable rank-decomposition matrices into attention layers. This reduces trainable parameters by over 99% while maintaining comparable performance. For healthcare deployments, PEFT enables rapid adaptation to new document types without provisioning massive GPU clusters.

SFT performance is measured on a held-out test set of clinical documents never seen during training. Key metrics include:

• Macro F1-score: Averages performance across all document classes, ensuring rare types like 'Genetic Report' are weighted equally with common ones like 'Progress Note'
• Confusion matrix analysis: Identifies systematic errors, such as misclassifying 'Consult Note' as 'History & Physical'
• Confidence calibration: Ensures the model's probability outputs reflect true likelihood of correctness for downstream routing decisions

A model fine-tuned on documents from one healthcare system may degrade when deployed at another due to domain shift—differences in physician dictation styles, EHR templates, or regional abbreviation conventions. For example, a classifier trained on Epic-generated notes may struggle with Cerner-formatted documents. Continuous monitoring and periodic re-fine-tuning with institution-specific examples are required to maintain production accuracy above the 95% threshold typically demanded by clinical workflows.

The target architecture being adapted through SFT. A text classification model is a machine learning algorithm trained to automatically assign predefined category labels to unstructured clinical text documents. In medical document classification, this could mean routing a document to 'Pathology Report' or 'Discharge Summary' based on its content. SFT is the process that transforms a general-purpose language model into a specialized text classifier by training it on domain-specific, labeled examples.

A formal, hierarchical classification system defining the semantic categories of clinical documents. Before SFT can begin, a document type ontology must be defined to establish the target labels. This ontology structures categories like:

• Discharge Summaries
• Operative Notes
• Pathology Reports
• Radiology Reports The quality and granularity of this ontology directly determines the usefulness of the fine-tuned model's output.

A filtering mechanism that routes AI predictions with low probability scores to a manual review queue. After SFT, a model outputs a probability distribution over document types. Confidence thresholding ensures high accuracy for automated decisions by only accepting predictions above a set threshold (e.g., 0.95). Documents falling below this threshold are sent to an exception queue for human review, creating a safety net for ambiguous or edge-case documents.

A contrasting approach to SFT where a model categorizes documents into labels it has never explicitly seen during training. Zero-shot classification relies on semantic similarity between the label name and document content rather than labeled examples. While useful for rapid prototyping or when labeled data is scarce, SFT typically achieves higher accuracy on well-defined clinical document types because it learns the specific linguistic patterns of each category from curated examples.

A workflow design pattern where human auditors validate or correct AI-generated document classifications before finalization. This process is critical for SFT because it:

• Generates new labeled data from corrections, creating a feedback loop for continuous model improvement
• Catches edge cases the model was not exposed to during initial fine-tuning
• Maintains clinical safety by ensuring no misclassified document reaches a downstream system unchecked

The methodology of building AI systems around rigorous, quantitative benchmarking. For SFT, this means defining clear metrics before training begins:

• Precision and Recall per document type
• F1 Score as a harmonic mean
• Confusion Matrix analysis to identify which classes are frequently confused A held-out test set of labeled documents, never seen during training, provides an unbiased estimate of real-world performance.