Supervised Fine-Tuning (SFT)

SFT provides the initial policy model for subsequent alignment techniques like RLHF or DPO. It teaches the model the basic format and style of desired responses before preference-based training refines quality and safety. Without a strong SFT base, reinforcement learning can struggle due to poor initial outputs.

• Prerequisite for RLHF: The model trained via SFT becomes the 'reference policy' in RLHF.
• Reduces Exploration Burden: Gives the RL policy a sensible starting point, making reward learning more stable.

The performance of SFT is directly constrained by the quality, diversity, and cleanliness of its labeled demonstration dataset. Common sources include:

• Human-written examples: Crafted by experts for specific tasks (e.g., code generation, customer service).
• Synthetic data: Generated by a more powerful model (e.g., GPT-4) and filtered for quality.
• Crowdsourced datasets: Large-scale collections like OpenAssistant or Anthropic's HH-RLHF.

Data contamination (leakage of test benchmarks into training) and label noise are critical failure modes.

SFT is the primary method for domain specialization. By training on a curated dataset, the model's knowledge and output distribution are shifted towards a specific use case.

Examples:

• Medical Chatbot: Fine-tune on doctor-patient Q&A transcripts.
• Code Assistant: Train on (natural language prompt, code solution) pairs.
• Legal Summarization: Use case law and corresponding summary documents.

This process updates all or a large subset of the model's parameters, which is computationally expensive but highly effective for narrow tasks.

A primary use of SFT is instruction tuning, where the model learns to interpret and execute a wide variety of natural language commands. The dataset consists of (instruction, ideal response) pairs covering diverse tasks.

Key Outcomes:

• Zero-shot generalization: The model can attempt novel tasks framed as instructions.
• Improved prompt adherence: Reduces off-topic or incomplete responses.
• Format consistency: Learns to output JSON, lists, or other structured formats as requested.

Models like InstructGPT and many open-source instruction-tuned LLMs are products of this process.

Full SFT (updating all parameters) of a large language model is resource-intensive, requiring significant GPU memory and time. This has driven the adoption of Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA for SFT.

Key Limitations:

• Catastrophic Forgetting: The model may lose general knowledge or capabilities not present in the fine-tuning data.
• Amplification of Biases: Inherits and can amplify biases present in the demonstration dataset.
• Limited to Demonstrated Behavior: Cannot learn nuanced preferences (e.g., 'style A is slightly better than style B') without moving to preference-based methods.

SFT is often made practical for large models through Parameter-Efficient Fine-Tuning techniques. These methods freeze the base model's weights and inject small, trainable adapters.

Common PEFT approaches for SFT:

• Low-Rank Adaptation (LoRA): Injects trainable low-rank matrices into attention layers.
• Prefix Tuning: Prepends a sequence of trainable continuous vectors to the input.
• Adapter Layers: Inserts small feed-forward networks between transformer blocks.

Using PEFT for SFT drastically reduces GPU memory requirements, often allowing SFT of large models on a single consumer-grade GPU, while preserving most of the performance gains.

SFT is the primary method for creating capable instruction-following models and chat assistants. By training on curated datasets of (instruction, ideal response) pairs, models learn to interpret user intent and generate appropriate, helpful, and safe completions.

• Examples: Training models like Claude or Llama 2-Chat on datasets such as Anthropic's HH-RLHF or OpenAI's InstructGPT data.
• Process: The base model (e.g., GPT-3) is fine-tuned on thousands of examples demonstrating how to follow diverse instructions, refuse harmful requests, and adopt a helpful persona.
• Outcome: Transforms a next-token prediction engine into a controllable, interactive agent.

SFT tailors a general model for expert performance in a specific vertical or task, such as legal analysis, medical Q&A, or code generation. This involves training on a high-quality corpus of domain-specific input-output pairs.

• Key Benefit: Achieves high accuracy on narrow tasks without the cost of training a model from scratch.
• Real-World Use:
  • Code Models: Codex was created by SFT of GPT-3 on a massive corpus of public code.
  • Medical Models: Fine-tuning clinical LLMs on doctor-patient dialogue and medical literature Q&A.
  • Legal Tech: Adapting models to summarize contracts or answer questions on case law.
• Data Requirement: Success depends entirely on the quality, accuracy, and breadth of the supervised dataset.

SFT provides the essential initialized policy for the Reinforcement Learning from Human Feedback (RLHF) pipeline. It creates a model that is already competent at the target task, which is then refined by reinforcement learning to better match human preferences.

• Pipeline Role: In the standard RLHF workflow (e.g., for ChatGPT), SFT is Step 1. A reward model (Step 2) and PPO (Step 3) build upon this SFT model.
• Why it's Necessary: RLHF requires a starting policy that generates coherent, on-topic responses. Training a reward model on random base model outputs would be ineffective.
• Alignment Tax Mitigation: A high-quality SFT model helps minimize the alignment tax—the loss of general capabilities during subsequent preference tuning.

SFT is used to imbue a model with a specific writing style, brand voice, or tonal consistency. This is critical for enterprise applications where generated content must adhere to corporate guidelines.

• Mechanism: The model is trained on examples of the desired style (e.g., formal reports, casual blog posts, concise technical documentation).
• Applications:
  • Customer Service Bots: Adopting a brand's specific supportive and professional tone.
  • Marketing Content: Generating copy that matches a company's established voice.
  • Localization: Adapting not just language, but cultural nuance and communication norms.

SFT teaches models to produce outputs in strictly defined formats, such as JSON, XML, API calls, or database queries. This enables reliable integration with downstream software systems.

• Use Case: Enabling tool use and function calling by training the model to generate valid arguments for predefined schemas.
• Process: The training dataset consists of natural language prompts paired with correctly formatted structured outputs.
• Example: A model fine-tuned to always return {"action": "create_calendar_event", "parameters": {...}} in response to "Schedule a meeting tomorrow." This is a precursor to more advanced agentic frameworks.

Applying Parameter-Efficient Fine-Tuning (PEFT) methods to SFT drastically reduces computational cost. Instead of updating all model weights, techniques like LoRA, prefix-tuning, or adapters are used to learn a small set of new parameters.

• Advantage: Makes SFT feasible for large models (e.g., 70B+ parameters) on limited hardware, enabling rapid iteration and multi-task adaptation.
• Workflow: The base model remains frozen. A small, trainable low-rank adapter (in LoRA) is optimized on the SFT dataset. The adapter is then merged for inference or swapped for different tasks.
• Result: Achieves performance close to full fine-tuning at a fraction of the memory and storage cost, a key technique for on-device AI and edge deployment.

A specialized form of Supervised Fine-Tuning (SFT) where a language model is trained on a dataset of (instruction, response) pairs. The goal is to improve the model's ability to understand and follow natural language task descriptions, making it more controllable and useful for downstream applications.

• Core Objective: Generalize to unseen instructions, not just memorize task-specific patterns.
• Dataset Example: Datasets like Alpaca or FLAN contain millions of diverse (instruction, output) examples.
• Relationship to SFT: Instruction tuning is a primary use case for SFT, focusing on broad instruction-following capability rather than narrow task performance.

A multi-stage alignment technique that uses human preference data to further refine a model after Supervised Fine-Tuning (SFT). It involves training a reward model on human rankings of responses, then using reinforcement learning (e.g., PPO) to optimize the SFT model against this reward signal.

• Pipeline Stages: 1) SFT, 2) Reward Model Training, 3) RL Fine-Tuning.
• Purpose: Aligns model outputs with nuanced human preferences for qualities like helpfulness, harmlessness, and style.
• Key Challenge: Reward overoptimization, where the model exploits the reward model's flaws.

An alignment algorithm that serves as an efficient alternative to the RLHF pipeline. DPO directly optimizes a language model policy to satisfy human preferences using a closed-form loss derived from the Bradley-Terry model, bypassing the need to train a separate reward model or perform complex reinforcement learning.

• Mechanism: Derives an implicit reward function from the policy itself and the preference data.
• Advantage over RLHF: Simpler, more stable training that is often more computationally efficient.
• Input: A preference dataset of paired outputs with human-chosen preferences.

A neural network trained to predict a scalar reward value representing human preference for a given language model output. It is a critical component in RLHF, trained on a preference dataset where humans have ranked multiple responses to the same prompt.

• Function: Acts as a proxy for human judgment during the RL fine-tuning stage.
• Training Data: Typically learns from pairwise comparisons (Chosen vs. Rejected responses).
• Output: A single score used by the RL algorithm (e.g., PPO) to guide policy updates.

A curated collection of data used for model alignment, containing examples where human annotators have indicated a preferred response. This dataset is the foundational training data for reward models in RLHF and for algorithms like DPO.

• Common Format: (prompt, chosen_response, rejected_response).
• Scale: Often requires tens of thousands to millions of human annotations.
• Quality Critical: Noisy or biased preferences directly lead to a poorly aligned model.

A regularization term added to the reward function during the RL fine-tuning stage of RLHF. It penalizes the fine-tuned policy for deviating too far from the initial SFT model, preventing reward overoptimization and catastrophic forgetting of language capabilities.

• Purpose: Maintains trust region constraints, ensuring policy updates are stable and conservative.
• Effect: Encourages the aligned model to remain coherent and linguistically competent.
• Formula: The penalty is typically β * KL(π_θ || π_ref), where β is a scaling coefficient.