Curriculum Learning

The core mechanism of curriculum learning. Training begins with simpler, shorter examples and gradually introduces more complex ones. In legal pre-training, this might mean starting with statutory definitions and isolated clauses before moving to full contracts and multi-case syntheses. This mimics human pedagogy and prevents the model from being overwhelmed by complex reasoning tasks early in training, leading to faster convergence and often a better local minimum.

A critical implementation detail: how 'difficulty' is measured. Common heuristics for legal text include:

• Sequence length: Shorter texts are easier.
• Syntactic complexity: Measured by parse tree depth or clause embedding.
• Vocabulary rarity: The frequency of low-probability legal terms.
• Reasoning hops: The number of distinct legal concepts that must be linked. Models can also self-assess difficulty via loss values on a pre-trained model, where higher loss indicates a harder example.

Not all curricula are easy-to-hard. Research identifies variants:

• Anti-curriculum: Training starts with the hardest examples first. This can act as a strong regularizer in some contexts.
• Baby-step curriculum: The classic approach where the difficulty is incrementally increased.
• Self-paced learning: The model itself dynamically selects the next batch of examples based on its current competence, measured by the loss on a held-out validation set of varying difficulty.

Curriculum learning is highly synergistic with domain-adaptive pre-training (DAPT). A common legal AI pipeline involves:

1. General pre-training on broad internet text (the foundation).
2. DAPT on a large, uncurated legal corpus.
3. Curriculum-based fine-tuning on a structured legal dataset, moving from simple legal questions to complex multi-document reasoning tasks. This staged approach systematically builds capability without inducing catastrophic forgetting of the model's general language understanding.

The pacing function defines the rate at which harder examples are introduced. Key strategies include:

• Fixed schedule: A predefined number of steps for each difficulty level.
• Competence-based: New difficulty levels are introduced only when the model's performance on the current level exceeds a threshold (e.g., perplexity drops below a target).
• Root-p and exponential functions: Mathematical formulas that smoothly increase the proportion of complex examples over time, preventing sudden distributional shifts that can destabilize training.

A key risk when fine-tuning on a curriculum is that the model forgets earlier, simpler tasks. This is addressed by:

• Experience Replay: Interleaving a small percentage of 'easy' examples from earlier in the curriculum into later training batches.
• Elastic Weight Consolidation (EWC): Penalizing changes to parameters that were critical for solving earlier curriculum stages. This ensures the final model retains competence across the full spectrum of legal reasoning complexity, from simple clause extraction to complex case synthesis.

A sampling technique ensuring a pre-training corpus proportionally represents key legal sub-domains. In curriculum learning, stratification is applied sequentially:

• Phase 1: Broad, balanced legal English (statutes, simple contracts)
• Phase 2: Specialized sub-domains (tax code, securities filings)
• Phase 3: Complex multi-document reasoning tasks This prevents a model from overfitting to a single legal text type before mastering foundational patterns.

The tendency of a neural network to abruptly lose general language capabilities when trained on a narrow domain. Curriculum learning directly mitigates this by interleaving complexity levels rather than abruptly switching from general to legal data. Complementary techniques include:

• Elastic Weight Consolidation (EWC): Penalizes changes to parameters important for prior knowledge
• Experience Replay: Reintroduces earlier, simpler examples throughout training

An intrinsic evaluation metric measuring how 'surprised' a model is by held-out legal text. A well-designed curriculum should show monotonically decreasing perplexity across phases. Key indicators:

• Phase transition spikes: Small, temporary perplexity increases when introducing harder data are normal
• Divergence: A sustained perplexity increase signals the curriculum step was too aggressive
• Final score: Lower legal perplexity vs. random-order training validates the curriculum strategy

A self-supervised approach that pulls semantically similar legal text pairs together in embedding space. Curriculum learning enhances this by:

• Easy negatives: Early training uses clearly dissimilar texts (contracts vs. recipes)
• Hard negatives: Later phases introduce nuanced distinctions (majority opinions vs. dissents) This staged difficulty mirrors how legal experts learn to distinguish closely related doctrines.

The maximum number of tokens a model can process in a single forward pass. Curriculum learning directly addresses this architectural constraint:

• Short sequences first: Train on legal definitions and single clauses (512 tokens)
• Medium sequences: Progress to full contracts and single judicial opinions (4,096 tokens)
• Long sequences: Finally, multi-document synthesis tasks (8,192+ tokens) This progressive extension, combined with FlashAttention, makes long-context legal reasoning computationally feasible.