Demonstration Ordering

Places the most relevant or important example immediately before the target query. This leverages the recency effect, where the model's generation is most strongly influenced by the last content it processed. For classification, the final example's label often has disproportionate weight. For complex reasoning, the last demonstration's chain-of-thought provides the immediate reasoning template.

• Primary Use: Emphasizing a critical pattern or correct answer format.
• Risk: Can cause the model to overfit to the final example, ignoring broader task patterns from earlier demonstrations.

Orders examples from simplest to most complex. This scaffolds the model's understanding, establishing a foundational pattern before introducing nuance. A simple example defines the core input-output mapping, while subsequent ones add exceptions, edge cases, or multi-step reasoning.

• Primary Use: Tasks with a clear difficulty gradient or hierarchical reasoning structure.
• Example: Sentiment analysis starting with obvious ("Great!" → Positive) before ambiguous examples ("It was fine, I guess." → Neutral).

Arranges demonstrations so their inputs are progressively more semantically similar to the target query. This is often achieved via embedding-based retrieval. The model is gently guided from general task understanding to the specific sub-task required by the query.

• Primary Use: Retrieval-augmented ICL systems where a datastore of examples is available.
• Mechanism: Compute query embedding, retrieve k-nearest neighbor examples, and order them by descending cosine similarity to the query.

Strategically interleaves examples of different output classes to prevent label bias. In classification, presenting all Positive examples first may prime the model to over-predict that label. Alternating labels (e.g., A, B, A, B) or using a round-robin pattern provides a balanced label space view.

• Primary Use: Multi-class classification tasks to ensure the model sees all possible outputs.
• Consideration: Must be combined with clear input-output delineation to avoid confusing the model.

Leads with one or more perfectly formatted, canonical examples that establish the exact output structure (e.g., JSON schema, XML tags, bulleted list). Subsequent examples can vary in content but must adhere to this established template. This is critical for structured output generation.

• Primary Use: Enforcing deterministic output formats in API calls or data extraction tasks.
• Implementation: The first demonstration is a syntactic archetype; later ones reinforce that the format is invariant.

Intentionally includes an incorrect demonstration followed by its correction. This teaches the model not just the task, but also how to identify and avoid common mistakes. The pattern is often: [Input] -> [Bad Output] -> [Explanation of Error] -> [Corrected Output].

• Primary Use: Hallucination mitigation and training the model for self-correction.
• Caution: Requires clear meta-instructions to distinguish the error example from the correct ones.

For multi-step reasoning tasks, ordering demonstrations from simple to complex scaffolds the model's understanding. Start with a trivial example that breaks down the core logic, then provide progressively more intricate ones. This primes the model to apply a structured, stepwise approach to the final query, reducing logical leaps and hallucinations.

• Example: For a math word problem, first show: Input: 'John has 2 apples. He buys 3 more. How many?' -> Output: 'John started with 2. He adds 3. Total = 2 + 3 = 5.'
• Follow with a harder problem involving multiple operations.
• This progressive scaffolding is more effective than presenting the hardest example first, which can overwhelm the initial context.

Large language models exhibit a recency bias, weighting the final examples in a prompt more heavily. For classification tasks with imbalanced label distributions, place examples of the minority class last. This increases the probability the model will select that class for the target query.

• Scenario: Classifying sentiment as Positive, Neutral, or Negative, where Neutral is rare.
• Ineffective Order: Positive, Negative, Positive, Neutral, Positive, [Query].
• Effective Order: Positive, Negative, Positive, Positive, Neutral, [Query].
• By positioning the sole Neutral demonstration immediately before the query, you counter the model's tendency to default to the majority class.

To enforce strict output formatting like JSON, XML, or a custom template, the last demonstration should be a perfect exemplar of the desired structure. The model's strong recency effect makes it likely to replicate the syntax and stylistic conventions of the most recent example.

• Application: Generating API call parameters in JSON.
• Order Strategy: Provide 2-3 varied examples of valid JSON outputs, but ensure the final example is syntactically flawless and includes all optional fields you wish to see.
• This is more reliable than placing the best example first, as earlier context can be overwritten by later patterns.

For tasks requiring fine-grained discrimination between similar concepts, use a contrastive pair as your final demonstrations. Show two examples that are superficially similar but have critically different outputs, highlighting the decisive rule.

• Use Case: Legal clause analysis distinguishing indemnification from hold harmless.
• Final Pair:
  • Example 1 (Input: Clause A about future losses)... Output: This is INDEMNIFICATION.
  • Example 2 (Input: Clause B about existing liabilities)... Output: This is HOLD HARMLESS.
• Placing this direct contrast immediately before the query sharpens the model's focus on the differentiating features, not just superficial keywords.

In Retrieval-Augmented ICL systems, demonstration ordering is often dynamic. Retrieved examples are ranked by semantic similarity to the query, but the optimal sequence isn't always most-similar-first. A hybrid strategy places the most relevant example last (for recency) but may start with a highly diverse example to establish broad task scope.

• Process:
  1. Retrieve top k examples (e.g., k=5) using vector similarity.
  2. Re-order: Place the most generally representative example first.
  3. Place the example most specific to the query last.
• This balances task priming with query-specific guidance, outperforming simple similarity-ranked lists.

To reduce specific failure modes, include a negative example (demonstrating an incorrect approach) followed immediately by its corrected version. This order—wrong then right—explicitly teaches the model to avoid a common pitfall.

• Structure: [Example 1: Correct output] [Example 2: *Incorrect* output with explanation] [Example 3: *Corrected* version of Example 2] [Query]
• Why it works: The juxtaposition creates a strong contrastive signal. The model learns not just the target pattern, but also to recognize and reject a specific class of error. The correct version last reinforces the desired behavior.

In-context learning (ICL) is the foundational paradigm where a pre-trained language model performs a new task by conditioning its output on a few input-output examples provided within its prompt, without updating its internal parameters. Demonstration ordering is a critical optimization within ICL.

• Core Mechanism: The model uses the provided demonstrations as a temporary, task-specific "dataset" to infer the desired input-output mapping.
• Contrast with Fine-Tuning: Unlike fine-tuning, ICL is a gradient-free, parameter-free adaptation method; the model's weights remain frozen.

Demonstration selection is the strategic process of choosing which specific examples to include in a few-shot prompt to maximize performance. It is a prerequisite to ordering.

• Key Strategies:
  • Semantic Similarity Selection: Retrieves examples whose inputs are most similar to the current query, often using embedding-based retrieval.
  • Demonstration Diversity: Selects examples that cover a broad, representative range of the task's input space to improve generalization.
  • Exemplar Quality: Prioritizes examples that are clear, correct, and unambiguous.
• Related Technique: Retrieval-Augmented ICL dynamically retrieves relevant demonstrations for each query from a datastore.

Example formatting refers to the structural presentation of input-output pairs within a prompt, which works in tandem with ordering to clarify the task.

• Elements Include: The use of delimiters (e.g., ###, ---), whitespace, labels (Input:, Output:), and consistent syntax.
• Structured Demonstrations: Presenting examples in a highly organized format, such as a table or strict schema, to make the mapping explicitly clear.
• Interaction with Ordering: Poor formatting can negate the benefits of optimal ordering by introducing noise or ambiguity.

Dynamic few-shot prompting is an adaptive technique where the selection, quantity, and potentially the ordering of demonstrations are determined on-the-fly for each query.

• Contrast with Static: Moves beyond a fixed set of examples used for all queries.
• Drivers for Adaptation:
  • Query complexity or difficulty.
  • Available space in the model's context window.
  • Real-time retrieval scores from a k-NN demonstration retrieval system.
• Advanced Form: Adaptive demonstration strategies may also adjust the sequence based on predicted model performance.

Context priming is the psychological effect where earlier content in a prompt sets a contextual frame that biases the model's processing of subsequent content. Demonstration ordering directly manipulates this effect.

• Mechanism: Initial examples establish a "mental set" for the task, influencing how the model interprets later instructions and the target query.
• Recency vs. Primacy: Ordering strategies often trade off between:
  • Primacy Effect: Strong influence of the first example.
  • Recency Effect: Stronger influence of the most recent examples just before the query.
• Application: Strategic ordering aims to prime the model with the most relevant or clarifying examples at the most influential positions.

Inference-time adaptation is the broad category of techniques that modify a model's behavior dynamically during the forward pass, based on provided context, without updating weights. Demonstration ordering is a key method within this category.

• Encompasses: In-context learning, prompt tuning (for models with soft prompts), and certain calibration techniques.
• Core Principle: Adaptation occurs purely through the information encoded in the input sequence.
• Frozen Model Inference: The model executes in frozen state; all task-specific guidance comes from the engineered prompt, making the sequence and content of demonstrations paramount.