Few-Shot Chain-of-Thought (Few-Shot CoT)

Few-Shot CoT operates within the in-context learning paradigm, where a model learns a task dynamically from demonstrations provided within its prompt, without updating its internal weights. The examples serve as a temporary conditioning signal, teaching the model the expected format and depth of reasoning for the target task. This is distinct from fine-tuning, which permanently alters model parameters.

• Key Mechanism: The model identifies patterns in the provided demonstrations (problem structure, reasoning steps, answer format) and applies them to the new query.
• Primary Benefit: Enables rapid adaptation to new tasks without costly retraining or fine-tuning.

The core innovation is the inclusion of explicit intermediate reasoning steps within each few-shot example. Instead of just showing a question and final answer (Q → A), demonstrations follow a Q → Reasoning → A template. This scaffolds the model's internal computation, guiding it away from intuitive, one-step answers and toward a deliberate, decomposable thought process.

• Contrast with Standard Few-Shot: Standard few-shot learning lacks this step-by-step breakdown, often leading to leaps in logic or incorrect answers on arithmetic, symbolic, or multi-step reasoning problems.
• Example Structure: Question: If a zoo has 15 lions and 7 escape, then 3 new lions arrive, how many lions are there? Reasoning: Start with 15 lions. 7 escape, so 15 - 7 = 8 lions remain. 3 new lions arrive, so 8 + 3 = 11 lions. Answer: 11

A properly constructed set of Few-Shot CoT demonstrations teaches a generalizable reasoning strategy, not just answers to specific problems. The model learns how to approach a class of problems (e.g., multi-step arithmetic, temporal reasoning, commonsense deduction) and can apply this learned strategy to novel, unseen instances within that domain.

• Critical for Robustness: This transfer capability is what makes the technique powerful for production systems, allowing a single prompt template to handle a distribution of related queries.
• Domain-Specific Conditioning: Demonstrations are typically selected from the same or a highly similar domain as the target task to maximize relevance and transfer efficacy.

The performance of Few-Shot CoT is highly sensitive to the quality, diversity, and ordering of the example demonstrations. Effective prompt architecture involves careful curation.

• Diversity: Examples should cover different sub-types or potential pitfalls of the task to prevent the model from overfitting to a narrow pattern.
• Complexity Gradient: Often, ordering examples from simpler to more complex can provide a clearer learning trajectory.
• Relevance: Each demonstration must be semantically relevant to the target query domain. This concept is advanced by techniques like Active Prompting, which dynamically selects examples based on uncertainty metrics.

Few-Shot CoT was pioneered to dramatically improve performance on tasks requiring multi-step symbolic manipulation and arithmetic, where standard language models frequently fail. By breaking down a calculation like ((15 - 7) + 3) into explicit steps, the model is less prone to off-by-one errors, operation order mistakes, or hallucinated numbers.

• Underlying Cause: Language models are not innate calculators; they approximate computation via pattern matching. CoT provides the correct patterns for decomposition.
• Empirical Result: The original 2022 paper demonstrated performance jumps from ~17% to ~78% accuracy on GSM8K, a benchmark of grade-school math word problems.

Few-Shot CoT is not an endpoint but a foundational method that enables more sophisticated reasoning frameworks. It provides the basic proof-of-concept that models can follow explicit reasoning traces, which is essential for:

• Self-Consistency: Running Few-Shot CoT multiple times and taking a majority vote on the final answer.
• Least-to-Most Prompting: Using CoT to solve decomposed sub-problems.
• Automatic Chain-of-Thought (Auto-CoT): Using the model itself to generate the few-shot demonstrations.
• Process Supervision: Training models where rewards are given for each correct reasoning step, inspired by the CoT structure.

Its structured output also makes model reasoning more interpretable and debuggable compared to a black-box final answer.

The foundational technique upon which Few-Shot CoT is built. Chain-of-Thought Prompting is a method that elicits a language model to generate a step-by-step reasoning trace before producing a final answer. It works by explicitly prompting the model to "think aloud," which significantly improves performance on complex arithmetic, commonsense, and symbolic reasoning tasks. Unlike Few-Shot CoT, standard CoT can be applied in a zero-shot manner with a simple instruction like "Let's think step by step."

A prompting method that triggers step-by-step reasoning without any provided examples. Instead, it uses a simple, universal trigger phrase appended to a query, such as "Let's think step by step" or "We need to reason through this." The model then generates its own reasoning chain. While effective and simple, Zero-Shot CoT can be less reliable than Few-Shot CoT on highly specialized or novel tasks where the model benefits from seeing a demonstration of the expected reasoning style and format.

A decoding strategy that enhances the robustness of Chain-of-Thought reasoning. Instead of generating a single reasoning path, the model samples multiple, diverse chains of thought for the same problem. The final answer is selected by taking a majority vote over the conclusions from all sampled paths. This approach marginalizes over the model's uncertainty in the reasoning process itself, often leading to more accurate and reliable results than using a single CoT output. It is commonly applied on top of Few-Shot or Zero-Shot CoT prompts.

A method to automate the creation of Few-Shot CoT demonstrations. Manual curation of high-quality reasoning examples is labor-intensive. Auto-CoT addresses this by:

• Using the language model itself (often via Zero-Shot CoT) to generate reasoning chains for a set of selected example questions.
• Applying clustering or diversity sampling to select a representative set of demonstrations. This creates a prompt library automatically, bridging the gap between Zero-Shot and manually engineered Few-Shot CoT. It demonstrates the principle of using the model to bootstrap its own conditioning context.

A framework that interleaves language model reasoning with external actions. ReAct prompts a model to generate a Thought (a reasoning step), an Action (e.g., a tool/API call like Search(...) or Calculator(...)), and then an Observation (the result of that action). This loop continues until the task is solved. While Few-Shot CoT is purely internal reasoning, ReAct integrates external knowledge and computation, making it suitable for tasks requiring up-to-date information or precise calculation. Few-Shot examples in ReAct demonstrate the pattern of interleaving thought and action.

A prompting method where the intermediate reasoning step is executable code. Instead of a natural language reasoning chain, the model generates a program (e.g., in Python) that, when executed, computes the answer. This is particularly powerful for mathematical and algorithmic problems where symbolic computation is more reliable than textual reasoning. PoT can be used in a few-shot manner, where examples pair a problem with a code snippet that solves it. It represents a shift from reasoning in language to reasoning via computation.