Chain-of-Thought (CoT) Prompting

CoT instructs the model to decompose a complex problem into a sequence of discrete, natural language reasoning steps before arriving at a final answer. Unlike standard zero-shot prompting where the model jumps directly to a conclusion, CoT forces the model to externalize its internal computation.

• Each step is a declarative statement that builds on prior steps
• The reasoning chain is human-readable and auditable
• Errors in logic can be isolated to specific steps rather than the entire output

Example: Instead of answering '42' to a math word problem, the model outputs: 'First, calculate the total cost: 3 × $10 = $30. Then subtract the discount: $30 - $5 = $25. Finally, divide by the number of people: $25 ÷ 5 = $5 per person.'

CoT reasoning is an emergent capability that only manifests reliably in sufficiently large models. Smaller models (typically below 10B parameters) attempting CoT often produce fluent but logically incoherent reasoning chains that degrade rather than improve accuracy.

• Performance gains from CoT scale non-linearly with model size
• Models below ~50B parameters may show marginal or negative improvement
• The reasoning depth achievable correlates directly with parameter count and training compute

This scaling behavior suggests CoT leverages latent reasoning circuits that only crystallize at sufficient model capacity, making it a key differentiator between small and frontier models.

A critical enhancement to CoT is self-consistency, which samples multiple diverse reasoning paths and selects the most frequent final answer via majority voting. This technique mitigates the stochasticity of individual reasoning chains.

• Generate 5-40 independent CoT completions with non-zero temperature
• Cluster final answers and select the consensus result
• Particularly effective on arithmetic and logical reasoning tasks where there is a single correct answer

Self-consistency transforms CoT from a single-path reasoning strategy into a sampling-based ensemble method, often yielding 5-15% additional accuracy gains over single-path CoT on benchmarks like GSM8K and SVAMP.

CoT excels at tasks requiring compositional generalization — combining multiple logical operations, mathematical functions, or symbolic manipulations in sequence. The model explicitly tracks intermediate states that would otherwise be lost in a single forward pass.

• Arithmetic: Multi-digit operations, fractions, percentages
• Symbolic: Letter concatenation, last-letter extraction, boolean logic
• Commonsense: Temporal reasoning, spatial navigation, counterfactual analysis
• Code: Algorithm tracing, variable state tracking, debugging

CoT is less beneficial for tasks solvable via pattern matching alone (e.g., sentiment classification, factual recall) and most impactful where sequential dependencies exist between reasoning steps.

Effective CoT prompts follow a rigid structural template that clearly separates the reasoning phase from the answer extraction phase. Ambiguous formatting can cause the model to conflate intermediate steps with final outputs.

• Use delimiters like 'Reasoning:' and 'Answer:' to separate phases
• Number reasoning steps explicitly for complex multi-part problems
• Include explicit termination cues like 'Therefore, the answer is...'
• For programmatic parsing, request structured output (JSON with 'reasoning' and 'answer' fields)

Example template: 'Q: [problem]\nA: Let's think step by step.\nStep 1: [reasoning]\nStep 2: [reasoning]\nTherefore, the final answer is [answer].'