Self-Consistency

Instead of a single deterministic output, the model generates multiple candidate solutions using a non-zero temperature and varied sampling. This explores the space of possible logical derivations.

• Mechanism: Uses stochastic decoding to produce 5-40 independent chains-of-thought.
• Diversity Driver: Higher temperature and nucleus sampling prevent the model from repeating the same superficial pattern.
• Key Insight: A correct final answer can be reached via multiple distinct logical routes, while errors tend to be idiosyncratic and inconsistent.

The final answer is selected by aggregating the outputs of all sampled paths, typically using majority voting or a normalized frequency count. This process marginalizes out the randomness of individual reasoning steps.

• Aggregation: The most common final answer across all chains is chosen.
• Unsupervised: Requires no external verifier or ground-truth labels during inference.
• Complexity Weighting: Advanced variants weight paths by their logical length or inverse perplexity to favor more deliberate reasoning.

Standard greedy decoding selects the single highest-probability token at each step, often leading to brittle, surface-level reasoning. Self-Consistency sacrifices local token optimality for global answer robustness.

• Greedy Failure Mode: A single high-confidence mistake early in a chain derails the entire derivation.
• Self-Consistency Advantage: By sampling, the model can 'recover' from local errors in some paths, allowing the correct global answer to emerge as the consensus.
• Computational Trade-off: Requires N times more compute than greedy decoding, where N is the number of samples.

Self-Consistency is most effective for tasks with a fixed, verifiable answer that requires multi-step logical or mathematical derivation. It provides marginal gains on open-ended creative tasks.

• High Impact: Arithmetic reasoning, commonsense question answering, symbolic manipulation, and code execution.
• Low Impact: Creative writing, summarization, and translation, where diversity is subjective and no single 'correct' answer exists.
• Benchmark Example: Boosts accuracy on GSM8K (grade-school math) from ~60% with greedy decoding to over 80%.

A refinement of simple majority voting that accounts for the model's confidence in the full reasoning path, not just the final answer. This prevents a lucky guess from a low-probability chain from overriding a consensus of high-confidence derivations.

• Probability Product: Weights a path by the product of token probabilities normalized by sequence length.
• Soft Voting: Aggregates weighted probabilities for each unique answer rather than counting discrete votes.
• Result: Improves robustness when the distribution of sampled paths is noisy or contains many unique, low-quality derivations.

Self-Consistency is a wrapper strategy applied on top of Chain-of-Thought (CoT) prompting. CoT provides the reasoning structure, while Self-Consistency provides the statistical robustness.

• Dependency: Requires CoT to generate intermediate steps; it cannot function on direct answer prediction.
• Synergy: CoT surfaces the reasoning, making the diversity of paths meaningful. Self-Consistency then validates which reasoning cluster is most reliable.
• Combined Power: The pairing is a standard baseline for complex reasoning benchmarks, often outperforming single-path models with 10x the parameters.