Outcome Supervision

A training methodology that provides feedback on each intermediate step of a model's reasoning chain. Unlike outcome supervision, it rewards correct logical progression and penalizes flawed steps even if the final answer is accidentally correct. This approach directly addresses post-hoc rationalization and improves the faithfulness of the generated reasoning trace.

A specialized model trained to evaluate the correctness and logical validity of each step in a reasoning chain. The PRM is used to guide and score the step-by-step generation of a policy model. It is the core architectural component enabling process supervision, providing dense, granular feedback signals that a simple outcome-based verifier cannot.

A bootstrapping technique where a model is trained on its own correctly reasoned answers. The process involves:

• Generating rationales for a dataset using few-shot prompting.
• Filtering for rationales that led to a correct final outcome (a form of outcome supervision).
• Fine-tuning the model on these successful (question, rationale) pairs. This iteratively improves intrinsic reasoning capabilities from sparse outcome signals.

A training paradigm that uses human preferences to align models. While often used to shape stylistic outputs, RLHF can be applied to reasoning. A reward model is trained on human comparisons of model outputs. When the comparison is based solely on the final answer's quality, it functions as a learned form of outcome supervision, optimizing the policy without requiring step-level human annotation.