Success Criterion

A core characteristic is that a success criterion must be objectively measurable. It should define a specific, quantifiable outcome that can be programmatically or clearly assessed, moving beyond subjective judgment.

• Examples: 'The response must be under 100 words,' 'The JSON object must validate against this schema,' 'The answer must include citations from the provided context.'
• Avoids: Vague goals like 'be helpful' or 'be creative,' which are open to interpretation.

The criterion must be verifiable from the model's generated text or structured output alone. It should not require inferring the model's internal state or intent.

• Observable: Presence/absence of required data fields, adherence to a specified format (JSON, XML), word count, inclusion of specific keywords or disclaimers.
• Not Observable: 'The model understood the nuance,' or 'The model intended to be concise.' Evaluation relies solely on the final artifact.

A success criterion is not universal; it is tailored to the specific task defined in the system prompt. It operationalizes the high-level goal into a concrete checkpoint.

• Example Task: 'Summarize this technical document.'
• Aligned Criterion: 'The summary must list the three main architectural components described in section 2.'
• This ensures the evaluation directly measures the intended utility of the task.

A well-defined criterion enables a clear pass/fail determination or a scalar score. This is essential for automated testing and performance benchmarking.

• Binary: 'Does the output contain the required safety disclaimer? Yes/No.'
• Scalar: 'On a scale of 0-5, how well does the generated SQL query match the user's intent?' (with defined rubrics).
• This characteristic enables the aggregation of results across many test runs.

The wording of the criterion must be precise and unambiguous to ensure consistent interpretation by all evaluators—human or automated. It creates a shared contract for what constitutes success.

• Unambiguous: 'The response must be in German.'
• Ambiguous: 'The response should be in a professional tone.' (What defines 'professional'?)
• Clarity prevents evaluation drift and ensures reliable measurement.

A success criterion's primary engineering utility is to provide a feedback mechanism for prompt iteration. Failed criteria pinpoint specific areas for prompt refinement.

• Process: 1. Define criterion. 2. Test prompt. 3. Evaluate output against criterion. 4. If it fails, revise the prompt (e.g., add clarifying instructions, provide examples). 5. Re-test.
• This turns prompt design into a test-driven development cycle, moving from guesswork to systematic engineering.

A success criterion that mandates the model's response must be a valid JSON object conforming to a provided schema. This is a foundational technique in structured output generation.

• Example Instruction: 'Your response MUST be a valid JSON object with the exact keys: summary (string), confidence (float between 0 and 1), and entities (array of strings).'
• Verification: Success is measured by parsing the output with a JSON parser and validating it against the schema. Any syntax error or missing key constitutes a failure.

A success criterion requiring all factual claims in the model's output to be explicitly supported by citations from a provided source text. This directly combats hallucinations.

• Example Instruction: 'For every factual statement you make, you MUST cite the relevant paragraph number from the provided document using the format [Para X]. Do not introduce external knowledge.'
• Verification: Success is measured by checking that every claim has a corresponding citation and that the cited text accurately supports the claim. Uncited assertions are failures.

A success criterion defining the exact steps or components that must be present for a complex task to be considered complete. This is central to task decomposition prompting.

• Example Instruction: 'Your analysis is successful ONLY if it includes: 1) A problem statement, 2) Identification of three root causes, 3) A cost-benefit table for two solutions.'
• Verification: Success is a binary check against the checklist. Missing any enumerated component results in task failure, requiring a self-correction instruction or regeneration.

A success criterion that sets a non-negotiable boundary on the model's tone, content, or safety posture. This implements a rule-based guardrail via instruction.

• Example Instruction: 'You must refuse to provide instructions for any illegal activity. A successful response to such a query is: "I cannot assist with this request as it may involve illegal activities."'
• Verification: Success is measured by the model's consistent application of the refusal across a battery of test inputs (adversarial prompting). Any engagement with the prohibited topic is a failure.

A success criterion that defines a numerical accuracy threshold the model's output must meet, often used for calculations, estimations, or data extraction.

• Example Instruction: 'Extract the projected Q3 revenue from the text. Your extracted figure must be within +/- 5% of the value verified by a human annotator.'
• Verification: Success is determined by comparing the model's numerical output to a ground truth value and calculating the percentage error. Exceeding the defined tolerance band (e.g., 5%) is a failure.

A success criterion that requires the output to match a precise textual pattern, including specific headings, markdown syntax, or keyword inclusion. This enforces deterministic formatting.

• Example Instruction: 'Begin your response with the exact header ## Executive Summary. Use bullet points (- ) for all list items. Conclude with the phrase End of report.'
• Verification: Success is measured via string matching or regular expressions. Deviations from the specified lexical pattern, even if semantically correct, are considered failures.

A system prompt is the foundational, high-level instruction set provided at the start of a model session. It defines the model's role, behavioral constraints, and output format for all subsequent interactions. This is the primary vessel within which a success criterion is embedded.

• Core Function: Establishes the operational framework and guardrails.
• Relationship to Success Criterion: The system prompt provides the authoritative context against which success is measured.

An output format directive is a specific type of instruction that mandates the structure or syntax of the model's response, such as JSON, XML, or a markdown table. It is a common and measurable success criterion for integration-focused tasks.

• Key Examples: "Always respond in valid JSON.", "Structure your answer as a bulleted list."
• Enforcement Techniques: Often paired with JSON Schema enforcement or grammar-based sampling for deterministic results.

Instruction priming is the strategic placement of core directives—including the success criterion—at the beginning of the context window to maximize their influence on model generation. This technique combats instruction decay.

• Mechanism: Leverages model attention patterns where initial tokens have disproportionate weight.
• Best Practice: Place the most critical success criteria first in the system prompt.

A meta-instruction is a directive that governs how the model should process other instructions. It often frames the evaluation process itself, acting as a higher-order success criterion for reasoning quality.

• Common Forms: "Think step by step.", "Evaluate your answer for correctness before responding."
• Purpose: Elicits chain-of-thought reasoning or self-correction, making the model's adherence to the primary success criterion more transparent and reliable.

Prompt testing frameworks are systematic methodologies for evaluating prompt robustness, reliability, and performance. They operationalize the success criterion by providing the tools for quantitative measurement.

• Process: Involves creating benchmark datasets, defining evaluation metrics (accuracy, format compliance), and running automated test suites.
• Output: Generates pass/fail rates and performance scores that directly report on whether success criteria are met.

Deterministic formatting is the engineering goal of ensuring a model's output consistently matches a precise, repeatable structure. It represents a high-stakes class of success criterion where any deviation constitutes a failure.

• Enabling Technologies: Achieved through a combination of strict output format directives, JSON Schema enforcement, and constrained decoding techniques like grammar-based sampling.
• Use Case: Critical for API integrations where downstream systems parse the model's output automatically.