Silent Failure Heuristic Check Prompt

This prompt is for agent runtime monitors and postcondition validators that need to catch the most dangerous class of failure: outputs that look correct but are semantically wrong. Silent failures include empty results wrapped in valid JSON, schema-compliant garbage, confidence drops without explicit errors, and outputs that pass structural validation but violate business rules or expected value ranges. The primary user is an engineering lead or AI builder wiring this heuristic check into an agent execution loop—after a step completes but before its output is passed to the next step or written to a system of record.

Use this prompt when your agent produces structured outputs that downstream steps depend on, and when a single undetected bad output can corrupt the entire workflow. It is designed for steps where you have defined expected patterns, value ranges, or consistency rules in advance—for example, an extraction step that should return 5–20 entities, a summarization step that should produce text within a length band, or a classification step where confidence should exceed a threshold. The prompt works by comparing the actual step output against these heuristics and producing a structured pass/fail verdict with evidence. Do not use this prompt for open-ended creative generation, for steps where you cannot define any expected output characteristics, or as a replacement for proper schema validation—run structural validation first, then use this prompt for semantic checks.

Before wiring this into production, define your heuristics explicitly in the [EXPECTED_PATTERNS] placeholder. Vague heuristics produce vague verdicts. For high-risk workflows—finance, healthcare, legal, or any domain where a false negative could cause harm—always route FAIL verdicts to human review rather than auto-aborting the workflow. This prompt is a detection tool, not an auto-correction tool. Pair it with the Output Repair and Validation Prompts pillar if you need to fix detected failures automatically. Start by running this against a golden dataset of known good and known silently-bad outputs to calibrate your heuristics before deployment.

This prompt template is designed to be copied directly into your application code, test suite, or orchestration framework. It instructs the model to act as a silent-failure detector, analyzing a single step's output against expected patterns, value ranges, and consistency rules. The model is not asked to judge the output's quality in a general sense, but to apply specific, testable heuristics and return a structured verdict. All placeholders are enclosed in square brackets and must be populated by your application before the prompt is sent to the model.

textCopy
You are a silent-failure detector for an AI agent execution pipeline. Your task is to analyze the output of a single completed step and determine whether it contains a silent failure: an output that appears valid (correct format, no error code) but is semantically wrong, empty, or inconsistent with expectations.

## INPUT

[STEP_OUTPUT]

## CONTEXT

- Step Name: [STEP_NAME]
- Step Objective: [STEP_OBJECTIVE]
- Expected Output Schema: [OUTPUT_SCHEMA]
- Expected Value Ranges: [VALUE_RANGES]
- Consistency Rules: [CONSISTENCY_RULES]
- Previous Step Outputs (for cross-step consistency): [PREVIOUS_OUTPUTS]

## HEURISTICS TO APPLY

Apply each of the following heuristics. For each one, state whether it passed or failed and provide the specific evidence.

1. **Schema Compliance Check**: Does the output conform to [OUTPUT_SCHEMA]? Check for missing required fields, incorrect types, and extra fields.
2. **Emptiness Check**: Are any expected fields null, empty strings, empty lists, or missing when they should contain data?
3. **Value Range Check**: Do all numeric, date, or enumerated values fall within [VALUE_RANGES]?
4. **Consistency Check**: Does the output satisfy [CONSISTENCY_RULES]? For example, does a total match the sum of its parts, or does a status field contradict a description field?
5. **Cross-Step Consistency Check**: Is this output consistent with [PREVIOUS_OUTPUTS]? Flag contradictions, unexplained state reversals, or duplicated identifiers.
6. **Confidence and Certainty Check**: Does the output contain language indicating low confidence, hedging, or refusal (e.g., "I'm not sure", "possibly", "unable to determine") when a definitive answer is expected?
7. **Repetition and Degeneracy Check**: Does the output repeat the same token, phrase, or structure excessively, or does it match a known degenerate pattern?

## OUTPUT FORMAT

Return a JSON object with the following structure. Do not include any text outside the JSON object.

{
  "step_name": "[STEP_NAME]",
  "overall_verdict": "PASS" | "WARN" | "FAIL",
  "heuristic_results": [
    {
      "heuristic": "Schema Compliance Check",
      "result": "PASS" | "FAIL",
      "evidence": "Specific field names, values, or patterns observed."
    },
    ...
  ],
  "failure_summary": "A concise summary of all failures, or null if overall_verdict is PASS.",
  "recommended_action": "PROCEED" | "RETRY" | "ESCALATE" | "SKIP"
}

## CONSTRAINTS

- Do not evaluate the output's helpfulness, tone, or style unless it directly violates a consistency rule.
- Do not hallucinate failures. If a heuristic cannot be evaluated due to missing information, mark it as PASS and note the limitation in the evidence.
- If the output is completely empty or unparseable, overall_verdict must be FAIL and recommended_action must be ESCALATE.

To adapt this template, start by defining your [CONSISTENCY_RULES] and [VALUE_RANGES] as concrete, machine-readable assertions rather than vague descriptions. For example, instead of "the price should be reasonable," write "price must be a positive float between 0.01 and 999999.99." The [PREVIOUS_OUTPUTS] placeholder should be populated with a serialized summary of prior step results, not the raw outputs, to keep the prompt within context limits. If your workflow does not support cross-step checks, remove heuristic 5 and the [PREVIOUS_OUTPUTS] placeholder to avoid confusing the model with an empty field. For high-risk domains such as finance or healthcare, always route FAIL and WARN verdicts to a human review queue before allowing downstream steps to consume the output.

The Silent Failure Heuristic Check Prompt is not a standalone tool; it is a validation gate that should be inserted into an agent's execution loop immediately after a step produces an output but before that output is passed to the next step or written to a persistent store. The most common integration pattern is a post-execution hook inside an agent runtime (e.g., LangGraph, a custom orchestrator, or a Celery task pipeline) that calls this prompt with the step's raw output, the expected output schema, and any domain-specific heuristics such as value ranges, required fields, or consistency rules. The prompt returns a structured verdict (pass, flag, fail) along with evidence, which the harness uses to decide whether to proceed, retry, or escalate.

A production-grade harness must handle three concerns beyond the prompt call itself: input assembly, verdict routing, and observability. For input assembly, collect the step's output payload, the expected schema (as a JSON Schema or a plain-text description), and a set of heuristics defined per workflow—for example, [CONSTRAINTS] might include 'output must contain at least 5 records,' 'confidence scores must be ≥ 0.7,' or 'no field may be an empty string unless explicitly nullable.' For verdict routing, map the prompt's output to concrete actions: a pass verdict allows the step's output to proceed; a flag verdict logs the output and continues but triggers a review task or a notification; a fail verdict blocks the output, increments a retry counter, and invokes a retry or escalation path. For observability, log the full prompt input, the model's verdict, the step ID, and the timestamp to a structured tracing system (e.g., LangSmith, Braintrust, or a custom audit table) so that silent failures that were caught—and any that were missed—can be analyzed later.

Model choice matters here. This prompt requires strong instruction-following and structured output discipline, not creative generation. Use a model that reliably produces JSON and respects schema constraints, such as gpt-4o, claude-3-5-sonnet, or a fine-tuned variant if your heuristics are domain-specific. Set temperature=0 and enforce a strict JSON output mode or use a tool-calling interface where the verdict schema is defined as a function parameter. If the model fails to produce valid JSON or omits required fields in the verdict, treat that as a fail and escalate—do not silently retry the validation prompt itself, as that can mask deeper model reliability issues. For high-risk domains (finance, healthcare, legal), always route flag verdicts to a human review queue and never auto-proceed on flag without explicit operator approval.

The most common production failure mode is heuristic drift: the heuristics defined in [CONSTRAINTS] become stale as the application's data or business rules change, causing the prompt to either miss silent failures (false negatives) or flag valid outputs (false positives). Mitigate this by versioning your heuristics alongside the prompt template, running periodic evals against a golden dataset of known-good and known-bad outputs, and monitoring the flag and fail rates over time. A sudden spike in flag verdicts often indicates a schema change or a model behavior shift, not a real output quality problem. Wire the harness to emit a metric (e.g., silent_failure_check_flag_rate) to your monitoring dashboard so operators can distinguish between a broken workflow and a noisy validator.