Source Credibility Eval Check and Failure Mode Catalog Prompt

This prompt is designed for QA engineers and verification pipeline developers who need to systematically stress-test a source credibility assessment prompt before it ships. The job-to-be-done is not authoring a new prompt from scratch, but rather taking a candidate prompt that already exists and running it through a battery of adversarial test cases to catalog where it breaks, over-calibrates, under-calibrates, or produces inconsistent outputs. The ideal user has a working source assessment prompt—likely one that produces structured credibility scores, authority classifications, or evidence-weighting decisions—and needs to know its failure boundaries before it becomes a dependency in an automated fact-checking pipeline. Required context includes the candidate prompt itself, a set of test sources with known characteristics (edge cases, ambiguous formats, deliberate adversarial inputs), and the expected output schema that the candidate prompt should conform to.

This is a batch QA gate tool, not a single-source evaluator. It expects multiple test sources as input and produces a structured evaluation report that catalogs failure modes, edge cases, and calibration gaps across the entire test set. The output should include: a failure mode catalog with specific detection criteria for each pattern, per-source evaluation results with pass/fail flags, calibration drift warnings where scores are systematically too high or too low, and a summary readiness assessment. Wire this into a CI/CD pipeline for prompt releases by running it against a golden test set whenever the candidate prompt changes. The test set should include sources with missing metadata, conflicting date signals, ambiguous formats (preprints, press releases, syndicated content), deliberately obscured conflicts of interest, and domain-mismatched authority claims. Each test source should have a known expected behavior so the evaluation can compare actual outputs against ground truth.

Do not use this prompt for initial prompt authoring or for evaluating a single source in isolation. It is not a replacement for unit tests on individual prompt components, nor is it a human review workflow. It is a pre-release QA gate that answers the question: 'Where does this prompt break, and is it calibrated well enough to ship?' After running this evaluation, the reader should triage failures by severity, fix the candidate prompt or its constraints, and re-run the test suite until the failure mode catalog shows acceptable risk levels. Avoid the trap of treating a clean run as proof of production readiness—adversarial inputs evolve, and the test set should be updated regularly with new edge cases discovered in production traces.

This template is designed for QA engineers who need to systematically stress-test a source credibility evaluation prompt. It instructs the model to act as an adversarial evaluator, probing the target prompt with a diverse set of source scenarios and producing a structured catalog of failure modes. The output is not a simple pass/fail but a detailed report that identifies specific weaknesses, such as over-reliance on domain authority, failure to detect conflicts of interest, or miscalibration of recency scores, enabling targeted prompt refinement.

textCopy
You are an expert adversarial evaluator testing a source credibility assessment prompt. Your task is to generate a structured evaluation report that identifies failure modes, edge cases, and calibration gaps in the target prompt provided below.

[TARGET_PROMPT_TO_EVALUATE]

[EVALUATION_CRITERIA]

[TEST_SOURCE_CATALOG]

[OUTPUT_SCHEMA]

[CONSTRAINTS]

# Steps
1.  **Analyze the Target Prompt:** Deconstruct [TARGET_PROMPT_TO_EVALUATE] to understand its intended logic, scoring dimensions, and expected output format.
2.  **Execute Test Cases:** For each source in [TEST_SOURCE_CATALOG], simulate the target prompt's execution. If the catalog is empty, generate a diverse set of 10 adversarial test sources designed to probe the specific failure modes listed in [EVALUATION_CRITERIA].
3.  **Identify Failures:** Compare the simulated output against the expected behavior defined in [EVALUATION_CRITERIA]. Flag any deviation as a failure mode.
4.  **Catalog and Diagnose:** For each identified failure mode, provide a name, a description of the triggering condition, a concrete example from the test sources, and a diagnosis of the likely root cause in the target prompt's instructions.
5.  **Calibrate Severity:** Assign a severity level (Critical, High, Medium, Low) to each failure mode based on its potential impact on a production verification pipeline.
6.  **Generate Report:** Structure the entire analysis strictly according to [OUTPUT_SCHEMA].

# Output Format
Produce the final output as a single JSON object conforming to the provided [OUTPUT_SCHEMA]. Do not include any text outside the JSON object.

To adapt this template, replace the placeholders with your specific assets. [TARGET_PROMPT_TO_EVALUATE] should contain the exact text of the source credibility prompt you are testing. [EVALUATION_CRITERIA] must define the expected behavior, such as 'Must flag sources with undisclosed funding as High Risk' or 'Must not downgrade a .gov source for recency if the claim is about a historical statute.' [TEST_SOURCE_CATALOG] can be a pre-built list of tricky sources, or you can leave it empty to force the model to generate its own adversarial examples. The [OUTPUT_SCHEMA] placeholder is critical for integration; define a strict JSON schema with fields like failure_modes (array of objects with name, description, trigger_condition, example, root_cause_diagnosis, severity) and calibration_notes (string). [CONSTRAINTS] can be used to add rules like 'Do not test for prompt injection; focus only on credibility logic flaws.'

Before running this prompt at scale, validate the output against your [OUTPUT_SCHEMA] in your application harness. A common failure mode of this evaluation prompt itself is generating vague root cause diagnoses like 'the prompt was unclear.' Mitigate this by including a strong example of a good diagnosis in your [EVALUATION_CRITERIA] or as a one-shot example within the prompt. For high-stakes verification systems, always have a human QA lead review the generated failure mode catalog to confirm that the identified issues are genuine and the severity ratings are calibrated to your operational risk tolerance.

This prompt is not a one-off chat interaction; it is a QA harness component designed to run inside a continuous evaluation pipeline. The primary integration point is a test runner that feeds the prompt a source assessment prompt (the subject under test) and a catalog of adversarial source descriptions. The output is a structured evaluation report that your CI/CD or release gate can consume. Wire this prompt as a post-generation step: after your team authors or modifies a source credibility prompt, the QA harness evaluates it against known failure modes before the prompt is promoted to production.

Integration pattern: Wrap the prompt in a function that accepts [SUBJECT_PROMPT], [SOURCE_CATALOG], and [EVALUATION_CRITERIA] as inputs. The function should call the LLM with response_format set to a strict JSON schema matching the expected evaluation report structure. On the application side, validate the returned JSON immediately: confirm that every failure mode entry contains a non-empty detection_criteria and observed_behavior field. If validation fails, retry once with the validation error injected into the [CONSTRAINTS] block. After two failed attempts, log the raw output and flag the run for human review. For high-stakes verification pipelines (e.g., healthcare, legal, financial claims), require a human sign-off on the evaluation report before the subject prompt is cleared for deployment.

Model selection and tool use: Use a model with strong instruction-following and structured output capabilities (e.g., GPT-4o, Claude 3.5 Sonnet) for the evaluation run. This prompt does not require external tools or retrieval—the source catalog is provided inline. However, if your source catalog exceeds ~8,000 tokens, split it into batches and run the evaluation per batch, then merge the reports with a deduplication step. Logging and observability: Log the full prompt, the subject prompt version hash, the model response, and the validation result to your prompt observability platform. Attach the evaluation report to your prompt version record so that every production prompt has a traceable QA report. What to avoid: Do not run this evaluation once and consider the subject prompt 'safe.' Source credibility prompts degrade as model behavior shifts, new source types appear, and adversarial patterns evolve. Schedule this evaluation to run on every subject prompt change and at a regular cadence (e.g., weekly) against an updated source catalog.