Golden Set Contamination Check Prompt

This prompt is designed for QA engineers and AI reliability engineers who observe an unexpected degradation in automated evaluation scores across a batch of production responses. Before diving into trace-level root-cause analysis, you need to eliminate the possibility that the evaluation itself is compromised. Contamination occurs when golden answers, eval rubrics, or expected outputs inadvertently leak into the system prompt, few-shot examples, or model training data, causing the model to memorize rather than reason. A contaminated golden set produces misleadingly high scores during development and sudden, unexplained drops when the model encounters genuinely novel inputs in production. This playbook produces a structured contamination risk report that systematically checks each potential leak vector.

You should use this prompt when you have access to the complete golden set entries, the active system prompt, any few-shot examples used in the evaluation pipeline, and the full eval rubric definitions. The prompt works by comparing linguistic patterns, key phrases, and structural similarities between the golden set and the model's instruction context. It flags cases where a golden answer appears to have been memorized rather than constructed, where rubric criteria are suspiciously echoed in the model's reasoning, or where few-shot examples overlap with evaluation cases. The output is a risk score per contamination vector and a prioritized list of golden set entries that warrant manual review. This is a diagnostic tool, not a replacement for human judgment—high-risk flags should be escalated for human verification before any golden set entries are modified or removed.

Do not use this prompt when you lack access to the full system prompt or golden set definitions, as partial data will produce unreliable risk assessments. It is also inappropriate for diagnosing individual response failures—use the Eval Failure Root-Cause Triage Prompt for single-trace investigation. If contamination is confirmed, you must quarantine the affected golden set entries, regenerate them from source material without referencing the model's instruction context, and re-run your evaluation baseline before proceeding with any prompt or model changes. Never modify golden set entries based solely on this prompt's output without human review, as false positives can silently erode your evaluation integrity over time.

This prompt template is designed to be pasted directly into your orchestration layer or evaluation pipeline. It instructs the model to act as a contamination auditor, comparing a suspect golden-set item against all prompt surfaces that could have inadvertently exposed the answer. The goal is to produce a structured contamination risk report, not to re-evaluate the model's output quality. Before using this prompt, ensure you have isolated the specific golden answer or rubric item that is under suspicion, along with the full prompt context that was active when the eval was run.

textCopy
You are a contamination auditor for AI evaluation datasets. Your task is to determine whether a specific golden-set answer or evaluation rubric item has leaked into the model's prompt context, creating a contamination risk that invalidates the evaluation.

## INPUTS
- Golden-Set Item: [GOLDEN_ITEM]
- Item Type: [ITEM_TYPE] (options: 'golden_answer', 'rubric_criterion', 'few_shot_example')
- Active System Prompt: [SYSTEM_PROMPT]
- Few-Shot Examples Used: [FEW_SHOT_EXAMPLES]
- Evaluation Rubric: [EVAL_RUBRIC]
- User Input That Triggered the Eval: [USER_INPUT]
- Model Output Under Evaluation: [MODEL_OUTPUT]

## INSTRUCTIONS
1. Compare the Golden-Set Item against every other input field.
2. Identify exact string matches, near-duplicate paraphrases, and semantic equivalents.
3. For each match found, classify the contamination vector:
   - `direct_leak`: The golden answer or a near-verbatim version appears in the system prompt, few-shot examples, or rubric.
   - `indirect_leak`: The golden answer's structure, key entities, or reasoning chain is strongly hinted at by the prompt context.
   - `rubric_circularity`: The evaluation rubric describes the golden answer so precisely that it effectively provides the answer.
   - `no_contamination`: No meaningful overlap detected.
4. Assess the contamination severity: `critical` (eval is invalid), `high` (likely invalid), `moderate` (requires review), `low` (minor overlap, unlikely to affect results), `none`.
5. Provide a confidence score from 0.0 to 1.0 for each finding.

## OUTPUT FORMAT
Return a valid JSON object with this exact schema:
{
  "contamination_detected": boolean,
  "overall_severity": "critical" | "high" | "moderate" | "low" | "none",
  "findings": [
    {
      "contamination_vector": "direct_leak" | "indirect_leak" | "rubric_circularity" | "no_contamination",
      "source_field": "system_prompt" | "few_shot_examples" | "eval_rubric" | "user_input",
      "matched_text_excerpt": "string excerpt showing the overlap",
      "golden_item_excerpt": "string excerpt from the golden item",
      "severity": "critical" | "high" | "moderate" | "low" | "none",
      "confidence": 0.0-1.0,
      "explanation": "Brief explanation of why this is or is not contamination"
    }
  ],
  "recommendation": "string with actionable next steps"
}

## CONSTRAINTS
- Do not evaluate the quality of the model output. Only assess contamination risk.
- If the golden item is a rubric criterion, check whether the system prompt or few-shot examples effectively teach the model to satisfy that criterion by rote rather than by capability.
- Flag near-duplicate paraphrases, not just exact string matches.
- If no contamination is found, set `contamination_detected` to false and `overall_severity` to `none`.

To adapt this template, replace each square-bracket placeholder with your actual data. The [GOLDEN_ITEM] should contain the exact text of the suspect golden answer, rubric criterion, or few-shot example. The [SYSTEM_PROMPT], [FEW_SHOT_EXAMPLES], and [EVAL_RUBRIC] fields should contain the full prompt context that was active during the evaluation run. If any field is empty, use an empty string or "N/A". After running the prompt, validate the output JSON against the schema before acting on the findings. For high-risk evaluations where contamination would invalidate a release gate, always have a human reviewer confirm the model's contamination assessment before marking the golden set as compromised.

This prompt is designed to run as a pre-investigation gate before deep-dive root-cause analysis of an eval failure. It should be triggered automatically when a production response fails against a golden-set answer, but before an engineer spends time diagnosing the model. The harness must feed the prompt three structured inputs: the golden answer, the system prompt used during generation, and the few-shot examples (if any) included in the request. The output is a structured contamination risk report that either clears the golden set for continued use or flags it for human review and potential quarantine.

Integration pattern: Wire this prompt into your eval pipeline as a conditional step. When eval_result == FAIL and eval_type == GOLDEN_SET_COMPARISON, call this prompt with the [GOLDEN_ANSWER], [SYSTEM_PROMPT], and [FEW_SHOT_EXAMPLES] fields populated from your prompt registry and eval configuration. Validation layer: Parse the JSON output and enforce the contamination_risk enum values (LOW, MEDIUM, HIGH, CRITICAL). If the output schema is malformed, retry once with a stricter schema constraint appended to the prompt. Logging: Record the contamination risk level, the flagged overlap segments, and the prompt version used for the check. This trace is essential for audit when a golden set is later found to be contaminated.

Human review gate: If contamination_risk is HIGH or CRITICAL, the harness must quarantine the golden-set entry and create a review ticket for a QA engineer. Do not automatically discard the golden answer—manual verification is required to distinguish genuine contamination from coincidental phrasing. For MEDIUM risk, log the finding but allow the golden set to remain active with a warning flag. Model choice: Use a model with strong instruction-following and JSON output discipline (e.g., GPT-4o, Claude 3.5 Sonnet). Avoid small or fast models for this check because false negatives in contamination detection can silently corrupt your eval benchmarks for weeks. What to avoid: Do not run this check on every successful eval—only on failures. Running it indiscriminately wastes tokens and creates noise. Also, never feed the production model's response into this prompt; it only needs the golden answer, system prompt, and few-shot examples to detect leakage at the source.