Golden Dataset Bias Detection Prompt

This prompt is for responsible AI teams, evaluation engineers, and ML platform leads who need to audit a golden dataset before it becomes the gatekeeper for prompt releases. A golden dataset with representational gaps will silently approve prompts that fail on underrepresented groups. The prompt analyzes the inputs in your test suite for demographic skew, linguistic bias, domain over-concentration, and missing edge cases. It produces a structured bias risk report with stratified sampling recommendations so you can fix coverage gaps before they reach production.

Use this prompt when you are building a new golden dataset, expanding an existing one, or preparing for a fairness audit. It is designed to run as a pre-release checkpoint in your evaluation pipeline, not as a one-time academic exercise. The prompt expects a structured representation of your dataset inputs—typically a JSON array of test cases with their input fields, metadata tags, and any demographic or domain labels you already track. If your dataset lacks metadata, you should first run a classification pass to tag inputs by category, language, formality, or protected attribute proxies before feeding them into this bias detection prompt. The output is a risk report with specific, actionable flags: which subgroups are underrepresented, which input patterns are over-concentrated, and which edge cases are entirely absent. Each flag includes a severity rating and a concrete sampling recommendation.

Do not use this prompt for real-time production monitoring or for evaluating model outputs directly. It audits the test suite itself, not the model. It also cannot detect bias that stems from your labeling schema or evaluation rubric—those require separate rubric audit prompts. If your golden dataset is small (fewer than 50 examples), statistical skew analysis will be unreliable; the prompt will flag this and recommend expansion before bias analysis. Finally, this prompt does not replace a formal fairness audit with human domain experts. It is a engineering checkpoint that surfaces coverage risks early, when they are cheapest to fix. After running this prompt, feed the stratified sampling recommendations into your dataset expansion workflow and re-run the audit to confirm gaps have closed.

This prompt template is designed to be pasted directly into your evaluation harness. It instructs the model to act as a bias auditor, analyzing a provided golden dataset for representational gaps that could mask disparate performance in production. The template uses square-bracket placeholders for all dynamic inputs, allowing you to swap in your specific dataset, domain context, and protected categories without modifying the core instruction logic. Before using this in a CI/CD pipeline, ensure you have a human-in-the-loop review step for any high-severity bias flags, as automated bias detection can produce false positives or miss context-dependent harms.

textCopy
You are a responsible AI auditor specializing in dataset bias detection. Your task is to analyze a golden dataset used for regression testing and identify representational gaps that could lead to disparate model performance across different user groups.

## INPUT DATA
[GOLDEN_DATASET]

## DOMAIN CONTEXT
[DOMAIN_DESCRIPTION]

## PROTECTED CATEGORIES TO AUDIT
[PROTECTED_CATEGORIES]

## OUTPUT SCHEMA
Return a valid JSON object with the following structure:
{
  "dataset_summary": {
    "total_examples": <integer>,
    "audit_timestamp": "<ISO 8601>",
    "categories_audited": ["<category1>", "<category2>"]
  },
  "bias_risk_flags": [
    {
      "category": "<protected category>",
      "risk_level": "high|medium|low|none",
      "finding": "<description of the gap or skew detected>",
      "evidence": "<specific examples or counts from the dataset>",
      "potential_impact": "<how this gap could mask performance issues>"
    }
  ],
  "stratified_sampling_recommendations": [
    {
      "category": "<protected category>",
      "current_distribution": "<observed distribution>",
      "recommended_distribution": "<target distribution>",
      "additional_examples_needed": <integer>,
      "priority": "critical|high|medium|low"
    }
  ],
  "overall_risk_assessment": "<summary paragraph>",
  "requires_human_review": <boolean>
}

## CONSTRAINTS
- Only flag bias risks where you have clear evidence from the dataset. Do not speculate.
- If a protected category cannot be inferred from the data, note it as "insufficient data" rather than assuming absence of bias.
- For each risk flag, cite specific examples or distributional evidence.
- Prioritize intersectional gaps where multiple categories overlap.
- If no bias risks are detected, return an empty bias_risk_flags array and explain why in the overall_risk_assessment.

## RISK LEVEL
[HUMAN_REVIEW_REQUIRED]

To adapt this template, replace each square-bracket placeholder with your actual data. For [GOLDEN_DATASET], provide the full set of test inputs, expected outputs, and any metadata tags. For [PROTECTED_CATEGORIES], list the specific dimensions you need to audit—such as language, dialect, demographic indicators, geographic references, or domain-specific attributes. The [DOMAIN_DESCRIPTION] field should include enough context for the model to understand what constitutes a meaningful gap in your specific use case. Set [HUMAN_REVIEW_REQUIRED] to true for high-stakes audits where false positives could block releases; set to false only when the audit output feeds into a broader review process with additional checks. After running this prompt, validate the output JSON against the schema before passing results to your reporting dashboard or release gate logic.

The Golden Dataset Bias Detection Prompt is not a one-off audit tool—it should run as a gated step in your prompt release pipeline. Every time a golden dataset is created, updated, or pruned, this prompt should execute against the dataset's input distribution before the dataset is accepted into the regression suite. The harness must capture the structured bias flags, stratified sampling recommendations, and coverage gap warnings in a machine-readable format so downstream release gates can block deployment if representational risks exceed your team's threshold.

Wire the prompt into your evaluation runner as a pre-validation hook. The harness should: (1) serialize the golden dataset inputs into the [DATASET_INPUTS] placeholder as a JSON array of records with input text and metadata tags; (2) inject the [DOMAIN_CONTEXT] describing the application's user base, languages, and use cases; (3) call the model with temperature=0 and a strict JSON output schema that includes bias_flags, severity_scores, affected_groups, coverage_gaps, and stratified_sampling_plan; (4) validate the response against the schema and retry once on parse failure with a repair prompt that includes the schema and the raw malformed output; (5) log the full prompt, response, and validation result to your prompt observability store with the dataset version hash and timestamp. If the model returns severity_scores.critical > 0, the harness should block the dataset from entering the regression suite and route the findings to a human review queue with the flagged inputs, affected groups, and recommended remediation actions.

For high-stakes domains such as hiring, lending, healthcare, or content moderation, add a mandatory human approval step before any bias flag is dismissed. The harness should surface the specific input examples the model flagged, the model's reasoning for each flag, and the stratified sampling plan so reviewers can verify false positives and confirm true gaps. Store reviewer decisions (confirmed, dismissed, remediated) alongside the prompt run so you can measure the judge's precision and recall over time. Avoid running this prompt silently in CI without a blocking gate—bias detection that doesn't stop a release is just documentation of a problem you chose to ship.