Dataset Curation Review Prompt

This prompt is designed for QA leads and ML engineers who are responsible for maintaining the long-term health of a golden dataset used in regression testing. Its primary job is to systematically review existing examples for staleness, bias, coverage gaps, and schema drift. The output is a structured curation report that includes specific recommendations, priority scores, and a list of deprecation candidates, enabling the team to make data-driven decisions about dataset maintenance.

Use this prompt when your golden dataset has been in production for several development cycles, after a significant model upgrade, or when you observe a drift in production traffic patterns that your test suite no longer reflects. It is also valuable before a major release gate review to ensure the quality of your evaluation benchmarks. The prompt requires a representative sample of your golden dataset as input, along with the original output schema and any known failure mode taxonomies. It is not a replacement for automated schema validation or a one-time dataset generation tool; it is a periodic review mechanism for ongoing curation.

Do not use this prompt for initial dataset creation, for real-time production filtering, or as a substitute for human judgment in high-risk domains. The recommendations it produces are advisory and require human review, especially for bias and fairness findings. If your dataset is small (fewer than 50 examples), a manual review is more effective. The prompt works best when integrated into a scheduled maintenance workflow, such as a quarterly dataset audit, where its output can be tracked in your issue management system and acted upon by the team.

The prompt below is designed to be dropped into your existing QA or MLOps pipeline. It expects a batch of golden dataset records, your current output contract, and any known failure mode taxonomies. The model will review each example for staleness, bias, coverage gaps, and schema drift, then return structured recommendations with priority scores and explicit deprecation candidates. Use this template as a starting point—adjust the constraints and output schema to match your team's specific review rubric and risk tolerance.

textCopy
You are a dataset curator responsible for maintaining the quality of a golden regression test suite used to evaluate AI prompt behavior.

Review the following golden dataset examples against the provided output contract, failure mode taxonomy, and coverage requirements. For each example, assess:
- Staleness: Is the example still representative of current production traffic?
- Bias risk: Does the example encode or reinforce harmful stereotypes or representational gaps?
- Coverage contribution: Does this example test a unique scenario, or is it redundant?
- Schema compliance: Does the expected output match the current output contract?
- Edge case value: Does this example stress an important boundary condition?

Produce a curation report with the following structure.

[INPUT]
Golden dataset records to review, each containing an input, expected output, metadata, and provenance.

[OUTPUT_SCHEMA]
{
  "review_date": "ISO 8601 date of review",
  "reviewer": "model-and-version identifier",
  "summary": {
    "total_examples_reviewed": <integer>,
    "deprecation_candidates": <integer>,
    "augmentation_recommendations": <integer>,
    "coverage_gaps_identified": <integer>
  },
  "findings": [
    {
      "example_id": "<string>",
      "staleness_score": <float 0.0-1.0, where 1.0 is completely stale>,
      "bias_risk": "none|low|medium|high|critical",
      "bias_risk_rationale": "<string>",
      "coverage_contribution": "unique|complementary|redundant",
      "schema_compliant": <boolean>,
      "schema_drift_details": "<string or null>",
      "edge_case_value": "none|low|medium|high",
      "recommendation": "retain|deprecate|augment|review",
      "priority": "P0|P1|P2|P3",
      "rationale": "<string explaining the recommendation>"
    }
  ],
  "coverage_gaps": [
    {
      "gap_category": "<string>",
      "description": "<string>",
      "severity": "low|medium|high|critical",
      "suggested_example_count": <integer>,
      "generation_template": "<string describing how to create examples for this gap>"
    }
  ],
  "deprecation_candidates": [
    {
      "example_id": "<string>",
      "deprecation_reason": "<string>",
      "replacement_priority": "P0|P1|P2|P3",
      "migration_notes": "<string>"
    }
  ]
}

[CONSTRAINTS]
- Do not recommend deprecation without a clear reason tied to staleness, redundancy, or schema violation.
- Flag bias risks even if the example appears technically correct.
- For each coverage gap, provide a concrete generation template that a separate prompt can use to create new examples.
- If an example is schema-noncompliant, mark it as a deprecation candidate unless the schema itself is under review.
- Prioritize P0 for issues that would cause immediate regression test failures or safety incidents.
- Do not invent examples or coverage gaps not supported by the provided data.

[EXAMPLES]
Input example:
{
  "example_id": "ex-0042",
  "input": "What is the return policy for electronics?",
  "expected_output": "Electronics can be returned within 30 days with original receipt.",
  "metadata": { "source": "prod_trace", "capture_date": "2024-01-15", "persona": "customer" }
}

Output example:
{
  "example_id": "ex-0042",
  "staleness_score": 0.2,
  "bias_risk": "none",
  "bias_risk_rationale": "Generic customer query with no demographic or identity dimensions.",
  "coverage_contribution": "complementary",
  "schema_compliant": true,
  "schema_drift_details": null,
  "edge_case_value": "low",
  "recommendation": "retain",
  "priority": "P3",
  "rationale": "Example is recent, schema-compliant, and covers a common customer intent. No action required."
}

[RISK_LEVEL]
High. Golden dataset quality directly impacts regression test reliability. Incorrect deprecation can mask real regressions. Missed bias can encode harmful behavior into evaluation benchmarks. All P0 and P1 recommendations must be reviewed by a human QA lead before execution.

Adapt this template by replacing the output schema with your team's specific review dimensions. If your golden dataset includes multi-turn conversations, add a turn_count and context_dependency field to the findings. For teams using LLM judges downstream, include a judge_calibration_risk flag that warns when an example might produce flaky automated evaluations. Always route P0 and P1 deprecation candidates through a human approval step before removing them from the regression suite—automated dataset pruning without review can silently degrade test coverage.

The Dataset Curation Review Prompt is designed to run as a scheduled or triggered job within a golden dataset management pipeline, not as a one-off chat interaction. It expects a batch of examples—typically serialized as JSON or CSV—along with metadata about the dataset's intended use, schema version, and coverage requirements. The prompt's output is a structured curation report containing priority-scored recommendations, deprecation candidates, and gap flags. This report should be treated as a machine-readable artifact that downstream systems can parse and act on, not just a human-readable summary.

Wire the prompt into an application by first constructing the input payload from your dataset registry. The [INPUT] placeholder should receive a serialized array of examples, each with fields like id, input, expected_output, created_date, tags, and schema_version. The [CONTEXT] placeholder should carry dataset-level metadata: total example count, coverage targets, known bias dimensions, and the current schema contract. After calling the model, validate the output against a strict JSON schema that enforces the expected structure: a recommendations array where each item has example_id, action (one of keep, deprecate, review, augment), priority_score (0.0–1.0), rationale, and affected_dimension. Reject and retry any response that fails schema validation, contains hallucinated example IDs not present in the input, or produces priority scores outside the defined range. Log every validation failure for later prompt debugging.

For high-stakes datasets used in safety-critical or regulated workflows, insert a human review gate between the model's output and any automated dataset mutation. The curation report should be surfaced in a review UI that lets a QA lead accept, reject, or modify each recommendation before changes are applied to the golden set. Track which recommendations were overridden and why, feeding that signal back into prompt improvement and evaluator calibration. Avoid wiring this prompt directly to automated dataset deletion or modification without human approval when the golden set gates production model behavior. For model choice, prefer a capable instruction-following model with strong JSON output discipline; if using a model prone to hallucinated IDs, add a post-processing step that cross-references every example_id against the input payload and strips invalid entries before presenting results to a reviewer.