Golden Dataset Quality Scorecard Prompt

This prompt is designed for QA leads and ML engineers who own the quality of golden datasets used in regression testing. Its primary job is to evaluate a dataset snapshot across five critical dimensions—completeness, correctness, consistency, currency, and coverage—and produce a structured, actionable scorecard. The ideal user is someone preparing for a release gate review, conducting a quarterly dataset audit, or onboarding a new golden set from a different team. They need a standardized, repeatable method to assess dataset health before it becomes the foundation for critical model evaluations.

Use this prompt when you already have a dataset to evaluate and can provide rich metadata about its intended use, schema, and source. The prompt assumes you can supply a representative sample of records, the expected output contract, and the original collection criteria. It is not a prompt for generating new test cases, evaluating individual model outputs, or performing real-time production monitoring. The output is a structured scorecard with dimension-level scores, specific findings, and a regression risk indicator. To wire this into an application, you should validate the scorecard's JSON schema before accepting it, log the results for auditability, and flag any 'High' regression risk for mandatory human review before the dataset is promoted to a release gate.

Do not use this prompt as a substitute for evaluating the model itself. A high-quality golden dataset is a necessary but insufficient condition for a reliable regression suite. The scorecard tells you if your test data is trustworthy, not if your prompt is correct. After receiving the scorecard, your next step should be to address any critical findings, re-run the evaluation to confirm improvement, and then proceed with regression testing against the validated dataset. Avoid the temptation to use this prompt on datasets that are still under active construction or lack a stable schema, as the findings will be noisy and the recommendations premature.

This prompt template is designed to be pasted directly into your evaluation harness. It instructs the model to act as a QA auditor and produce a structured scorecard for a golden dataset. The template uses square-bracket placeholders that you must replace with your specific dataset metadata, sample records, and scoring rubric before execution. The output is a JSON report card with dimension scores, specific findings, and prioritized improvement recommendations.

textCopy
You are a QA lead auditing a golden dataset used for regression testing AI prompts. Your task is to evaluate the dataset's overall health and produce a structured quality scorecard.

## DATASET CONTEXT
- Dataset Name: [DATASET_NAME]
- Dataset Description: [DATASET_DESCRIPTION]
- Total Records: [TOTAL_RECORDS]
- Creation Date: [CREATION_DATE]
- Last Updated: [LAST_UPDATED]
- Intended Use: [INTENDED_USE]

## SAMPLE RECORDS
Below is a representative sample of [SAMPLE_SIZE] records from the dataset. Each record includes an input, expected output, and any annotations.

[SAMPLE_RECORDS]

## EVALUATION DIMENSIONS
Score each dimension from 1 (critical issues) to 5 (excellent). For each dimension, provide specific evidence from the sample records and a brief justification.

1. **Completeness**: Are all required fields populated? Are there missing inputs, outputs, or annotations? Are edge cases represented?
2. **Correctness**: Do the expected outputs accurately reflect what a correct model response should be? Are there factual errors, mislabelings, or incorrect annotations?
3. **Consistency**: Are similar inputs paired with similar expected outputs? Are annotation standards applied uniformly? Are there contradictory examples?
4. **Currency**: Does the dataset reflect current product behavior, schema versions, and domain knowledge? Are there stale or deprecated examples?
5. **Coverage**: Does the dataset span the full range of expected inputs, user personas, difficulty levels, and failure modes? Are there obvious gaps?

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "dataset_name": "string",
  "evaluation_date": "YYYY-MM-DD",
  "overall_health_score": number (1-5),
  "dimension_scores": {
    "completeness": {
      "score": number (1-5),
      "findings": ["string"],
      "evidence": ["string"]
    },
    "correctness": {
      "score": number (1-5),
      "findings": ["string"],
      "evidence": ["string"]
    },
    "consistency": {
      "score": number (1-5),
      "findings": ["string"],
      "evidence": ["string"]
    },
    "currency": {
      "score": number (1-5),
      "findings": ["string"],
      "evidence": ["string"]
    },
    "coverage": {
      "score": number (1-5),
      "findings": ["string"],
      "evidence": ["string"]
    }
  },
  "critical_issues": [
    {
      "severity": "high" | "medium" | "low",
      "dimension": "string",
      "description": "string",
      "affected_records": ["record_ids or indices"],
      "recommendation": "string"
    }
  ],
  "improvement_recommendations": [
    {
      "priority": number (1=highest),
      "dimension": "string",
      "action": "string",
      "expected_impact": "string"
    }
  ],
  "regression_risk_indicators": [
    {
      "risk": "string",
      "likelihood": "high" | "medium" | "low",
      "description": "string",
      "mitigation": "string"
    }
  ]
}

## CONSTRAINTS
- Base all findings on the provided sample records. Do not speculate about records you haven't seen.
- If the sample is too small to assess a dimension confidently, note this limitation and score conservatively.
- Flag any records that appear malformed, contradictory, or potentially harmful.
- Prioritize findings that would cause regression tests to produce false positives or false negatives.
- Be specific: reference record indices or IDs when describing issues.

To adapt this template, replace each square-bracket placeholder with concrete values from your dataset. For [SAMPLE_RECORDS], provide a representative subset that includes both typical examples and known edge cases. If your dataset is large, sample strategically across different input categories, difficulty levels, and creation dates. After running the prompt, validate the output JSON against the schema before ingesting it into your QA dashboard. For high-stakes datasets used in regulated workflows, always have a human QA lead review the scorecard and confirm critical findings before acting on recommendations.

The Quality Scorecard Prompt is not a one-off audit tool; it is designed to run as a scheduled evaluation gate within a regression testing or dataset management pipeline. The prompt expects a structured [DATASET_MANIFEST] containing metadata about the golden set (size, creation date, last review date, schema version, domain) and a [SAMPLE_RECORDS] payload with a representative subset of input-output pairs. The model returns a scored report card across five dimensions—completeness, correctness, consistency, currency, and coverage—along with improvement recommendations and regression risk indicators. To operationalize this, wrap the prompt in a harness that validates the output schema before accepting the scorecard, retries on malformed JSON, and logs every run for trend analysis over time.

Start by building a validation layer that checks the model's JSON output against a strict schema. The scorecard must include numeric scores (0.0–1.0) for each dimension, a list of findings with severity levels, and actionable recommendations. Use a JSON Schema validator or a library like Pydantic to reject responses missing required fields, containing out-of-range scores, or producing recommendations without corresponding findings. If validation fails, implement a single retry with the error message injected into the [CONSTRAINTS] field: 'Previous output failed validation: [ERROR_DETAILS]. Return valid JSON only.' After two consecutive failures, log the raw response and escalate for human review rather than silently accepting a degraded scorecard. For high-risk datasets used in regulated workflows, always require human approval on scorecards where any dimension falls below a configurable threshold (e.g., 0.7) or where regression risk indicators are flagged.

Integrate the scorecard into a CI/CD or scheduled job that runs weekly or on dataset version changes. Store each run's output in a versioned artifact store alongside the dataset manifest hash, so you can track quality trends and correlate score degradation with specific dataset updates. Use the regression risk indicators as triggers: if the prompt flags 'high risk of coverage regression' or 'currency drift detected,' automatically notify the QA lead and block promotion of that dataset version to production test suites. Avoid running the scorecard on the entire dataset in a single call; instead, sample 50–100 records stratified by difficulty, source, and age, and pass them as [SAMPLE_RECORDS]. This keeps latency predictable and prevents the model from producing shallow assessments on oversized inputs. For model choice, prefer a capable reasoning model (e.g., Claude 3.5 Sonnet or GPT-4o) that can handle multi-dimensional rubric evaluation without hallucinating scores; avoid smaller, faster models that may produce plausible but uncalibrated numbers.