Golden Dataset Quality Metric Dashboard Prompt

This prompt is for evaluation platform teams who need to monitor the health of their golden datasets over time. The job-to-be-done is computing aggregate quality metrics—annotation consistency, coverage breadth, staleness, duplicate rate, and difficulty distribution—and producing a dashboard-ready summary with trend alerts. The ideal user is an MLOps engineer, evaluation lead, or QA architect responsible for maintaining regression suites where data quality directly impacts the trustworthiness of automated evaluations. You should use this prompt when you have a structured golden dataset and need a quantitative, repeatable health report that can feed into a monitoring dashboard or a periodic review cadence.

The prompt expects a structured dataset manifest as input, typically including test case IDs, input texts, expected outputs, metadata tags, creation dates, and difficulty labels. It works best when the dataset is already in a machine-readable format (JSON, CSV, or a data frame description) and when you have defined what 'stale' means in your context—for example, test cases older than 90 days or those referencing deprecated API versions. The output is a structured metrics payload with fields for each quality dimension, trend indicators comparing current values to a prior baseline, and a prioritized list of alerts for dimensions that have crossed configurable thresholds. This is not a prompt for building the dataset itself; use the Golden Dataset Construction Prompt Template for that. It is also not a prompt for running regression tests; use the Regression Test Suite Runner Prompt Template when you need to execute test cases against a prompt version.

Do not use this prompt when the dataset is too small for statistical aggregation (fewer than ~50 test cases), when you lack a prior baseline for trend comparison, or when the primary need is debugging individual test case failures rather than assessing dataset-level health. The prompt is designed for aggregate monitoring, not root-cause analysis of specific regressions. For failure categorization, use the Regression Failure Categorization Prompt Template. Before deploying this prompt into a production dashboard pipeline, ensure you have a human review step for the first few runs to calibrate alert thresholds and verify that the computed metrics align with your team's operational definitions of quality. Staleness and difficulty distribution are particularly sensitive to domain-specific definitions that the model may not infer correctly without explicit constraints.

This template is designed to be the core instruction set for an LLM tasked with auditing a golden evaluation dataset. It moves beyond simple pass/fail counts to compute structural health metrics like annotation consistency, coverage breadth, staleness, and difficulty distribution. The prompt is structured to accept a serialized dataset snapshot and a set of configurable thresholds, producing a structured JSON report that can be directly ingested by a monitoring dashboard or CI/CD pipeline. Use this template as the starting point for your dataset health agent, adapting the placeholders to match your specific taxonomy, schema, and alerting rules.

markdownCopy
You are an evaluation dataset auditor. Your task is to analyze the provided golden dataset and compute aggregate quality metrics. You will produce a dashboard-ready JSON summary that includes trend alerts based on historical baselines.

## INPUT
- **Dataset:** [DATASET_JSON]
- **Historical Baseline:** [BASELINE_METRICS_JSON]
- **Current Timestamp:** [CURRENT_DATE]

## METRICS TO COMPUTE
1.  **Annotation Consistency:** For any input that appears more than once, check if the `expected_output` is identical. Calculate the ratio of consistent duplicates to total duplicates.
2.  **Coverage Breadth:** Map each test case to the intent categories defined in [CATEGORY_TAXONOMY]. Report the number of categories with zero test cases.
3.  **Staleness:** Identify test cases where the `last_verified_date` is older than [STALENESS_THRESHOLD_DAYS] days. Calculate the percentage of stale cases.
4.  **Duplicate Rate:** Calculate the percentage of test cases that are exact duplicates (same input and expected output) beyond the first occurrence.
5.  **Difficulty Distribution:** Classify each test case into a difficulty tier (e.g., 'easy', 'medium', 'hard') based on the criteria in [DIFFICULTY_RUBRIC]. Report the percentage distribution.

## TREND ANALYSIS
Compare the current metrics against the [BASELINE_METRICS_JSON]. For any metric where the change exceeds its respective threshold defined in [ALERT_THRESHOLDS], generate a trend alert.

## OUTPUT_SCHEMA
Return a single valid JSON object with this exact structure:
{
  "dataset_id": "string",
  "evaluation_timestamp": "string",
  "metrics": {
    "annotation_consistency": {
      "value": "number (0-1)",
      "total_duplicate_groups": "number",
      "inconsistent_groups": "number"
    },
    "coverage_breadth": {
      "value": "number (0-1)",
      "covered_categories": "number",
      "uncovered_categories": ["string"],
      "total_categories": "number"
    },
    "staleness": {
      "value": "number (0-1)",
      "stale_case_count": "number",
      "total_cases": "number"
    },
    "duplicate_rate": {
      "value": "number (0-1)",
      "duplicate_case_count": "number",
      "total_cases": "number"
    },
    "difficulty_distribution": {
      "easy_pct": "number",
      "medium_pct": "number",
      "hard_pct": "number"
    }
  },
  "trend_alerts": [
    {
      "metric": "string",
      "current_value": "number",
      "baseline_value": "number",
      "change_pct": "number",
      "severity": "'info' | 'warning' | 'critical'",
      "message": "string"
    }
  ]
}

## CONSTRAINTS
- Do not modify the dataset. Only analyze it.
- If a metric cannot be computed (e.g., no duplicates exist), set its value to 1.0 (perfect score) and explain in a note.
- Use the exact category names from [CATEGORY_TAXONOMY].
- If the baseline is empty, omit the `trend_alerts` array.

To adapt this template, start by replacing the core placeholders. [DATASET_JSON] should be your serialized test suite, typically an array of objects each containing input, expected_output, category, difficulty, and last_verified_date. [CATEGORY_TAXONOMY] is a critical input—provide it as a flat list of strings representing every valid intent or feature area your tests should cover. The [DIFFICULTY_RUBRIC] needs to be a clear, deterministic set of rules (e.g., 'Hard: requires multi-step reasoning or external knowledge'). Finally, [ALERT_THRESHOLDS] should be a JSON object mapping metric names to their maximum allowable percentage change before an alert fires (e.g., {"staleness": 0.05} for a 5% threshold). If you lack a historical baseline, pass an empty object {} for [BASELINE_METRICS_JSON] to skip trend analysis.

Before deploying this prompt into an automated pipeline, you must implement a validation layer around the raw model output. The JSON structure is strict, but the model can still produce malformed types or miss required fields. A post-processing script should validate the output against the defined schema, retry with an error message if validation fails, and log any metric that falls outside a 0-1 range. For high-stakes environments where a faulty dashboard could mask a real regression, route outputs with critical severity alerts for human review before they are published. This prompt is a measurement tool, not a decision-maker; its value comes from surfacing anomalies reliably, not from acting on them autonomously.

This prompt is designed to be called by an evaluation platform's backend service, not directly by a human. It expects a structured payload representing a snapshot of the golden dataset's metadata and recent activity. The harness should assemble this payload from the platform's database, including annotation logs, test case metadata, and run histories. Because the output is a dashboard-ready summary, the harness must validate the JSON structure before storing or displaying it. A failed validation should trigger a retry with a more constrained schema reminder, not a silent fallback.

Wire the prompt into a scheduled job (e.g., a cron or a workflow orchestrator like Temporal or Airflow) that runs daily or after significant dataset changes. The harness should: (1) Query the database for all test cases, their annotations, last-modified timestamps, difficulty tags, and recent evaluation run results. (2) Assemble the [DATASET_SNAPSHOT] input as a JSON object containing arrays of test cases, annotation events, and run logs. (3) Call the LLM with the prompt template, setting response_format to json_object and providing the expected output schema. (4) Validate the returned JSON against a strict schema that checks for required fields like overall_health_score, metrics, and alerts. If validation fails, retry once with an explicit error message injected into the prompt's [CONSTRAINTS] field. (5) On success, upsert the result into a dataset_health_snapshots table and check for any alerts with a severity of high or critical. If found, publish an event to the team's incident channel (e.g., Slack, PagerDuty).

Model choice matters here. Use a model with strong JSON mode and long-context capabilities, such as gpt-4o or claude-3.5-sonnet, because the input snapshot can be large. Avoid models known to truncate or summarize structured inputs aggressively. For cost control, consider summarizing raw annotation logs into pre-aggregated counts before passing them to the prompt, but ensure the aggregation logic is tested independently. Log every prompt call, including the input snapshot hash, the raw LLM response, and the validation result, to an append-only audit table. This log is essential for debugging metric drift or alert fatigue. Do not use the LLM's output to directly trigger automated dataset mutations (e.g., deleting test cases flagged as stale); always route such recommendations to a human review queue with the supporting evidence highlighted.