Golden Trace Coverage Gap Analysis Prompt

This prompt is designed for QA engineers and AI ops teams who need to systematically identify blind spots in their evaluation datasets by comparing production trace distributions against their existing golden dataset. The core job-to-be-done is to answer: 'What real-world inputs and failure modes are we not testing against?' Use this when you have a production trace store with sufficient volume to surface distribution patterns and a golden dataset that is supposed to represent your production surface area. The prompt produces a structured coverage gap report that prioritizes missing input categories, underrepresented edge cases, and concrete recommendations for expanding your golden dataset.

The ideal user has access to production trace metadata (intent clusters, input embeddings, tool call patterns, or error signatures) and can provide summary statistics rather than raw traces. You should have at least a week of production data to avoid sampling noise. The prompt works best when you can supply pre-computed distribution summaries—such as top intent categories with their frequencies, common failure mode clusters, and edge-case buckets—rather than expecting the model to compute statistics from raw logs. If your production traces are unlabeled or you lack any clustering or categorization layer, apply a trace clustering prompt first to generate the input categories this prompt requires.

Do not use this prompt when you need real-time anomaly detection or when you are investigating a single known regression. This is a strategic coverage analysis tool, not an operational alert. It is also not a substitute for statistical drift detection—use a dedicated distribution shift detection prompt if you need to quantify drift magnitude with statistical rigor. Avoid this prompt when your golden dataset is empty or trivially small; the output will be a generic 'add more examples' recommendation that provides no actionable signal. For high-risk regulated domains, always have a human reviewer validate the recommended golden dataset expansions before they become part of your release gate criteria, and ensure any production trace statistics you supply are scrubbed of PII and sensitive payloads before they reach the model.

This prompt template is designed to be copied directly into your prompt management system, evaluation harness, or orchestration layer. It expects a structured description of your golden dataset coverage and a summary of production trace distributions. The model will produce a coverage gap report identifying missing input categories, underrepresented edge cases, and recommended expansions to your golden dataset. All placeholders use square brackets and must be populated before execution.

textCopy
You are a QA engineer specializing in evaluation dataset coverage analysis for AI systems. Your task is to compare the coverage profile of a golden evaluation dataset against observed production trace distributions and identify blind spots that could allow regressions to reach users undetected.

## GOLDEN DATASET COVERAGE
[GOLDEN_DATASET_COVERAGE]

## PRODUCTION TRACE DISTRIBUTION
[PRODUCTION_TRACE_DISTRIBUTION]

## OUTPUT SCHEMA
Return a JSON object with the following structure:
{
  "coverage_gaps": [
    {
      "category": "string (the input category or dimension with insufficient coverage)",
      "gap_type": "missing | underrepresented | stale",
      "production_volume": "string (observed frequency in production: high, medium, low, or percentage)",
      "golden_coverage": "string (current coverage level: none, minimal, partial, or count of examples)",
      "risk_level": "critical | high | medium | low",
      "example_production_inputs": ["string (anonymized representative inputs from production)"],
      "recommended_expansion": "string (specific guidance on what examples to add to the golden dataset)",
      "estimated_examples_needed": "number (rough count of additional examples required)"
    }
  ],
  "coverage_summary": {
    "total_categories_identified": "number",
    "critical_gaps": "number",
    "high_gaps": "number",
    "medium_gaps": "number",
    "low_gaps": "number",
    "overall_coverage_score": "number (0-100, where 100 means production is fully covered)"
  },
  "priority_expansion_plan": [
    {
      "rank": "number",
      "category": "string",
      "rationale": "string (why this gap should be addressed first)"
    }
  ],
  "false_coverage_warnings": [
    {
      "category": "string",
      "warning": "string (cases where golden coverage exists but may not represent production reality)"
    }
  ]
}

## CONSTRAINTS
- Only report gaps supported by evidence in the provided distributions.
- Do not invent categories not present in the production trace distribution.
- If production trace data is insufficient to assess a dimension, flag it as "insufficient_data" rather than guessing.
- Prioritize gaps by risk: missing categories > underrepresented high-volume categories > stale examples.
- For each gap, provide at least one anonymized production input example.
- The overall_coverage_score must reflect weighted risk, not just raw category counts.
- If no significant gaps exist, return an empty coverage_gaps array and explain why in the coverage_summary.

## RISK LEVEL DEFINITIONS
- critical: Missing coverage for a high-volume or safety-sensitive input category.
- high: Missing coverage for a medium-volume category or severely underrepresented high-volume category.
- medium: Underrepresented coverage for a medium-volume category or stale examples for a high-volume category.
- low: Minor underrepresentation or stale examples for low-volume categories.

## EXAMPLES
[EXAMPLES]

## TOOLS
[TOOLS]

Adaptation guidance: Replace [GOLDEN_DATASET_COVERAGE] with a structured description of your evaluation dataset—include intent categories, input types, edge cases, and example counts per category. Replace [PRODUCTION_TRACE_DISTRIBUTION] with a summary of production trace distributions, including input categories, volume percentages, and any observed shifts over time. The [EXAMPLES] placeholder should contain 2-3 few-shot examples of correct coverage gap reports if your model benefits from demonstrations. The [TOOLS] placeholder is optional and should only be populated if you want the model to invoke external functions for statistical comparison or trace retrieval. If your production trace data is large, pre-aggregate it into distribution summaries rather than passing raw traces into the prompt. For high-risk domains where coverage gaps could mask safety regressions, route outputs with critical gaps to human review before accepting the expansion plan.

This prompt is designed to be called programmatically as part of a scheduled QA job or a CI/CD gate, not as a one-off chat interaction. The primary input is a structured comparison payload containing production trace distribution statistics and golden dataset coverage metadata. The harness should assemble this payload from your observability backend (e.g., querying a trace store for input category histograms) and your test case management system (e.g., reading golden dataset tags and counts). The model's output is a structured coverage gap report that downstream systems can parse to generate tickets, update dashboards, or block a release.

To integrate this into an application, wrap the prompt in a function that accepts [PRODUCTION_TRACE_STATS] and [GOLDEN_DATASET_COVERAGE] as structured JSON objects. Before calling the model, validate that both inputs contain the required fields: input_categories, sample_counts, edge_case_flags, and distribution_timestamps. After receiving the model's response, run a strict schema validator against the expected [OUTPUT_SCHEMA] to ensure the JSON contains coverage_gaps, missing_categories, underrepresented_edge_cases, and recommended_expansions arrays. If validation fails, retry once with the error message injected into the [CONSTRAINTS] field. Log every input, output, and validation result to your prompt observability platform, tagging the trace with prompt_id: golden-trace-coverage-gap and version: [PROMPT_VERSION] for later audit.

For high-stakes environments where a missed coverage gap could allow a regression to reach production, implement a human review step for any report where the max_severity field is critical or where the number of missing_categories exceeds a configurable threshold. Route these reports to a QA lead via your incident management tool. Avoid using this prompt on raw trace data without pre-aggregation; sending individual traces will exceed context windows and produce unreliable results. Instead, pre-compute distribution statistics using a batch job that runs hourly or daily, then feed the aggregated stats into this analysis prompt. The model choice should favor a frontier model with strong JSON mode and long-context capabilities, as the coverage comparison requires precise reasoning over structured data.