Production Trace Sampling for QA Prompt

QA leads and AI ops engineers face a hard problem: production generates thousands of traces daily, but only a fraction can be promoted into regression test suites. Random sampling misses high-risk edge cases. Manual selection introduces bias. This prompt produces a stratified sample covering high-risk, high-volume, and edge-case traces with explicit coverage justification. Use it when you need to convert raw production traces into a defensible regression dataset that catches real-world failures before your next prompt deployment.

The prompt requires a batch of raw production traces as [INPUT_TRACES], each containing the full request-response payload, any tool calls, user feedback signals, and latency metadata. You must also supply [STRATIFICATION_DIMENSIONS] that define your sampling axes—typically risk score, volume bucket, error rate, and novelty. The prompt expects [SAMPLE_SIZE] to cap the output and [COVERAGE_TARGETS] to specify minimum representation for each stratum. Without these inputs, the model cannot produce a statistically meaningful sample. Do not use this prompt when your trace volume is below 100 records, when traces lack consistent metadata, or when you need real-time sampling rather than batch curation.

After generating the sample, you must validate representativeness by comparing the output distribution against your original trace population using the provided harness checks. The prompt includes explicit coverage justification, but you should still run statistical tests on the sampled set before promoting traces into your golden dataset. Avoid the temptation to manually curate the output—manual overrides reintroduce the selection bias this prompt is designed to eliminate. If the prompt flags gaps in your stratification dimensions, expand your trace metadata before re-running rather than accepting an under-sampled stratum.

This prompt template is designed to be copied directly into your AI harness. It instructs the model to act as a QA architect, selecting a subset of production traces that will form a high-signal regression test suite. The template uses square-bracket placeholders for all dynamic inputs. Before sending this prompt to a model, you must replace every placeholder with concrete values from your observability and trace storage systems. The prompt forces the model to justify its selections across multiple coverage dimensions—high-risk, high-volume, and edge-case traces—and to output a structured report that your pipeline can parse and validate.

textCopy
You are a QA architect responsible for building a regression test suite from production traces.

Your task is to select a stratified sample of traces from the provided [TRACE_POOL] that maximizes coverage for high-risk, high-volume, and edge-case scenarios. You must produce a structured report that justifies each selection and explains how the sample improves the existing golden dataset.

# INPUTS
- Trace Pool: [TRACE_POOL]
- Existing Golden Dataset: [GOLDEN_DATASET]
- Risk Taxonomy: [RISK_TAXONOMY]
- Sample Size Target: [SAMPLE_SIZE]
- Coverage Dimensions: [COVERAGE_DIMENSIONS]

# OUTPUT_SCHEMA
Return a valid JSON object with the following structure:
{
  "sample_traces": [
    {
      "trace_id": "string",
      "selection_rationale": "string",
      "coverage_dimensions_addressed": ["string"],
      "risk_level": "high|medium|low",
      "volume_representativeness": "high|medium|low",
      "is_edge_case": true|false,
      "golden_dataset_gap": "string"
    }
  ],
  "coverage_summary": {
    "dimensions_covered": ["string"],
    "dimensions_missed": ["string"],
    "risk_distribution": {"high": 0, "medium": 0, "low": 0},
    "edge_case_count": 0
  },
  "exclusion_notes": ["string"]
}

# CONSTRAINTS
- Select exactly [SAMPLE_SIZE] traces, no more, no less.
- Prioritize traces that fill gaps in the [GOLDEN_DATASET].
- Ensure at least one trace per category in [RISK_TAXONOMY].
- Flag any trace that appears to contain personally identifiable information (PII) and exclude it, noting the reason in exclusion_notes.
- If the trace pool is insufficient to meet the coverage dimensions, state which dimensions are missed in coverage_summary.dimensions_missed.
- Do not invent trace data. Only select from the provided [TRACE_POOL].

# EXAMPLES
[EXAMPLES]

# RISK_LEVEL
[RISK_LEVEL]

To adapt this template, start by populating the [TRACE_POOL] with a JSON array of trace objects from your production store. Each trace object must include at least a trace_id, the user's input, the model's output, any tool calls, and a timestamp. The [GOLDEN_DATASET] should be a summary or full list of existing test cases, so the model can identify gaps. The [RISK_TAXONOMY] is your organization's list of failure categories, such as ["hallucination", "refusal", "tool_error", "format_violation"]. The [SAMPLE_SIZE] is an integer, typically between 20 and 100 for a manageable review cycle. The [COVERAGE_DIMENSIONS] are the axes you care about, like ["intent_category", "user_segment", "tool_used", "session_length"]. The [EXAMPLES] placeholder should contain one or two few-shot examples of a correctly selected trace with its rationale, to calibrate the model's output quality. The [RISK_LEVEL] placeholder should contain your organization's definition of what constitutes a high, medium, or low-risk trace. After populating these placeholders, validate the model's JSON output against the schema before promoting any trace to your test suite. A human QA lead should review the final sample to confirm that the selections are sensible and that no PII has leaked through.

This prompt is designed to be called programmatically as part of a scheduled QA pipeline, not as a one-off chat interaction. The harness should fetch a batch of recent production traces from your observability store (e.g., LangSmith, Arize, or a custom trace database), pass them to the model along with the prompt template, and receive a structured JSON payload containing the stratified sample, coverage justification, and exclusion reasons. The output must be validated before any traces are promoted into your golden dataset or regression test suite.

The implementation should enforce a strict contract. First, validate that the returned JSON matches the [OUTPUT_SCHEMA] exactly—reject any response missing the sample, coverage_justification, or exclusions keys. Second, verify that the number of traces in the sample matches the requested [SAMPLE_SIZE] and that each trace ID exists in the input batch. Third, check that the coverage justification references the required strata (high-risk, high-volume, edge-case) and that no stratum is empty without an explicit reason. If any validation fails, retry once with the same input and an added instruction to correct the specific violation. After a second failure, log the raw response and flag the batch for human review. Use a model with strong JSON mode and low temperature (0.0–0.1) to maximize output stability.

For production deployment, wrap this prompt in a job that runs on a regular cadence—daily for high-traffic systems, weekly for lower-volume ones. Log every sampling run: input trace count, output sample size, validation pass/fail, retry count, and the coverage justification summary. Store the promoted traces in a versioned golden dataset with metadata linking back to the sampling run ID. Before the sampled traces become active regression test cases, route them through a human review queue where a QA lead can accept, reject, or modify the selection. Never auto-promote traces into a regression suite without this review step—the prompt identifies candidates, but a human must confirm they represent meaningful coverage improvements and are free of PII or noise.