Golden Trace Capture Prompt for Production Logs

This prompt is built for observability engineers and AI ops teams who need to bootstrap or expand a regression test suite with real-world data. The core job-to-be-done is converting raw, unstructured production traces into a curated, sanitized golden dataset. The ideal user has access to production logs, understands the system's expected behavior, and needs to create test records that represent correct behavior, interesting edge cases, or instructive failures—not synthetic approximations. You should use this prompt when your team is moving from ad-hoc testing to structured regression gates and needs high-signal examples that reflect actual user interactions and system responses.

The prompt requires several concrete inputs to function reliably. You must provide a set of raw production traces in a structured format (e.g., JSONL with trace_id, input, output, metadata). You must also supply a [SELECTION_CRITERIA] block that defines what makes a trace valuable for your test suite—this could include correctness markers, latency thresholds, tool-call patterns, or specific error codes. A [SANITIZATION_RULES] block is required to specify which fields to redact (PII, secrets, internal IPs) and how to replace them. The prompt expects an [OUTPUT_SCHEMA] that defines the shape of each golden record, including fields for the sanitized input, expected output, trace metadata, inclusion justification, and a privacy redaction log. Without these inputs, the prompt will produce inconsistent or unsafe results.

Do not use this prompt when you lack access to production logs or when the logs do not contain enough signal to distinguish correct behavior from noise. It is also inappropriate for generating synthetic test data from scratch—use the Golden Example Generation Prompt Template for that purpose. Avoid using this prompt on traces that contain unredactable sensitive data or where the sanitization process would strip away the very details that make the trace valuable for testing. Finally, this prompt is not a replacement for human review of the resulting golden dataset; every selected trace should be reviewed by an engineer who understands the system's expected behavior before it enters the regression suite.

The following prompt template is designed to be dropped directly into your observability pipeline or a manual review workflow. It instructs the model to act as a strict curator, selecting only traces that meet specific criteria for correctness, diversity, and safety. The template uses square-bracket placeholders for all dynamic inputs, ensuring you can wire it into a script, a Retool UI, or a CI/CD harness without modification.

codeCopy
You are a senior QA engineer curating a golden dataset for regression testing. Your task is to analyze a batch of production traces and select the best candidates.

## INPUT DATA
[PRODUCTION_TRACE_BATCH]

## SELECTION CRITERIA
Analyze each trace and select candidates that meet these rules:
1. **Correctness**: The final output correctly fulfills the user's intent without factual errors or hallucinations.
2. **Diversity**: The batch should represent a mix of simple, complex, and edge-case interactions. Avoid selecting near-duplicate traces.
3. **Interesting Failure**: Include traces where the model correctly refused, asked for clarification, or hit a known limitation gracefully. Do not include traces with unhandled exceptions or toxic outputs.
4. **Sanitization**: Before inclusion, redact all personally identifiable information (PII), API keys, and internal hostnames from the `input` and `output` fields. Replace them with placeholders like `[REDACTED_EMAIL]`.

## OUTPUT FORMAT
Return a single JSON object with the following schema:
{
  "selected_traces": [
    {
      "trace_id": "string",
      "sanitized_input": "string",
      "sanitized_output": "string",
      "inclusion_reason": "string (one of: 'correct_behavior', 'edge_case', 'graceful_failure')",
      "difficulty_rating": "string (one of: 'easy', 'medium', 'hard')"
    }
  ],
  "rejected_trace_ids": ["string"],
  "rejection_reasons": {
    "trace_id": "string (brief reason for rejection)"
  }
}

## CONSTRAINTS
- Do not select more than [MAX_TRACES] traces.
- If no traces meet the criteria, return an empty `selected_traces` array.
- Ensure the `sanitized_input` and `sanitized_output` do not contain raw user data.

To adapt this template, replace [PRODUCTION_TRACE_BATCH] with your JSONL or structured log data. The [MAX_TRACES] variable should be set based on your review budget—typically between 5 and 20 for a manual review cycle. The output schema is designed to be machine-readable so you can pipe the results directly into a database or a test harness. If your traces contain multi-turn conversations, ensure the sanitized_input field includes the full message history to preserve the context required for regression testing.

Before deploying this prompt, validate its output against a known set of traces. A common failure mode is the model being too permissive, selecting traces that contain subtle hallucinations or formatting errors. To guard against this, implement a post-processing validation step in your harness that checks for the presence of redacted placeholders and rejects any trace where the inclusion_reason is not from the allowed enum list. For high-risk domains, always require a human to approve the final batch before it is committed to the golden dataset.

The Golden Trace Capture Prompt is designed to run as a batch or streaming post-processing step in your observability pipeline, not as a real-time user-facing prompt. After your AI application logs request-response pairs, tool calls, and metadata, this prompt reviews candidate traces and selects those worth promoting to your golden regression dataset. The harness must enforce three gates before any trace enters the golden set: schema validation (the output matches the expected JSON structure with required fields like trace_id, inclusion_justification, and sanitized_output), privacy verification (PII and secrets are redacted, with a redaction_log present), and human approval (a reviewer confirms the trace represents correct behavior, an interesting edge case, or a failure mode worth testing).

Wire the prompt into a message queue or batch job that triggers after a configurable sampling window (e.g., every 1,000 production requests or every hour). The harness should: (1) fetch raw traces from your logging store (e.g., OpenTelemetry, Datadog, custom log tables), (2) assemble the prompt with [TRACE_BATCH] containing up to 20 candidate traces in a structured JSON array, [SELECTION_CRITERIA] describing what makes a trace valuable (e.g., 'correct complex reasoning', 'edge case with ambiguous input', 'failure with informative error'), and [OUTPUT_SCHEMA] specifying the exact JSON shape, (3) call the model with response_format set to json_object and a low temperature (0.0–0.2) for deterministic selection, (4) validate the output against a JSON Schema that requires selected_traces as an array of objects with trace_id, sanitized_input, sanitized_output, inclusion_category, inclusion_justification, redaction_log, and confidence_score, (5) run a secondary PII scanner (e.g., Presidio, AWS Comprehend, or a regex-based secret detector) on the sanitized_input and sanitized_output fields to catch any leaks the model missed, and (6) write approved traces to a staging table with a review_status of pending_human_review. Use exponential backoff with a max of 3 retries if the model returns malformed JSON or fails schema validation; after 3 failures, log the batch for manual triage.

For model choice, prefer a model with strong structured output support and low latency for batch processing—GPT-4o or Claude 3.5 Sonnet work well. If cost is a concern, use a smaller model (e.g., GPT-4o-mini) for initial filtering and escalate ambiguous candidates to a larger model. Do not skip human review for traces flagged as inclusion_category: 'failure_mode' or confidence_score < 0.8; these require an engineer to verify the trace is reproducible and worth testing. Wire the human review step into your existing ticketing or review queue system (e.g., Jira, Linear, a custom admin panel) with a link to the raw trace and the model's justification. Once approved, move the trace to the golden dataset and update the review_status to approved. Track metrics: selection rate (traces selected per batch), validation failure rate (how often the model output fails schema or PII checks), human rejection rate (how often reviewers override the model), and golden set growth rate over time. These metrics will surface when the prompt needs tuning—for example, if the selection rate drops below 5%, your criteria may be too strict, or if the human rejection rate exceeds 30%, the model is over-selecting noisy traces.