Golden Dataset Construction Prompt Template

This prompt is designed for ML engineers and QA leads who are formalizing their LLM evaluation strategy. The core job-to-be-done is moving from ad-hoc, manual prompt testing to a repeatable, version-controlled regression suite. You should use this prompt when you have a well-defined task description, a collection of representative source examples, and specific coverage requirements (e.g., happy path, edge cases, error handling). The ideal user is someone who needs to produce a structured artifact—a golden dataset—that can be checked into a repository and used as a gate for future prompt or model changes.

To use this prompt effectively, you must provide three concrete inputs. First, a clear task description that defines what the AI system is supposed to do. Second, a set of source examples that demonstrate the desired input-output behavior; these act as seeds for the dataset. Third, explicit coverage requirements that specify the types of cases the dataset must include, such as standard scenarios, boundary conditions, and known failure modes. Without these inputs, the prompt cannot generate a reliable dataset. The prompt works best when the task has a stable definition and the expected outputs are unambiguous enough to be judged correct or incorrect.

Do not use this prompt when you have no source examples to seed from, or when the task definition is too ambiguous to specify expected outputs. If you are still exploring what the AI should do, or if the acceptance criteria are purely subjective, this prompt will produce a dataset that looks structured but fails under scrutiny. In such cases, start with exploratory testing or human annotation to establish a baseline of expected behavior. Once you have that baseline, return to this prompt to scale your examples into a comprehensive, tagged, and version-controlled evaluation suite.

This prompt template is the core instruction set you will send to an LLM to construct a golden dataset for regression testing. It is designed to be adapted, not used verbatim. The square-bracket placeholders represent the variables you must define before execution: the specific task the model under test performs, seed examples that demonstrate desired behavior, the dimensions of coverage you care about, and the exact output schema your test harness expects. Treat this template as a contract between your test infrastructure and the dataset generation step.

textCopy
You are an expert test dataset architect. Your task is to generate a structured golden dataset for regression testing an AI system that performs the following task:

[TASK_DESCRIPTION]

## Seed Examples
Here are a few examples of high-quality input-output pairs that demonstrate the expected behavior. Use these to calibrate your understanding of the task, but do not simply copy them into the dataset. Generate new, diverse examples.

[SEED_EXAMPLES]

## Coverage Dimensions
Generate examples that systematically cover the following dimensions. For each dimension, produce at least the minimum number of examples specified. Ensure the examples are realistic and varied within each dimension.

[COVERAGE_DIMENSIONS]

## Output Schema
Return a single JSON object with the following structure. Do not include any text outside the JSON object.

[OUTPUT_SCHEMA]

## Constraints
- Every input must be a realistic, self-contained request that could appear in production.
- Every expected_output must be a valid, complete response that strictly follows the task rules.
- The coverage_report must honestly assess which dimensions are covered and flag any gaps you could not fill.
- Do not generate examples that are trivial, nonsensical, or violate the task's safety policies.

To adapt this template, start by replacing [TASK_DESCRIPTION] with a precise, one-paragraph definition of what the system under test does. Include its input format, its output contract, and any critical behavioral rules. Next, populate [SEED_EXAMPLES] with 3–5 real input-output pairs that your team has manually verified as correct. These seeds anchor the model's understanding and prevent it from drifting into plausible-but-wrong patterns. Define [COVERAGE_DIMENSIONS] as a list of axes you need to test—common dimensions include input length, entity density, sentiment, ambiguity, edge-case keywords, and output complexity. For each dimension, specify a minimum example count to force adequate representation. Finally, replace [OUTPUT_SCHEMA] with the exact JSON schema your test harness expects, including fields for a unique example ID, the input, the expected output, metadata tags for each coverage dimension, and a coverage_report object that summarizes gaps. After generation, always validate the output against your schema and manually review a sample of examples before committing them to your regression suite.

This prompt is not a one-off generator. It is a dataset factory that should be wired into a structured pipeline. The ideal implementation treats each generated example as a versioned artifact with provenance: which prompt version produced it, which model, which reviewer approved it, and which test suite it belongs to. The harness must enforce that every example carries the required metadata tags ([TAG_1], [TAG_2], etc.) and conforms to the [OUTPUT_SCHEMA] before it enters the golden dataset. Without this harness, you will accumulate unvalidated examples that silently corrupt your regression suite.

Pipeline stages: (1) Generation — call the prompt with your domain context, schema, and coverage targets. (2) Structural validation — validate every output against the [OUTPUT_SCHEMA] using a JSON Schema validator or equivalent. Reject any example missing required fields (input, expected_output, tags, difficulty, category). (3) Content quality check — run a secondary LLM judge prompt that scores each example on clarity, realism, and tag accuracy. Flag examples scoring below your threshold for human review. (4) Deduplication — compute semantic similarity between new examples and existing golden dataset entries. Reject near-duplicates (cosine similarity > 0.92 on embedding vectors is a common starting threshold). (5) Human review queue — route flagged, borderline, and high-difficulty examples to a review interface. Reviewers approve, edit, or reject each example. (6) Ingestion — append approved examples to the versioned golden dataset file with a timestamp, prompt version, and reviewer ID.

Model choice and retries: Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Set temperature between 0.3 and 0.6 to balance diversity with consistency. Implement a retry loop with max 3 attempts for structural validation failures, feeding the validation error back into the retry prompt. If content quality checks fail after retries, escalate to human review rather than looping indefinitely. Tool use: For domain-specific terminology, wire in a RAG retrieval step that pulls definitions, taxonomies, and real-world examples from your internal knowledge base before generation. This grounds the synthetic examples in your actual domain language.

Version control and audit: Store the golden dataset in a version-controlled repository (git) as JSONL or Parquet files. Each commit should reference the prompt version and model used. Tag releases of the dataset that correspond to prompt evaluation gates. What to avoid: Do not auto-approve generated examples without human review for high-risk domains (healthcare, finance, legal). Do not skip deduplication — duplicate examples inflate pass rates and mask regressions. Do not generate examples for categories where you lack domain expertise to validate correctness. A golden dataset with plausible-but-wrong examples is worse than a smaller, verified one.