Golden Dataset Generation Prompt for Disambiguation Testing

This prompt is for RAG evaluation engineers who need to build a regression test suite for their query disambiguation pipeline. The job-to-be-done is generating a diverse, labeled set of ambiguous queries paired with their correct resolutions. Without a golden dataset, you cannot measure whether changes to your disambiguation logic—such as a new entity linker or a rewritten system prompt—improve or regress real-world performance. The ideal user is someone who already has a working disambiguation system and needs a structured way to test it against known ambiguity types, including entity collisions, homonyms, temporal vagueness, and underspecified constraints.

Use this prompt when you need to create a benchmark that covers multiple ambiguity categories in a single run. It is designed to produce a structured output with input queries, expected resolutions, and ambiguity type labels, making it directly usable in evaluation frameworks like custom test runners or LLM-as-judge pipelines. You should provide a [DOMAIN_CONTEXT] that describes the knowledge base or product surface your RAG system operates over, and a [SAMPLE_QUERIES] list to ground the generation in realistic user language. The prompt works best when you also supply [AMBIGUITY_CATEGORIES] to control the distribution of test cases. Do not use this prompt if you lack a clear domain definition; without it, the generated ambiguities will be generic and fail to stress-test your specific retrieval surface.

This prompt is not a replacement for production traffic sampling. It generates synthetic test cases that complement, but do not substitute for, real user queries captured from logs. After generating the dataset, you must review the outputs for plausibility and alignment with your actual knowledge base. Treat the generated dataset as a starting point for curation, not a final artifact. The next step is to run your disambiguation system against the generated queries, compare outputs to the expected resolutions, and compute per-category accuracy metrics. Avoid the mistake of using this dataset for model training; it is an evaluation artifact, and training on it would invalidate its use as an independent test set.

This prompt template is the core engine for generating a high-quality evaluation dataset. It instructs the model to act as a disambiguation test designer, producing diverse, realistic examples that cover the specific failure modes you need to test in your retrieval system. The output is a structured JSON array, making it immediately ingestible by your evaluation harness. The template uses square-bracket placeholders for all variable components, allowing you to control the domain, ambiguity types, and output volume without rewriting the core instruction.

textCopy
You are a test designer creating a golden dataset for evaluating a query disambiguation system in a RAG pipeline. Your goal is to generate a diverse set of ambiguous user queries paired with their correct disambiguations.

[DOMAIN_CONTEXT]

Generate [NUM_EXAMPLES] examples. Each example must represent a distinct, realistic user query that is ambiguous in a way that would cause a retrieval system to return mixed or irrelevant results.

For each example, you must output a JSON object with the following keys:
- "ambiguous_query": The raw, ambiguous user query as a string.
- "correct_disambiguation": A clear, unambiguous rewrite of the query that resolves the ambiguity.
- "ambiguity_type": A label from this controlled vocabulary: [AMBIGUITY_TYPES].
- "explanation": A brief, one-sentence explanation of why the original query is ambiguous and how the rewrite resolves it.

[CONSTRAINTS]
- Ensure the examples cover a uniform distribution across the specified ambiguity types.
- Queries must be realistic and could plausibly be asked by a user in the given domain.
- The "correct_disambiguation" must be a single, definitive interpretation. Do not create examples that are still ambiguous after rewriting.
- Do not include any commentary outside the JSON array.

[OUTPUT_SCHEMA]
Return your response as a JSON array of objects, strictly following this schema:
[
  {
    "ambiguous_query": "string",
    "correct_disambiguation": "string",
    "ambiguity_type": "string",
    "explanation": "string"
  }
]

To adapt this template, replace the placeholders with concrete values. [DOMAIN_CONTEXT] should be a detailed description of your knowledge base, user base, and typical search tasks (e.g., 'An internal knowledge base for a fintech company containing product specs, compliance docs, and HR policies.'). [AMBIGUITY_TYPES] should be a comma-separated list from your taxonomy, such as 'entity_collision, homonym, temporal_vagueness, underspecification'. [NUM_EXAMPLES] controls the dataset size. For high-risk domains, add a [RISK_LEVEL] constraint that forces the model to avoid generating examples that could lead to harmful disambiguations, and always plan for human review of the generated dataset before it gates a production system.

This prompt is designed to be called programmatically as part of an offline evaluation data generation job, not as a one-off chat interaction. The primary integration point is a script or CI step that invokes the LLM, captures the structured output, validates it against the expected schema, and appends the results to a version-controlled golden dataset file. Because the generated dataset will be used to gate retrieval pipeline changes, the harness must enforce schema compliance, deduplication, and human review of a sample before the dataset is accepted into the test suite. Treat the output as a candidate set that requires curation, not a final artifact.

Wire the prompt into a Python or TypeScript script that reads a configuration specifying the number of examples to generate per ambiguity type, the target domain context, and any seed terms to include. The script should call the model with response_format set to the JSON schema defined in the prompt template, using a low-temperature setting (0.1–0.3) to prioritize consistency over diversity within a single generation run. Implement a retry loop with exponential backoff for malformed JSON responses: if the output fails schema validation, re-invoke the model with the validation error message appended as a correction instruction. Log every generation attempt, including the raw response, validation result, and retry count, so you can audit flaky generations later. For model choice, a capable instruction-following model with strong JSON mode support (such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro) is recommended; smaller models often struggle with the nested schema and consistent ambiguity classification.

After generation, run a deduplication pass that compares new examples against existing entries in the golden dataset using semantic similarity on the ambiguous_query field. Flag near-duplicates (cosine similarity above 0.92 on embedding vectors) for manual review rather than silently dropping them, since a superficially similar query may exercise a different disambiguation edge case. Implement a mandatory human review step: sample at least 20% of generated examples, stratified by ambiguity type, and have a domain expert verify that the expected_disambiguation is correct and unambiguous. Reject the entire batch if the error rate in the reviewed sample exceeds 5%, and adjust the prompt or domain context before regenerating. Store the accepted dataset with metadata including generation timestamp, model version, prompt version, and reviewer identity so every test case is traceable to its origin. Finally, integrate the dataset into your CI pipeline as a regression test: on every retrieval change, run the ambiguous queries through the disambiguation system and assert that the output matches the expected disambiguation within an acceptable tolerance, failing the build on regressions.