Synthetic User Query Generation Prompt for QA

This prompt is for QA engineers and ML platform teams who need to build diverse, realistic test suites for evaluating prompt behavior. The job-to-be-done is generating a high-coverage set of synthetic user queries that stress-test a target prompt across personas, intents, languages, difficulty levels, and edge cases. You use this when you have a prompt under development and need to populate a golden dataset with inputs that represent real-world usage before you can run regression tests. The ideal user has a clear specification of the target prompt's expected behavior, its input contract, and the failure modes they want to probe.

Do not use this prompt when you lack a defined prompt specification or expected output contract. It is not a replacement for production log sampling or human-labeled data when ground-truth annotations are required for compliance audits. The synthetic queries it generates are only as good as the personas, intents, and constraints you provide. Always validate generated queries for realism and distinctness before adding them to a golden dataset. For high-risk domains such as healthcare or finance, pair this prompt with a human review step to ensure generated scenarios do not introduce unrealistic or unsafe test conditions.

After generating queries, feed them into your target prompt and compare outputs against expected behavior tags. Use the coverage mapping to identify gaps in your test suite. Combine this prompt with the Edge Case Discovery Prompt for QA and the Coverage Gap Analysis Prompt to build a comprehensive regression suite. Avoid treating generated queries as static—re-run this prompt when the target prompt's requirements change to refresh your test surface.

This prompt template is designed to be the core instruction set you send to an LLM to generate synthetic user queries. It is built for adaptation: every element in square brackets is a placeholder you must replace with your specific application context, personas, and quality requirements. The goal is to produce a structured, machine-readable set of test inputs that can be directly ingested by your evaluation harness.

textCopy
You are a senior QA engineer designing a test suite for an AI system. Your task is to generate a diverse set of synthetic user queries based on the provided context.

## System Under Test
[SYSTEM_DESCRIPTION]

## Output Schema
Generate a JSON array of objects. Each object must conform to this exact schema:
[
  {
    "query_id": "string, a unique slug like 'q-001'",
    "user_query": "string, the exact text a user would type",
    "user_persona": "string, one of: [PERSONA_LIST]",
    "intent": "string, the primary goal of the query",
    "language": "string, BCP-47 language tag, e.g., 'en-US'",
    "difficulty": "string, one of: 'easy', 'medium', 'hard', 'adversarial'",
    "expected_behavior_tags": ["string"],
    "ambiguity_markers": ["string"]
  }
]

## Generation Constraints
- Generate exactly [NUMBER_OF_QUERIES] queries.
- Distribute personas evenly: [PERSONA_DISTRIBUTION_INSTRUCTIONS].
- Ensure at least [PERCENTAGE]% of queries are 'hard' or 'adversarial'.
- Include queries in these languages: [LANGUAGE_LIST].
- 'expected_behavior_tags' must be from this controlled vocabulary: [BEHAVIOR_TAG_VOCABULARY].
- 'ambiguity_markers' should describe what makes the query unclear (e.g., 'missing pronoun referent', 'underspecified entity', 'jargon').
- Adversarial queries should test prompt injection resistance, boundary conditions, or off-topic requests without being malicious.

## Coverage Requirements
Map each query to at least one of these prompt requirement areas: [REQUIREMENT_AREAS].

## Examples of Good Queries
[FEW_SHOT_EXAMPLES]

## Output Format
Return ONLY the valid JSON array. Do not include any text before or after the JSON.

To adapt this template, start by defining your [SYSTEM_DESCRIPTION] and [PERSONA_LIST] with precision. A vague persona like 'a user' will produce generic queries. Instead, use roles like 'a non-technical small business owner trying to export a report' or 'a developer debugging an API error at 3 AM.' The [BEHAVIOR_TAG_VOCABULARY] is critical for downstream filtering; use tags like 'safe-refusal-expected', 'tool-call-required', or 'grounding-required' to make the dataset immediately useful for automated evals. Always run the generated JSON through a schema validator before adding it to your golden set to catch format errors early.

This prompt is designed to be called programmatically as part of a test data generation pipeline, not as a one-off chat interaction. The harness should treat each invocation as a batch generation request: provide the required [PERSONA_COUNT], [DOMAIN], and [COVERAGE_TARGETS] placeholders, collect the structured JSON output, and validate it before ingestion into your golden dataset or test case management system. The prompt returns an array of query objects, each with a query_text, persona, intent, difficulty, expected_behavior_tags, and ambiguity_markers field. Your harness must enforce that every returned object conforms to this schema.

Wire the prompt into a script or microservice that handles the full lifecycle: (1) Input assembly — map your test coverage gaps to the [COVERAGE_TARGETS] list, select [PERSONA_COUNT] based on desired diversity, and set [DOMAIN] to scope the generation. (2) Model selection — use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Enable JSON mode or structured outputs if the provider supports it to reduce parsing failures. (3) Validation — after receiving the response, run a schema validator that checks required fields, enum values for difficulty (e.g., easy, medium, hard, adversarial), and non-empty query_text. Reject any batch where more than 5% of records fail validation. (4) Deduplication — compare generated queries against existing golden dataset entries using embedding similarity (cosine similarity threshold of 0.92 or higher) to flag near-duplicates. Queries that are too similar should be discarded or flagged for manual review. (5) Retry logic — if validation fails or the model returns malformed JSON, implement up to 2 retries with exponential backoff. On the second retry, append a repair instruction to the prompt: The previous output failed JSON schema validation. Ensure every record includes all required fields. If retries are exhausted, log the failure and alert the QA lead.

For high-stakes test suites (e.g., safety-critical applications, regulated domains), add a human review gate before generated queries enter the golden dataset. Route a random 10% sample of each batch to a review queue where a QA engineer checks for off-topic queries, unrealistic personas, or subtle bias. Track review outcomes to measure generator quality over time. Log every generation run with: timestamp, model version, prompt version, input parameters, batch size, validation pass rate, deduplication count, and reviewer actions. This audit trail is essential for debugging regressions when a prompt change unexpectedly alters the distribution of generated test queries. Avoid wiring this directly into CI/CD without the validation and deduplication steps — raw generated queries will contain noise that degrades test suite reliability.