Edge-Case Sampling Prompt for Production Inputs

This prompt is built for QA engineers and prompt designers who need to move from ad-hoc testing to systematic edge-case discovery. The core job-to-be-done is identifying boundary examples that stress-test your AI system's behavior using real production logs or input samples as source material. You should use this prompt when you have access to a representative sample of production inputs and a defined set of expected behaviors or failure categories to measure against. The ideal user is someone who already knows what 'correct' looks like for their system and needs to find the inputs most likely to break it before customers do.

The prompt requires several concrete inputs to be effective. You must provide a set of real production inputs (not synthetic guesses), a reference list of known failure categories (such as null handling, maximum token overflow, malformed JSON, or rare entity types), and a desired output schema for the prioritized edge-case list. The prompt works by analyzing your production data against these failure categories, identifying coverage gaps, and ranking examples by their likelihood to trigger undefined or incorrect behavior. It also flags categories where your current example set has no representative samples, giving you a clear map of your blind spots. For high-risk domains like healthcare or finance, you should add a human review step to validate that the identified edge cases represent realistic production scenarios rather than adversarial noise.

Do not use this prompt when you lack production data or when the system's expected behavior is still undefined. If you are still prototyping and have no stable behavioral contract, edge-case sampling will produce a list of inputs without a clear pass/fail criterion, which wastes time and creates false confidence. Similarly, avoid this prompt if your goal is to generate synthetic training data from scratch—this prompt analyzes existing inputs rather than fabricating new ones. Once you have production logs and a defined behavioral baseline, run this prompt before every major release or whenever you observe unexpected failures in production to catch regressions early.

This prompt template is designed to be pasted directly into your AI system. It instructs the model to act as a QA engineer, analyzing a sample of production inputs to identify and catalog boundary conditions. The goal is to produce a structured output—a prioritized list of edge-case examples—that your team can use to augment test suites, improve example diversity, and harden your application against rare but critical failures. Replace every square-bracket placeholder with your specific data, constraints, and output requirements before execution.

textCopy
You are a senior QA engineer specializing in edge-case discovery for production AI systems.

Your task is to analyze the provided [INPUT_SAMPLE] from production logs and generate a structured catalog of edge-case examples. These examples will be used to test and improve the robustness of an AI feature that [TARGET_SYSTEM_DESCRIPTION].

[INPUT_SAMPLE]:

[RAW_PRODUCTION_INPUTS]

codeCopy

[KNOWN_FAILURE_MODES]:
- [FAILURE_MODE_1]
- [FAILURE_MODE_2]
- [FAILURE_MODE_N]

[CONSTRAINTS]:
- Prioritize examples that are likely to cause silent failures, not just hard crashes.
- Include at least one example for each of the following categories: null/missing values, maximum field lengths, malformed syntax, conflicting constraints, and rare but valid patterns.
- For each edge case, explain *why* it is a boundary condition.

Generate your output as a JSON object matching this [OUTPUT_SCHEMA]:
{
  "edge_case_catalog": [
    {
      "id": "string",
      "category": "null_values | max_length | malformed_input | rare_pattern | conflicting_constraints | other",
      "input_example": "string (the raw, unmodified edge-case input)",
      "expected_behavior": "string (description of the ideal system response)",
      "failure_rationale": "string (why this input is likely to break the current system)",
      "priority": "critical | high | medium | low",
      "related_failure_mode": "string (reference to a [KNOWN_FAILURE_MODE] or 'new')"
    }
  ],
  "coverage_gap_analysis": "string (summary of input categories missing from the sample but likely in production)"
}

Before finalizing, check your output for coverage gaps against the [KNOWN_FAILURE_MODES] list.

To adapt this template, start by curating your [RAW_PRODUCTION_INPUTS]. A representative sample of 100-500 inputs is often sufficient, but ensure it includes both common and visibly problematic cases. The [KNOWN_FAILURE_MODES] list is critical for guiding the model's analysis; populate it with issues from your bug tracker, previous incident postmortems, or common validator rejections. If you don't have a formal list, seed it with generic categories like 'SQL injection attempts', 'Unicode rendering errors', or 'Timeout on large payloads'. The [OUTPUT_SCHEMA] is a starting point—add fields like detection_difficulty or recommended_test_tool if they fit your QA workflow. After generation, always have a domain expert review the catalog for relevance and accuracy before adding the examples to your test suite.

This prompt is designed to run as a scheduled or on-demand batch job against production logs, not as a real-time user-facing endpoint. The primary integration point is a log aggregation system (e.g., Elasticsearch, Datadog, Splunk) or an application database storing raw user inputs. The harness must extract a representative time window of inputs, deduplicate them, and feed them into the prompt in batches if the volume exceeds the model's context window. The output—a prioritized list of edge-case examples with coverage gap annotations—should be written to a test case management system or a dedicated QA dataset repository, not left in a chat interface.

Validation and retry logic is critical because the prompt produces structured output. Implement a JSON schema validator that checks for the required fields: edge_case_id, input_text, failure_mode, priority_score, and coverage_gap_flag. If validation fails, retry up to two times with a stricter [OUTPUT_SCHEMA] constraint and an explicit instruction to correct the parse error. Log all validation failures and retry attempts for observability. For high-risk domains (healthcare, finance, legal), route any output where coverage_gap_flag is true and priority_score is above a configurable threshold (e.g., 0.8) to a human QA reviewer for confirmation before the edge case is added to a regression test suite. The reviewer should verify that the sampled input is a realistic production example and not an artifact of log corruption or a synthetic anomaly.

Model choice and context assembly should favor models with strong instruction-following and structured output capabilities. For production log volumes, use a model with a large context window (e.g., 128k+ tokens) to maximize the number of examples processed per batch. Assemble the prompt context by prepending a fixed [FAILURE_MODE_CATALOG] drawn from your team's known failure modes (e.g., null handling, maximum string length, SQL injection patterns, emoji-only inputs, right-to-left text). This catalog should be maintained as a version-controlled configuration file, not hardcoded into the prompt template. After each run, compare the detected coverage gaps against this catalog and flag any production failure modes that are absent from the catalog for a separate review cycle. Do not treat this prompt as a one-time exercise; schedule it weekly or trigger it on deployment of new prompt versions to catch drift in production input distributions before they cause user-facing failures.