Context Overflow and Token Exhaustion Stress Test Prompt

This prompt is for infrastructure engineers and AI platform teams who need to verify that a prompt, model, and application stack behave predictably when pushed to the boundaries of the context window. The job-to-be-done is not to test model recall or reasoning quality, but to stress-test the system's memory pressure handling: truncation behavior, graceful degradation, error surfacing, and output stability when the input approaches or exceeds the token limit. The ideal user is someone responsible for production reliability who needs to know what breaks first before a user sends an unexpectedly large document or a runaway agent loop fills the context.

Use this prompt when you are evaluating a new model version, a new prompt architecture, or a context assembly pipeline (such as a RAG system that packs many retrieved chunks). It is appropriate when you need concrete evidence for a release gate decision, such as 'does the system fail closed or open under token pressure?' The prompt works by generating progressively larger inputs, near-limit padding, and fragmentation patterns that simulate real-world overload scenarios. You should pair it with a validation harness that checks for specific failure signatures: missing stop sequences, truncated JSON, dropped instructions, or silent mid-response cutoffs.

Do not use this prompt as a general quality benchmark or a substitute for task-specific regression tests. It does not measure accuracy, factuality, or instruction following under normal conditions. It is a destructive test designed to find breaking points. If your application already enforces strict input token limits with pre-truncation and user-facing errors, this prompt may confirm expected behavior rather than reveal new risks. After running the stress test, your next step should be to document the observed failure mode, decide whether the application layer or the prompt itself should handle it, and encode the acceptance criteria into your CI/CD eval harness.

The prompt template below is designed to be executed by a harness—not a human. It instructs the model under test to generate a series of adversarial payloads that stress the context window of a downstream system. Each payload is constructed to probe a specific failure mode: near-limit padding, mid-instruction truncation, delimiter fragmentation, and attention dilution through long irrelevant prefixes. The template uses square-bracket placeholders for the target model's context limit, the number of test cases to generate, and the output schema the harness expects. Adapt these placeholders to match your test runner's configuration before execution.

textCopy
You are a context-window stress-test generator. Your job is to produce a set of test payloads designed to push a target language model to its token limit and observe its behavior under memory pressure.

TARGET_CONTEXT_LIMIT: [MAX_TOKENS]
NUMBER_OF_TEST_CASES: [NUM_CASES]
OUTPUT_FORMAT: [OUTPUT_SCHEMA]

For each test case, generate a payload that exercises one of the following stress patterns:

1. NEAR-LIMIT PADDING: A payload that fills the context window to within 50 tokens of [MAX_TOKENS] with repetitive but syntactically valid text, followed by a simple instruction. The instruction should appear only in the final 20 tokens.

2. MID-TRUNCATION: A payload where a critical instruction is split across the truncation boundary. Place a long irrelevant prefix before the instruction, and ensure the instruction's key constraint (e.g., "respond only in JSON") appears in the last 10 tokens before [MAX_TOKENS].

3. DELIMITER FRAGMENTATION: A payload that uses the same delimiter characters as the system's output format markers (e.g., triple backticks, XML tags, or JSON brackets) inside the user input, positioned such that truncation could leave an unclosed delimiter.

4. ATTENTION DILUTION: A payload with a 2000-token preamble of dense, factual statements containing numbers, dates, and named entities, followed by a single-sentence instruction that contradicts one fact in the preamble. The instruction must appear after token [MAX_TOKENS] - 100.

5. RECURSIVE FRAGMENTATION: A payload that embeds instructions inside nested quoted blocks, parentheses, or markdown structures, where truncation at [MAX_TOKENS] could leave an unclosed nesting level.

For each test case, output a JSON object with the following fields:
- case_id: string identifier
- stress_pattern: one of the pattern names above
- payload: the full text payload
- expected_truncation_point: the approximate token position where truncation is expected
- instruction_survival_check: a description of what the evaluator should check to determine if the instruction survived truncation

Generate exactly [NUM_CASES] test cases, covering each stress pattern at least once.

To adapt this template for your own test harness, replace [MAX_TOKENS] with the actual context limit of the model under test (e.g., 128000 for GPT-4 Turbo, 200000 for Claude 3). Set [NUM_CASES] to the number of variants you need—start with 10 for a smoke test, scale to 50+ for a full regression run. The [OUTPUT_SCHEMA] placeholder should be replaced with the exact JSON schema your eval harness expects, or you can leave the inline schema as-is if your runner parses it directly. Wire the generated payloads into a separate execution step: feed each payload to the model under test, capture the truncated output, and run the instruction_survival_check assertions. Do not run these tests against production models without a sandboxed environment—truncated outputs can include malformed JSON, partial tool calls, or half-finished reasoning that downstream parsers may mishandle.

This prompt is not a one-off manual test. It belongs inside a repeatable harness that generates payloads, executes the prompt against a target model, captures the full response and metadata, and evaluates the results against predefined pass/fail criteria. The harness should be designed to run as part of a CI/CD prompt gate or a scheduled regression suite, not as an ad-hoc debugging tool. The core loop is: generate a stress payload, send it to the model, collect the output and token usage, and run eval checks on truncation behavior, output completeness, and graceful degradation signals.

The implementation should include a payload generator that constructs inputs at configurable token thresholds (e.g., 50%, 90%, 100%, and 105% of the model's context window). Use the target model's tokenizer to count tokens precisely rather than estimating by word or character count. The generator should produce three payload types: (1) a long document with a question at the end, (2) a long document with the question in the middle, and (3) a fragmented input with multiple sections and a final instruction. Each payload must include a verifiable task—such as extracting a specific fact, answering a question, or returning a structured field—so the eval harness can determine whether the model succeeded or silently dropped information. Validation layer: after each run, check finish_reason (expect stop for within-limit tests, length or max_tokens for overflow tests), compare output completeness against the expected answer, and log the token counts for both input and output. For models that support it, capture the truncation_notice or equivalent field. Retry logic: do not retry on overflow-induced failures; the point is to observe degradation, not recover from it. Retry only on transport errors or API timeouts, with a maximum of 2 attempts and exponential backoff.

Eval checks should be automated and produce a pass/fail report per payload. Key checks include: (1) Truncation detection: did the model acknowledge missing context or produce an incomplete answer? (2) Silent failure detection: did the model return a confident but wrong answer when context was truncated? (3) Output format integrity: did the model maintain the requested output schema under memory pressure? (4) Graceful degradation scoring: rate the model's response on a scale from 'clean failure with explanation' to 'hallucinated answer with no warning.' Store all results—payload, response, token counts, eval scores, and model version—in a structured log for trend analysis. Model choice: run this against every model in your production routing table, not just your primary model. Different models handle context overflow differently; a prompt that degrades gracefully on GPT-4o may hallucinate aggressively on Claude 3.5 Sonnet or an open-weight model. Human review gate: if the eval harness detects silent failures (confident wrong answers under overflow), flag the run for human review before promoting the prompt version. Do not ship a prompt that silently fabricates answers when context is exhausted.

Integration pattern: wrap the generator, executor, and evaluator in a single test function that returns a structured result object. Call this function from your CI pipeline with a matrix of model IDs and payload configurations. Set a hard threshold: if any model fails the graceful degradation check at 100% context utilization, block the release. For 105% overflow tests, require that the model either refuses, truncates visibly, or returns an error—never a confident fabrication. What to avoid: do not run this test only once during initial prompt development. Context window behavior changes across model updates, tokenizer changes, and prompt modifications. Schedule this test to run on every prompt version change and on every model provider upgrade. A prompt that passed six months ago may silently break after a model update changes how context is allocated.