Output Truncation Recovery Prompt

This playbook is for production engineers who receive an LLM response that has been cut off mid-structure by a max_tokens limit. The core job-to-be-done is to recover a valid, machine-readable payload from a partial output—such as a truncated JSON object, an unfinished code block, or a sentence broken mid-clause—without discarding the valuable computation already performed. The ideal user is an integration engineer or backend developer who owns a pipeline where a retry is either too expensive in terms of latency and cost, or where the partial output contains non-deterministic reasoning that would be lost on regeneration.

Use this prompt when the truncation point is close enough to the end of the structure that the model can infer a safe completion. For example, a JSON response missing only a closing brace and the final field value is a strong candidate. A code block missing its last line of a function is another. The prompt works by providing the model with the truncated text and explicit instructions to close the structure without inventing new substantive content. It is most effective when combined with a structural validator in your application harness: the repair is attempted, the result is parsed, and if it fails validation, the system falls back to a full retry or escalation. This pattern is particularly useful in streaming scenarios where you must flush an incomplete chunk to the client before the connection closes.

Do not use this prompt when the truncation point is so early that the model cannot reasonably infer the missing structure—for instance, if only the opening brace of a complex nested object has been emitted. In regulated domains such as healthcare or finance, where exactness is non-negotiable, a truncated output should trigger a full regeneration with a higher token limit or human review rather than an inferred completion. Similarly, if the partial output contains factual claims that are cut off, the repair prompt may hallucinate a plausible but incorrect ending. Always weigh the cost of a retry against the risk of a malformed or subtly incorrect completion reaching a downstream system. The next section provides the prompt template you can adapt and embed directly into your recovery harness.

This template provides a complete prompt for recovering from output truncation caused by max_token limits. It uses few-shot examples to demonstrate two distinct recovery strategies: safe structural closing (completing JSON objects, code blocks, or markdown fences) and explicit truncation markers (inserting [... truncated ...] tokens when safe completion is impossible). The prompt is designed to be dropped into a post-processing or retry pipeline where the partial output and the original generation context are available.

textCopy
You are an output repair assistant. Your job is to take a partially generated response that was cut off due to token limits and produce a clean, usable version.

You have two repair strategies:
1. **Safe structural closing**: If the output is truncated mid-structure (JSON object, code block, markdown fence, XML tag, or list), close the structure correctly without inventing new content. Add only the minimal closing tokens needed.
2. **Explicit truncation marker**: If the output is truncated mid-sentence or mid-thought where structural closing is impossible, insert the marker `[... output truncated ...]` at the cutoff point and preserve all existing content exactly.

Rules:
- Never invent facts, values, or conclusions that were not present in the partial output.
- Never complete a sentence or thought that was cut off mid-expression.
- Preserve all existing content verbatim up to the truncation point.
- If the partial output ends inside a string value, close the string and the containing structure.
- If the partial output ends inside a code block, close the fence and any containing structure.
- If the partial output is plain text with no open structures, insert the truncation marker at the cutoff.

[EXAMPLES]

[INPUT]
Partial output: {"name": "Alice", "address": {"street": "123 Main St", "city": "Spring
Original prompt: Generate a JSON object with user details.
[/INPUT]

[OUTPUT]
{"name": "Alice", "address": {"street": "123 Main St", "city": "Spring[... output truncated ...]"}}
[/OUTPUT]

---

[INPUT]
Partial output: ```python
def calculate_total(items):
    total = 0
    for item in items:
        total += item["p
Original prompt: Write a Python function to calculate totals.
[/INPUT]

[OUTPUT]
```python
def calculate_total(items):
    total = 0
    for item in items:
        total += item["p[... output truncated ...]

[/OUTPUT]

[INPUT] Partial output: The three main factors to consider are: 1) cost efficiency, 2) scalability, and 3) maintainability. Cost efficiency involves analyzing Original prompt: Explain key factors for system design. [/INPUT]

[OUTPUT] The three main factors to consider are: 1) cost efficiency, 2) scalability, and 3) maintainability. Cost efficiency involves analyzing[... output truncated ...] [/OUTPUT]

[INPUT] Partial output: {"results": [{"id": 1, "score": 0.95}, {"id": 2, "score": 0. Original prompt: Generate a JSON array of scored results. [/INPUT]

[OUTPUT] {"results": [{"id": 1, "score": 0.95}, {"id": 2, "score": 0.[... output truncated ...]}]} [/OUTPUT]

Now repair the following partial output:

[INPUT] Partial output: [PARTIAL_OUTPUT] Original prompt: [ORIGINAL_PROMPT] [/INPUT]

[OUTPUT]

Adaptation guidance: Replace [PARTIAL_OUTPUT] with the truncated model response and [ORIGINAL_PROMPT] with the prompt that produced it. The [EXAMPLES] section can be extended with domain-specific truncation patterns relevant to your output format. For JSON-heavy workloads, add examples showing nested object closing, array completion, and string-escape preservation. For code generation, add examples covering multiple language syntaxes. The truncation marker [... output truncated ...] should be replaced with a token that your downstream parsers recognize and handle gracefully—consider using a unique sentinel like <<<TRUNC>>> if the default marker conflicts with your content domain.

What to do next: Test this prompt against a golden dataset of 20–30 real truncation cases from your production logs. Measure two metrics: structural validity (does the repaired output parse correctly?) and content preservation (is all original content present verbatim?). Watch for the failure mode where the model attempts to complete a truncated sentence instead of inserting the marker—this is the most common hallucination pattern. If you observe this, add a counterexample showing the correct marker insertion for that specific case. For high-stakes workflows where hallucinated completions could cause downstream errors, route outputs containing the truncation marker to a human review queue rather than silently consuming them.

The Output Truncation Recovery Prompt is designed to be called as a post-processing repair step immediately after the primary model response is cut off by max_tokens or a context window limit. It should not be used as a standalone generation prompt. Instead, integrate it into your inference pipeline as a conditional recovery path: when the primary response ends without a valid closing token (e.g., unclosed JSON bracket, dangling sentence, incomplete code block), pass the partial output and the original generation context to this prompt for safe structural closure. The prompt's few-shot examples teach the model to distinguish between completable patterns (closing a known structure) and uncompletable patterns (mid-thought reasoning where any completion would be hallucination).

Harness integration steps: (1) Detect truncation by checking for parse failures, missing end delimiters, or token-count proximity to max_tokens. (2) Extract the last N tokens of the partial output (typically 200-500 tokens) plus the original [SYSTEM_INSTRUCTION] and [USER_QUERY] to provide full context. (3) Call the recovery prompt with these inputs and a strict [OUTPUT_SCHEMA] that requires either a completed_output field or an explicit truncation_marker with a completion_confidence score. (4) Validate the repaired output against the original expected schema before accepting it. (5) If completion_confidence is below your threshold (we recommend 0.7 for structured data, 0.5 for prose), discard the repair and either increase max_tokens on retry or return the partial output with an explicit truncation warning to the user. Model choice: Use the same model that produced the original output to avoid distribution mismatch. For cost-sensitive pipelines, a smaller model can handle structural closing (JSON, XML) while prose completion benefits from the original model's capabilities.

Failure modes to monitor: The most dangerous failure is the model hallucinating a plausible ending that contradicts the user's intent or the source evidence. Log every recovery attempt with the original partial output, the completed output, and the confidence score. Implement a human review queue for high-risk domains (legal, medical, financial) where any completion confidence below 0.9 triggers manual inspection. For code generation, never execute repaired code without syntax validation and a sandboxed test run. What to avoid: Do not use this prompt as a general-purpose output extender—it is specifically tuned for recovery from token-limit truncation, not for expanding short answers or adding detail. Do not chain multiple recovery calls; if the first repair fails validation, escalate to a higher token budget or human review rather than retrying the repair prompt.