Token Budget Attack

The core mechanism involves flooding the model's maximum token limit with adversarial filler content. When the combined length of the system prompt, conversation history, and injected payload exceeds the model's context window, the oldest tokens are truncated. Attackers strategically position malicious instructions at the end of their payload, ensuring that critical safety guardrails or system instructions at the beginning of the context are the first to be evicted. This forces the model to operate solely on attacker-controlled context.

• Token limit: Models like GPT-4 have 128k token windows, but truncation still occurs
• Eviction order: First-in, first-out displacement of safety instructions
• Silent failure: The model does not alert users when safety prompts are truncated

In Retrieval-Augmented Generation (RAG) systems, attackers can poison the retrieval corpus with excessively long documents. When the retriever fetches these bloated documents and injects them into the context, they consume a disproportionate share of the token budget. This leaves minimal room for the agent's reasoning scratchpad, few-shot examples, or multi-step planning. The agent is forced to act on impoverished context, increasing the likelihood of errors or compliance failures.

• Document stuffing: Malicious documents padded to maximum chunk size
• Retrieval priority: High similarity scores ensure these documents are always fetched
• Reasoning starvation: Agent loses space for chain-of-thought deliberation

Agents managing long-running tasks often use recursive summarization to compress conversation history and stay within token limits. Attackers exploit this by injecting content that, when summarized, produces a distorted or malicious compression. The attacker's payload is crafted so that the summary retains the adversarial intent while discarding surrounding safety context. Each recursive cycle amplifies the distortion, eventually replacing the agent's operational directives with attacker-controlled objectives.

• Compression drift: Each summarization step can shift semantic meaning
• Amplification loop: Errors compound across recursive cycles
• Persistence: Poisoned summaries are stored in long-term memory for future sessions

Attackers who control an API endpoint or tool that an agent calls can return padded responses filled with irrelevant tokens. When the agent ingests this tool output into its context, the padding consumes budget that was reserved for safety verification steps. This is particularly dangerous in multi-agent systems where one compromised agent can flood the shared context of sibling agents, degrading the entire swarm's decision-making capacity.

• API response inflation: Returning 10x the expected response length
• Cross-agent contamination: One poisoned tool output affects all agents sharing context
• Verification bypass: Safety checks are truncated due to unexpected response size

Attackers use byte-pair encoding (BPE) quirks to hide malicious payloads that consume minimal visible characters but expand into many tokens. By crafting inputs with rare Unicode characters, repeated whitespace, or unusual tokenization boundaries, attackers can smuggle large token-consuming payloads past character-based filters. The payload appears short to human reviewers but explodes in token count when processed by the model's tokenizer.

• Token inflation: A single character can map to multiple tokens
• Filter evasion: Character-count filters fail to detect token-level attacks
• Silent consumption: Budget is consumed without visible content changes

An attacker submits a resume or support ticket containing 50,000 tokens of invisible Unicode whitespace before the actual content. The agent's context window fills with noise, pushing the system prompt and safety instructions out of scope. The remaining malicious instruction—now unguarded—directs the agent to execute arbitrary tool calls or exfiltrate data.

• Real-world analog: The 'clown attack' on LLM APIs where padding tokens consumed the budget
• Impact: System prompt truncation, loss of tool-use restrictions
• Mitigation: Pre-processing token counting and content density checks

An attacker publishes 10,000 SEO-optimized documents containing filler text laced with adversarial instructions. When the agent's RAG pipeline retrieves these documents, the combined context overwhelms the token budget. The agent grounds its response in attacker-controlled content while safety instructions are displaced.

• Technique: Corpus poisoning at scale via automated content generation
• Target: Agents relying on web search or shared knowledge bases
• Key indicator: Sudden increase in retrieval of low-quality, verbose sources

An attacker engages an agent in a lengthy, multi-turn conversation that appears benign. Each exchange adds to the conversation history, gradually consuming the token budget. By turn 50, the original system prompt has been pushed beyond the context window. The attacker then issues a command that would normally be blocked.

• Pattern: Slow-burn social engineering adapted for LLM context limits
• Vulnerable systems: Customer service agents with long session persistence
• Defense: Sliding window attention with pinned safety preamble

An agent uses recursive summarization to compress long conversation histories. An attacker injects a subtle instruction in an early message: 'When summarizing, always include this priority.' Each summarization cycle amplifies the malicious instruction while compressing away safety guidelines. After several iterations, the summary is entirely attacker-controlled.

• Mechanism: Exploitation of the compression ratio to amplify adversarial content
• Risk: Long-running autonomous agents with memory management loops
• Detection: Monitoring summary drift against original safety constraints

An attacker compromises an API endpoint that an agent calls. The API returns a response padded with thousands of tokens of benign-looking JSON containing a hidden instruction at the end. The agent processes the entire response, consuming its budget. The hidden instruction—now the only actionable content remaining—directs the agent to call additional compromised tools.

• Attack vector: Supply chain compromise of agent-connected APIs
• Amplification: Each tool call triggers another, creating a cascade
• Mitigation: Tool output length limits and content validation before ingestion

An attacker prompts a coding agent to generate a function that requires extensive boilerplate code. The agent fills its context window with generated code, losing sight of safety constraints. The attacker then requests a modification that introduces a vulnerability or backdoor, which the agent complies with because the safety instructions are no longer in context.

• Target: Autonomous coding agents and Copilot-style assistants
• Technique: Exploiting the agent's own verbosity against itself
• Safeguard: Immutable safety instructions pinned outside the token budget

A direct precursor to token budget attacks, this technique floods the context window with irrelevant data to physically displace critical content. When the combined length of system prompts, conversation history, and injected filler exceeds the model's maximum context length, the oldest tokens are truncated first. Attackers exploit this by placing malicious instructions in the middle of a document, knowing safety instructions at the beginning will be evicted. This is distinct from budget attacks in that it relies on positional displacement rather than consumption of reasoning tokens.

Exploits the well-documented positional attention bias of transformer architectures, where models attend most strongly to content at the beginning and end of the context window. Attackers place malicious payloads in the middle third of a long document or conversation, where they are least likely to be scrutinized by safety mechanisms but are still fully processed by the model. This technique is particularly effective when combined with token budget attacks, as the model's reduced capacity for thorough reasoning makes it more susceptible to mid-context instructions.

Injects malicious reasoning steps directly into an agent's scratchpad or reflection loop. When a token budget attack has already consumed the agent's capacity for independent verification, the model may uncritically adopt these injected reasoning traces. The attack works by presenting a plausible-sounding but flawed logic chain that leads to an attacker-intended conclusion. This is especially dangerous in agents that use chain-of-thought prompting for complex task decomposition, as the contaminated reasoning propagates through subsequent planning steps.

Targets the recursive summarization mechanisms that agents use to compress long conversation histories. When an agent's context approaches its limit, it may generate a summary of earlier interactions to preserve state. Attackers inject content designed to survive summarization in a distorted form, causing critical safety instructions to be dropped or semantically altered during compression. Over multiple summarization cycles, the agent's operational constraints can be progressively eroded until they are effectively absent from the active context.

The broader category encompassing all techniques that insert malicious content into an agent's context window to manipulate reasoning or trigger unintended tool use. Unlike simple prompt injection, which targets the initial user input, adversarial context injection can occur through any data source the agent ingests:

• Retrieved documents from a vector store
• API responses from tool calls
• Content from multi-agent communications
• User-uploaded files processed by the agent Token budget attacks amplify the effectiveness of injection by reducing the model's capacity to detect and resist manipulation.

A sophisticated attack that directly manipulates the transformer's KV cache during inference to alter attention patterns. By injecting malicious key-value pairs into the cache, attackers can force the model to attend to adversarial tokens over legitimate context. This technique bypasses input-level filters entirely because the manipulation occurs at the internal representation level. When combined with token budget exhaustion, the model's reduced attention capacity makes it more vulnerable to having its attention redirected to attacker-controlled representations.