Token Consumption

The context window is the maximum number of tokens a model can process in a single request. Longer contexts allow for more comprehensive history and larger documents but consume tokens for every input, regardless of relevance.

• Permanent Cost: Every token in the prompt, including system instructions, conversation history, and retrieved documents, is counted.
• Inefficiency Risk: Loading excessive background data or verbose examples directly increases cost without guaranteed benefit.
• Example: A model with a 128K token window processing a 50K token document uses 50K input tokens before any generation begins.

Larger models with more parameters (e.g., 70B vs. 7B) typically have a higher per-token cost due to greater computational complexity. Architecture choices also influence efficiency.

• Dense vs. Sparse Models: Traditional dense transformers process all parameters for each token, while Mixture-of-Experts (MoE) models activate only a subset, potentially offering lower effective cost.
• Pricing Tiers: Providers like OpenAI and Anthropic price API calls based on model family (GPT-4 Turbo costs more per token than GPT-3.5-Turbo).
• Fixed Overhead: The base cost of initializing a model inference pass is incurred regardless of output length.

The number of tokens the model generates is a direct and controllable cost driver. Longer, more verbose responses are more expensive.

• Primary Variable: This is often the largest single cost component in chat/completion tasks.
• Controlled by Parameters: Settings like max_tokens, temperature, and stop_sequences directly limit or influence output length.
• Streaming Costs: Token costs are incurred as the output is streamed, not just at the end of generation.
• Example: A 1000-token article summary costs 10x more in output tokens than a 100-token bullet-point list.

Agentic architectures that use chain-of-thought, reflection, or planning consume tokens for each intermediate reasoning step, not just the final answer.

• Multi-Turn Cost: An agent that "thinks step-by-step" internally generates text that consumes tokens. A ReAct (Reasoning + Acting) loop may produce several reasoning chains before a final answer.
• Recursive Expansion: Techniques like tree-of-thoughts explore multiple reasoning paths, multiplying token consumption.
• Traceability Challenge: These intermediate tokens must be captured by agent telemetry pipelines for accurate cost attribution.

Integrating external tools via tool calling or function calling increases token consumption in several ways:

• Function Descriptions: Detailed schema definitions for tools are included in the context window, adding permanent input tokens.
• Extended Dialogues: The agent may engage in multiple request-response turns with tools (e.g., querying a database, then analyzing results), each requiring new model calls.
• Result Processing: Large JSON responses or data payloads from tools must be fed back into the model's context for analysis, consuming additional input tokens.

RAG architectures inject relevant documents into the prompt context, which is a major token cost driver.

• Retrieval Overhead: Every retrieved chunk (e.g., from a vector database) becomes part of the input context. Retrieving 10 chunks of 500 tokens each adds 5,000 input tokens.
• Precision vs. Cost Trade-off: Broad retrieval for higher recall dramatically increases costs. Optimizing semantic search relevance is a direct cost-control measure.
• Hybrid Search: Combining dense vector search with keyword filters can reduce the number of irrelevant, costly chunks sent to the model.

The systematic tracking and measurement of token consumption across an AI agent's operations. This includes granular logging of:

• Input tokens from the user prompt and system instructions.
• Output tokens generated by the model.
• Context window usage, including tokens from retrieved documents or previous conversation turns.

This data is foundational for cost analysis, budgeting, and identifying optimization opportunities, such as reducing verbose outputs.

The process of assigning the computational and financial expenses of an AI agent's execution to specific internal entities. This enables:

• Chargebacks to business units, projects, or client accounts.
• Understanding cost drivers by linking spend to specific features or user actions.
• Financial accountability by showing which teams are consuming resources.

Attribution requires integrating token consumption data with metadata like user IDs, project codes, and session identifiers.

The granular measurement and logging of every request an agent makes to external services. This is critical because agents often use multiple APIs beyond core LLMs. Metering captures:

• Request parameters and payload size.
• Response latency and size.
• Associated costs from each provider (e.g., Google Maps, database queries).

This data, combined with token consumption, provides a complete picture of an agent's operational cost.

The aggregation of all computational expenses incurred during a single, end-to-end execution of an autonomous agent. A 'session' spans from the initial user request to the final response and includes:

• Total token consumption across all LLM calls.
• Costs of all external tool/API calls.
• Infrastructure overhead (e.g., compute for retrieval).

This metric, often expressed as a Cost Per Session, is vital for evaluating the business viability of agentic workflows and setting pricing for end-users.

A pre-defined limit on the number of tokens an AI agent is allowed to consume for a given task or within a time period. Implementing token budgets is a key cost control mechanism that:

• Prevents runaway costs from infinite loops or overly expansive reasoning.
• Enforces efficiency by requiring agents to operate within constraints.
• Triggers fallback actions when exceeded, such as switching to a cheaper model or terminating the session.

Budgets can be applied at the request, user, or organizational level.

A financial metric that calculates the average expense for an AI agent to successfully complete a specific, valuable unit of work. Unlike generic token counts, CPA ties cost directly to business outcomes. Examples include:

• Cost to analyze and summarize a 50-page document.
• Cost to execute a successful customer support resolution.
• Cost to generate a validated data analysis report.

Optimizing for lower CPA, rather than just lower token consumption, aligns AI spending with business value delivery.