Cost Driver

The direct cost of model execution is the most significant driver, calculated by the number of tokens processed. This includes:

• Input (Prompt) Tokens: The cost to process the user's query, system instructions, and any provided context.
• Output (Completion) Tokens: The cost to generate the agent's response, which is typically more expensive than input.
• Context Window Usage: Longer context windows (e.g., 128K tokens) increase the base cost per call, as the entire context is processed. Token efficiency in prompts and outputs is paramount for cost control.

Each time an agent calls an external tool or API, it incurs additional costs and latency. Key factors include:

• API Call Volume: The number of distinct calls to services like databases, search APIs, or custom functions.
• Third-Party API Pricing: Costs from services like SerpAPI for web search or paid data providers.
• Internal Compute Costs: Execution of proprietary functions or microservices that consume compute resources. API call metering is essential for attributing these distributed expenses.

Agents using chain-of-thought, planning, or reflection loops perform multiple internal reasoning steps before a final output. This directly increases cost because:

• Multi-Turn Conversations: Each agent 'thought' or internal monologue consumes tokens.
• Iterative Refinement: Agents that re-evaluate and correct their work (ReAct, Reflexion) process significantly more tokens.
• Orchestration Overhead: Frameworks that manage these steps (e.g., LangGraph, CrewAI) add processing layers. More complex tasks lead to higher cost per session.

Providing the agent with relevant context from external sources is a major cost component. This involves:

• Retrieval-Augmented Generation (RAG): Costs for querying vector databases or search indexes to find relevant documents.
• Context Length: Ingesting large retrieved documents into the model's context window drastically increases token counts.
• Knowledge Graph Queries: Executing complex graph traversals to fetch structured data. Inefficient retrieval that returns irrelevant data wastes the most expensive resource: model tokens.

The choice of foundation model has a non-linear impact on cost. Considerations include:

• Model Size & Capability: Larger, more capable models (e.g., GPT-4, Claude 3 Opus) are orders of magnitude more expensive per token than smaller ones (e.g., GPT-3.5-Turbo, Claude Haiku).
• Provider Pricing Models: Costs vary between OpenAI, Anthropic, Google, and open-source providers (where cost is primarily compute footprint).
• Specialized vs. General Models: Fine-tuned or domain-specific models may have higher per-call costs but achieve goals in fewer steps, affecting total cost per action.

The supporting infrastructure for running agents introduces baseline and variable costs:

• Orchestration Runtime: Compute for the agent framework itself (e.g., CPU/memory for LangChain, AutoGen).
• State Management & Memory: Storage and query costs for maintaining agent conversation history and episodic memory.
• Observability & Logging: Processing and storing telemetry, distributed traces, and token audit trails.
• Networking & Latency: Data transfer costs, especially in multi-cloud or hybrid architectures. These costs scale with agent activity and are critical for cost attribution.

Token accounting is the systematic tracking and measurement of token consumption across an AI agent's operations. This is a foundational practice for cost analysis because token usage is often the largest direct cost driver for services like OpenAI's API or Anthropic's Claude.

• Tracks: Input tokens, output tokens, and context window usage.
• Purpose: Provides the raw data needed for cost attribution and budgeting.
• Implementation: Typically involves instrumenting the agent's LLM calls to log token counts per request.

Cost attribution is the process of assigning computational and financial expenses to specific causal factors. While a cost driver is the what (e.g., model size), attribution is the who or why.

• Links Expenses To: Business units, projects, user sessions, or individual agent tasks.
• Enables: Chargeback models, showback reporting, and ROI analysis.
• Requires: Integration of cost driver data (tokens, API calls) with business context (user ID, project code).

API call metering is the granular measurement and logging of requests to external services. For agents that use tool calling, this is a critical secondary cost driver beyond LLM tokens.

• Measures: Number of calls, parameters, response sizes, latency, and third-party service costs.
• Critical For: Agents that integrate with databases, payment processors, or custom software.
• Output: Data feeds API spend tracking systems and helps identify expensive or inefficient tool usage patterns.

Session costing aggregates all expenses incurred during a single end-to-end agent execution. It provides the cost per session metric, which is vital for understanding unit economics.

• Aggregates: LLM token costs, external API costs, and internal compute costs.
• Answers: "How much did it cost to handle this customer query or process this document?"
• Foundation: For calculating Cost Per Action (CPA) and evaluating agent efficiency.

Cost Per Action is a key business metric that calculates the average expense for an agent to complete a specific, valuable unit of work. It translates technical cost drivers into business value.

• Formula: (Total Session Cost) / (Number of Successful Actions).
• Example Actions: Processing an invoice, making a booking, resolving a support ticket.
• Use: Benchmarking agent performance, justifying automation ROI, and setting token budgets per task type.

A token budget is a pre-defined limit on token consumption for a task, session, or time period. It is a direct control mechanism applied to a primary cost driver.

• Purpose: Prevents cost overruns and enforces efficiency by limiting context length or reasoning steps.
• Implementation: Often enforced at the agent orchestration layer, cutting off sessions that exceed the budget.
• Related to: Cost overrun detection systems that trigger alerts when spend approaches budgetary thresholds.