Session Costing

The total number of tokens processed by a language model during a session, including input, output, and context. This is the primary cost driver for services like OpenAI's API and Anthropic's Claude. For example, a session analyzing a 10-page document might consume 15,000 tokens.

• Input Tokens: Text from the user prompt, system instructions, and context from memory.
• Output Tokens: The text generated by the model in its final and intermediate responses.
• Context Window Usage: The portion of the model's maximum context length filled during the session, which influences pricing tiers.

The granular measurement of requests to external services, which incur separate fees. Each tool call or data retrieval is a distinct cost component.

• External Model APIs: Calls to vision models, specialized LLMs, or embedding services.
• Software Tools & Databases: Invocations of functions like SQL queries, CRM updates, or payment gateways.
• Cost Variables: Pricing depends on the service's fee structure, request complexity, and data volume returned. Metering logs parameters, response sizes, and latency for audit.

The infrastructure cost for the processing time and hardware used. This is measured in standardized units like GPU-seconds or vCPU-hours.

• Inference Compute: The processing required to run the agent's core model(s) on specialized hardware (e.g., NVIDIA H100, Google TPU).
• Orchestration Overhead: CPU and memory used by the agent framework for planning, state management, and inter-agent communication.
• Cloud Pricing Models: Costs are often based on instance type, duration of execution, and region. Prolonged reasoning or reflection loops significantly increase compute units.

The framework for assigning expenses to specific sessions and their internal components. High cost granularity enables precise financial management.

• Per-Session Attribution: Linking all costs to a single user request's end-to-end execution.
• Per-Action Breakdown: Isolating cost for individual steps like a tool call, a retrieval-augmented generation (RAG) query, or a planning cycle.
• Resource Attribution: Mapping infrastructure usage (CPU, memory, I/O) back to specific agent sessions. This creates a token audit trail and enables spend attribution to business units.

The primary factors that determine total session expense. Understanding these allows for optimization and budgeting.

• Model Size & Context Length: Larger models (e.g., GPT-4) and longer context windows are more expensive per token.
• Reasoning Complexity: Sessions requiring multi-step planning, reflection, or extensive few-shot examples consume more tokens and compute time.
• Tool Call Volume & Latency: Each external API call adds direct cost and increases session duration, accruing more compute unit charges.
• Token Efficiency: A key metric measuring useful output per token consumed. Inefficient prompts or redundant retrievals waste budget.

Processes for predicting and monitoring session costs against budgets.

• Cost Forecasting: Predicting expenses based on historical patterns, such as average tokens per session type and planned agent workload.
• Token Budgets & Compute Budgets: Pre-defined limits for a session, project, or time period to prevent overruns.
• Cost Overrun Detection: Real-time alerts triggered when spending exceeds thresholds, indicating potential inefficiencies or errors.
• Cost Anomaly Identification: Detecting unexpected spending deviations, which may signal prompt injection attacks, faulty tool integrations, or changed API pricing.

Token accounting is the systematic tracking and measurement of token consumption across an AI agent's operations. This granular logging is the foundation of cost analysis.

• Tracks input tokens, output tokens, and context window usage.
• Primary Driver of cost for services like OpenAI's API and Anthropic's Claude.
• Enables per-session cost roll-up and efficiency analysis (e.g., tokens per successful task).

Cost attribution is the process of assigning computational and financial expenses to specific business units, projects, or user sessions. It transforms raw telemetry into actionable business intelligence.

• Links agent spend to a cost center (e.g., Marketing, R&D).
• Uses metadata like user ID, project ID, or tenant ID.
• Essential for internal chargeback and showback models to ensure financial accountability.

API call metering is the granular measurement and logging of every request an agent makes to external services. This captures a major secondary cost driver beyond core model inference.

• Logs timestamps, endpoints, parameters, response sizes, and latency.
• Tracks costs from integrated services (e.g., database queries, payment APIs, search).
• Critical for understanding the full cost per session, which often includes multiple tool calls.

Cost per session (CPS) is a key financial metric representing the total expense required to complete one discrete agent interaction from initial prompt to final response.

• Aggregates token costs, API call costs, and allocated compute infrastructure costs.
• Primary KPI for evaluating the business viability of agentic workflows.
• Used to benchmark efficiency gains from prompt optimization, caching, or model selection.

A token budget is a pre-defined, enforceable limit on the number of tokens an AI agent is allowed to consume within a given task or session. It is a core cost-control mechanism.

• Prevents runaway costs from infinite loops or overly verbose reasoning.
• Can be static (e.g., 10,000 tokens per query) or dynamic (based on request priority).
• Triggers fallback actions like early termination or switching to a cheaper model when exceeded.

A cost allocation model is the framework of rules that defines how aggregate AI operational expenses are distributed. It operationalizes the policy behind cost attribution.

• Examples: Proportional allocation by token usage, even split per session, or weighted by compute time.
• Incorporates business logic (e.g., R&D projects absorb higher cost for experimental agents).
• Outputs are used for departmental P&L statements and FinOps reporting.