Compute Unit

A compute unit provides a normalized metric for disparate hardware resources, enabling apples-to-apples cost comparison across different cloud providers and hardware types. Common examples include:

• GPU-second: Measures time a GPU is actively processing.
• vCPU-hour: Measures virtual CPU time consumed.
• TPU-core-hour: Specific to Google's Tensor Processing Units.
• FLOP-second: Measures floating-point operations performed. This standardization is critical for cost attribution models and creating a unified agent telemetry pipeline that tracks expenses across heterogeneous infrastructure.

Compute units are the primary cost driver for AI inference and training, directly translating to cloud bills. Key relationships include:

• Model Size & Complexity: Larger models (e.g., 70B+ parameters) consume more units per inference.
• Context Window Length: Longer prompts and conversations increase memory and compute usage.
• Inference Latency: Lower latency requirements often demand more powerful (and expensive) hardware, increasing the cost per unit. Monitoring compute unit consumption is therefore foundational to cost forecasting and preventing cost overruns in production agent systems.

Modern observability platforms break down aggregate compute costs by attributing units to specific entities, enabling spend attribution and accountability. This granularity allows costs to be tracked to:

• Individual Agent Sessions: The total cost per session for a single user interaction.
• Specific Tool or API Calls: Isolating the expense of external service invocations (API call metering).
• Business Unit or Project: Allocating expenses via a cost allocation model.
• Individual Model Invocations: Understanding the cost of each reasoning step. This level of detail is essential for token accounting and resource attribution.

While token consumption is a key cost factor for proprietary LLM APIs (e.g., OpenAI, Anthropic), compute units measure the underlying infrastructure cost, especially for self-hosted or open-source models. The relationship is crucial:

• API-Based Agents: Cost is primarily tokens + API call fees. Compute units are managed by the provider.
• Self-Hosted Agents: Cost is directly tied to compute unit consumption on your own infrastructure (GPU-hours).
• Hybrid Systems: Combine API calls (token costs) with self-hosted models (compute unit costs). Effective agent cost telemetry must track both dimensions to calculate the true compute footprint and session costing.

Compute units enable proactive financial management of AI operations. Teams use them to:

• Set compute budgets and token budgets for projects.
• Implement cost overrun detection by alerting on unusual spikes in unit consumption.
• Perform cost forecasting by analyzing historical unit usage trends against planned workloads.
• Optimize for token efficiency and compute allocation to maximize output per unit spent. This financial rigor turns observability data into actionable business intelligence, supporting FinOps practices for AI.

Measuring compute units identifies opportunities for inference optimization and latency reduction. Techniques that reduce unit consumption directly lower costs:

• Model Quantization: Reduces precision of model weights, decreasing compute and memory needs.
• Continuous Batching: Groups multiple requests to improve GPU utilization.
• Caching: Stores frequent computations (e.g., embeddings) to avoid redundant processing.
• Hardware Selection: Choosing the right instance type (e.g., inferentia vs. A100) optimizes cost per action. Monitoring units provides the baseline metric to validate the ROI of these optimization efforts.

The systematic tracking and measurement of token consumption across an AI agent's operations. This includes input tokens, output tokens, and context window usage, forming the primary data for cost analysis and budgeting when using language model APIs.

• Purpose: Provides granular visibility into the largest variable cost of LLM-based agents.
• Key Metric: Token-per-second burn rate.
• Example: Logging that a customer support agent used 1,250 input tokens and 450 output tokens for a single query.

The process of assigning the computational and financial expenses of an AI agent's execution to specific business units, projects, or user sessions. It transforms raw telemetry (tokens, API calls) into actionable business intelligence.

• Mechanisms: Uses tags, session IDs, and project identifiers embedded in telemetry data.
• Goal: Enables chargeback models and shows ROI for specific AI initiatives.
• Example: Attributing $450 of monthly OpenAI API costs to the Marketing Department's content generation agent.

The granular measurement and logging of every request an agent makes to external services. This is critical for cost telemetry as tool calls often incur separate fees.

• Captured Data: Endpoint, parameters, response size, latency, and cost.
• Importance: Provides a complete picture of operational cost beyond just model inference.
• Example: Metering a call to a paid weather API that costs $0.001 per request, logged with the session ID that triggered it.

The aggregation of all computational expenses incurred during a single, end-to-end execution of an autonomous agent to fulfill a user request. This is the atomic unit of business value for cost analysis.

• Components: Sums token costs, external API call costs, and internal compute unit usage.
• Output: A single Cost Per Session (CPS) metric.
• Example: Calculating that processing a complex travel itinerary request cost $0.18, combining LLM tokens and multiple flight API lookups.

A primary technical factor that has a direct and significant impact on the total operational expense of an AI agent. Identifying cost drivers is essential for optimization.

• Common Drivers: Context window length, model size/version, number of tool calls, reasoning steps, and retrieval operations.
• Analysis: Engineers analyze cost drivers to make trade-offs between performance, accuracy, and expense.
• Example: Discovering that using GPT-4 instead of GPT-3.5-Turbo is the dominant cost driver for a summarization agent, increasing cost by 20x.

The continuous measurement of infrastructure resource usage by AI agents, enabling accurate cost forecasting and capacity planning. This complements API-based metering for self-hosted or fine-tuned models.

• Metrics: GPU/CPU utilization (vCPU-hours), memory consumption, network I/O, and storage IOPs.
• Cloud Integration: Native tools like AWS CloudWatch or Google Cloud Monitoring provide this data.
• Example: Metering shows an agent's fine-tuned model consumes an average of 2.5 GPU-hours per day on an AWS g5.xlarge instance.