Span Attributes

Span Attributes are defined as key-value pairs, where the key is a string and the value can be a string, integer, boolean, or array of primitive types. This structure allows for flexible, structured metadata.

• Keys are typically namespaced using dot notation for clarity (e.g., http.method, tool.name).
• Values must be serializable to support export to various backends.
• This simple format enables powerful filtering and aggregation in observability platforms.

Attributes can be added to a Span during its execution, but become immutable once the Span is ended. This guarantees data consistency for analysis.

• Prevents race conditions or post-hoc modifications that would corrupt trace data.
• Ensures the recorded attributes represent the true state of the operation at the time it occurred.
• Backend systems can safely index and aggregate this immutable data.

To ensure interoperability, attributes should follow OpenTelemetry Semantic Conventions—standardized naming for common telemetry signals.

• Tool/API Calls: Use conventions like rpc.method, http.status_code, db.operation.
• Agent Context: Attributes like agent.session_id or llm.vendor provide agent-specific context.
• Adherence to conventions allows for universal dashboards and correlation across different services and frameworks.

High-cardinality attributes (those with many unique values, like full user IDs or request parameters) must be managed carefully due to their impact on observability system performance and cost.

• High Impact: Attributes like http.url (full path) or user.id can explode index size.
• Best Practice: Truncate, hash, or move high-cardinality data to Span Events or logs, while keeping low-cardinality identifiers (like user.tier or http.route) as attributes for grouping.

Attributes provide the contextual 'who, what, where' needed to diagnose issues. For a failing tool call, attributes answer critical questions:

• What failed? tool.name="stripe_api", http.method="POST"
• Why did it fail? http.status_code=429, error.type="RATE_LIMIT"
• For whom? customer.tier="enterprise", deployment.env="prod-eu"
• This turns a generic error into a specific, actionable incident.

Span Attributes are not isolated; they form a bridge between traces, metrics, and logs through exemplars and context propagation.

• Exemplars: Specific trace data (including attributes) can be attached to metric data points, allowing drill-down from a latency spike to the specific slow traces.
• Log Correlation: By emitting the Trace ID and Span ID in application logs, all signals can be linked, with Span Attributes providing the shared operational context.

A Span is the fundamental unit of work in distributed tracing, representing a single, named, and timed operation. For tool call instrumentation, a span encapsulates the execution of one specific external API or software tool.

• Represents: A logical step in an agent's workflow (e.g., call_weather_api, query_database).
• Contains: A start/end timestamp, status code, and a collection of Span Attributes.
• Purpose: Provides the structural container to which all metadata about a tool call is attached.

Distributed Tracing is a method of observing requests as they propagate through a distributed system. In agentic systems, it tracks an agent's task execution across its internal components and all external tool calls.

• Mechanism: Uses a unique Trace ID to correlate all related spans.
• Output: A Trace—a directed acyclic graph of spans showing the complete end-to-end workflow.
• Value: Enables performance debugging (identifying slow tool calls) and understanding causality in complex, multi-step agent operations.

Span Events are structured, timestamped log records attached to a Span. They denote discrete, significant moments that occur during the execution of a tool call, complementing the static metadata in Span Attributes.

• Examples: cache.hit, retry.initiated, request.sent, response.first_byte, error.thrown.
• Difference from Attributes: Events are point-in-time occurrences, while Attributes describe the operation as a whole.
• Use Case: Provides a high-resolution audit trail of the internal steps within a single tool call span.

Trace Correlation is the technique of propagating a unique Trace ID across service boundaries to link spans from disparate systems into a single, coherent end-to-end trace. This is critical for tracking agent tool calls to external APIs.

• Implementation: The Trace ID is passed via standardized HTTP headers (e.g., traceparent) or framework-specific context propagation.
• Challenge: Requires the external tool/service to also be instrumented to accept and forward this context.
• Result: Allows observability backends to reconstruct the full agent journey, including third-party service latency.

OpenTelemetry Instrumentation refers to libraries and agents that automatically add observability code to an application. For tool calls, it auto-generates spans, metrics, and logs compliant with the OpenTelemetry standard.

• Function: Automatically creates spans for common HTTP/gRPC clients and popular SDKs, attaching standard attributes (HTTP method, status code, URL).
• Benefit: Drastically reduces manual coding effort for observability and ensures consistency of attribute naming across services.
• Ecosystem: Part of the CNCF-graduated OpenTelemetry project, providing vendor-agnostic instrumentation.

A Service Level Indicator (SLI) is a quantitative measure of a service's behavior from the user's (or agent's) perspective. For tool calls, SLIs are derived from Span Attributes and metrics.

• Common Tool Call SLIs:
  • Latency: Measured from span start/end timestamps.
  • Success Rate: Calculated from the http.status_code or error attribute.
  • Availability: Based on successful calls vs. all attempted calls.
• Purpose: SLIs are the raw measurements used to define Service Level Objectives (SLOs) and Error Budgets for agent dependencies.