OpenTelemetry for Memory

Logs are timestamped, structured records of discrete events that provide operational context and a forensic audit trail.

Key log events in memory systems:

• Access Logs: Record all read/write operations with user/session IDs and resource identifiers.
• Eviction Logs: Document items removed from cache (e.g., evicted_key: "session_abc123", reason: "LRU").
• Schema Change Logs: Track modifications to memory data structures.
• Health Check Logs: Record the results of periodic dependency connectivity tests.

Logs should be enriched with trace_id and span_id for correlation with traces.

Resource attributes are immutable key-value pairs describing the source of the telemetry—the memory system itself. They attach essential context to all emitted traces, metrics, and logs.

Standard attributes include:

• service.name: e.g., "agent-memory-service".
• service.version: The deployed version of the memory system.
• deployment.environment: e.g., "production", "staging".

Custom resource attributes for memory systems:

• memory.store.type: e.g., "vector_database", "knowledge_graph".
• memory.embedding.model: e.g., "text-embedding-3-large".
• memory.index.algorithm: e.g., "HNSW".

This enables filtering and grouping telemetry by infrastructure characteristics.

OpenTelemetry Semantic Conventions are standardized naming schemes for attributes, ensuring consistency and interoperability across different observability tools. Using them is critical for effective analysis.

Relevant conventions for memory:

• Database Conventions: Use db.system (e.g., "redis", "pinecone"), db.operation (e.g., "GET", "QUERY"), and db.statement (for non-sensitive query info).
• General Conventions: Use error.type and error.message for failures.
• Custom Attributes: For domain-specific signals, prefix with memory. (e.g., memory.retrieval.score, memory.chunk.size_bytes).

Adherence to conventions allows for the creation of universal dashboards and alerts.

OpenTelemetry Exporters are components that send collected telemetry data to a chosen observability backend for storage, visualization, and analysis. The signals are useless without a destination.

• Protocol: Telemetry is typically sent via OTLP (OpenTelemetry Protocol) over gRPC or HTTP.
• Common Backends: Exporters exist for platforms like Jaeger (traces), Prometheus (metrics), Loki or Elasticsearch (logs), and commercial vendors (Datadog, New Relic, Grafana Cloud).
• Memory-Specific Views: In the backend, engineers build dashboards that correlate memory latency (memory.operation.duration) with cache performance (memory.cache.hit_ratio) and error rates to identify systemic issues.

A memory trace is a detailed, end-to-end record of all processing steps performed by an agentic memory system to fulfill a single request. In OpenTelemetry, this is a span within a larger distributed trace.

• Contents: Captures timing, internal operations (e.g., embedding generation, vector search, cache lookup), and any errors encountered.
• Purpose: Essential for debugging performance bottlenecks (e.g., slow vector similarity search) and understanding the complete lifecycle of a memory operation.
• Correlation: Uses a correlation ID to link all logs, metrics, and spans related to the same request across services.

A Memory Query API is the programmatic interface for searching and retrieving information from an agentic memory store. It is the primary point of interaction that OpenTelemetry instruments.

• Function: Accepts semantic or hybrid queries, often using vector embeddings, and returns ranked results.
• Observability Point: Key source for latency and throughput metrics. Trace spans are created for each query execution.
• Examples: A /v1/memory/query endpoint that takes a natural language question and returns relevant memory snippets with similarity scores.

A memory audit trail is a chronological, immutable log of all access and modification events within an agentic memory system. It provides a security and compliance layer complementary to performance telemetry.

• Logged Events: Records all create, read, update, and delete (CRUD) operations, along with user/agent identity, timestamps, and the data accessed.
• Purpose: Enforces accountability, supports forensic analysis after security incidents, and demonstrates compliance with regulations (e.g., GDPR right to erasure).
• Relation to OTel: While distinct from performance logs, audit events can be emitted as OpenTelemetry logs and enriched with trace context for full request lineage.

A memory dashboard is a visual interface that aggregates and displays key telemetry data from an agentic memory system. It is the consumer-facing layer built on top of OpenTelemetry-collected data.

• Typical Panels: Displays real-time graphs of latency percentiles, error rates, throughput, and system resource utilization.
• Data Sources: Pulls from observability backends where OpenTelemetry exports its data, such as Prometheus for metrics and Grafana Loki for logs.
• Function: Enables engineers and SREs to monitor system health at a glance, correlate incidents, and validate the impact of deployments.