Structured Logging

Structured logs are emitted in a consistent, schema-enforced format, most commonly JSON, but also Protocol Buffers or Avro. This allows automated systems to parse logs without fragile text scraping.

• Example: {"timestamp": "2024-01-15T10:30:00Z", "level": "INFO", "event": "llm_inference", "model": "gpt-4", "request_id": "abc-123", "input_tokens": 150, "duration_ms": 1250}
• Benefit: Enables immediate ingestion by log aggregators like Loki, Elasticsearch, or Datadog for indexing and analysis.

Every log entry contains a standardized set of fields (keys) with associated values. This consistency is critical for filtering, grouping, and performing statistical analysis across millions of log lines.

• Core Fields for LLM Ops: model_version, prompt_hash, user_id, latency_ms, output_tokens, finish_reason, total_cost.
• Practice: Teams define and enforce a logging schema to ensure all services emit the same critical fields, making cross-service tracing and cohort analysis possible.

Beyond simple status messages, structured logs capture the full context of an event. For LLM requests, this includes the complete prompt, the generated response, retrieved context chunks, and tool calls.

• Enables: Detailed debugging of specific failures, reproduction of user sessions, and analysis of prompt effectiveness.
• Consideration: Sensitive data (PII) in prompts/responses must be redacted or tokenized at ingestion to comply with privacy policies, while retaining the necessary context for debugging.

Because fields are indexed, you can filter and aggregate logs by any combination of high-cardinality dimensions, such as a specific user_id, prompt_template_id, or deployment_region.

• Use Case: "Show the P99 latency for model llama-3-70b for users in cohort beta_testers over the last 24 hours."
• Contrast: Unstructured logging makes this type of precise, multi-dimensional query impossible without extensive pre-processing.

Structured logs serve as a primary data source for deriving operational metrics and configuring precise alerts. Log aggregation platforms can calculate rates, percentiles, and averages from log fields.

• Metric Generation: Count of error-level logs per model, average tokens_per_second per hardware type, 95th percentile of time_to_first_token.
• Alerting: Trigger an alert when the error rate for a specific model_version exceeds 1% over a 5-minute window, or when hallucination_score crosses a defined threshold.

Structured logging is one pillar of the three pillars of observability, alongside distributed traces and metrics. They are designed to work together through shared context, like a trace_id.

• Correlation: A log entry from an LLM inference service can contain the OpenTelemetry trace_id, allowing engineers to jump from a high-latency alert to the full distributed trace of that request across multiple microservices.
• Unified View: Modern observability backends (e.g., Grafana) can correlate logs, traces, and metrics on the same dashboard, providing a holistic view of system state.

Unlike traditional plain-text logs, structured logging outputs data in a consistent, schema-like format, most commonly JSON. This transforms logs from human-readable narratives into machine-readable events. Each log entry becomes a discrete object containing key-value pairs.

Key fields for an LLM request might include:

• request_id: A unique identifier for tracing.
• timestamp: ISO 8601 format for precise timing.
• model_id: The specific model version invoked.
• input_tokens: Count of tokens in the prompt.
• output_tokens: Count of tokens in the response.
• latency_ms: Total request duration.
• user_id: For usage analytics and abuse detection.

This structure enables automated ingestion by systems like the OpenTelemetry collector or directly into databases, bypassing the need for complex and error-prone regular expression parsing.

In a microservices architecture typical of LLM applications, a single user request may traverse multiple services (gateway, orchestration, model endpoint, cache). Structured logs are the foundational data source for distributed tracing.

By including a universal trace_id and span_id in every log entry across all services, engineers can reconstruct the complete journey of a request. This is critical for:

• Diagnosing high latency: Identifying which service or model call is the bottleneck.
• Root Cause Analysis (RCA): Pinpointing where and why an error originated.
• Understanding dependencies: Mapping the flow between retrieval systems, LLMs, and post-processing filters.

Tools like Jaeger or Tempo aggregate these traces, using the structured log data to provide visualizations of request flows and performance waterfalls.

Structured logs feed directly into time-series databases (e.g., Prometheus) and analytics engines, enabling the calculation of Service Level Indicators (SLIs) and adherence to Service Level Objectives (SLOs).

Common derived metrics include:

• Latency Percentiles (P50, P90, P99): Calculated from the latency_ms field across all requests.
• Tokens per Second (TPS): Throughput derived from output_tokens and latency_ms.
• Error Rate: Percentage of requests with a status_code field indicating failure.
• Cost per Request: Estimated using token counts and known model pricing.

By aggregating these structured events, teams can create Grafana dashboards for real-time monitoring, set alerts based on SLO error budgets, and perform cohort analysis to compare performance across different user segments or model versions.

Beyond system metrics, structured logs are the primary vehicle for capturing model-centric telemetry, which is vital for detecting output drift and concept drift.

Critical behavioral fields to log:

• input_embedding (or a hash): To track distribution shifts in prompts.
• output_embedding: To monitor for embedding drift in responses.
• perplexity: The model's intrinsic confidence score on its own output.
• tool_calls: Detailed record of any function or API calls executed by the agent.
• safety_scores: Outputs from content moderation filters.

By logging this structured data, teams can compare statistical distributions against a golden dataset baseline. Anomaly detection systems can then flag significant deviations, triggering investigations into potential model degradation or emerging failure modes.

OpenTelemetry (OTel) provides the industry-standard framework for implementing structured logging (as part of its logs signal) alongside traces and metrics. It ensures consistency and vendor-agnostic instrumentation.

Typical implementation flow:

1. Instrumentation: Use OTel SDKs to automatically enrich logs with trace_id, span_id, and resource attributes (e.g., service.name, deployment.environment).
2. Structured Data: Log all events as JSON objects with semantic conventions where possible (e.g., using gen_ai.* attributes for LLM-specific data).
3. Collection: The OTel Collector receives, processes (e.g., filtering, batching), and exports log data to backends like Loki, Elasticsearch, or cloud monitoring services.
4. Context Propagation: Ensures the trace_id from a distributed trace is automatically included in all related log entries, providing seamless correlation.

This approach decouples instrumentation from the final analysis backend, providing future-proof flexibility.

Effective structured logging requires deliberate schema design and governance to avoid data chaos.

Core Best Practices:

• Define a Schema: Establish and version a contract for required and optional fields (e.g., llm_log_schema_v1).
• Use Consistent Naming: Adopt snake_case for field names and avoid arbitrary changes.
• High Cardinality Caution: Be mindful of fields with vast unique values (like full user prompts) which can overwhelm indexing; consider logging hashes or summaries instead.
• Separate Concerns: Log system events (startup, errors) separately from request/response payloads for cleaner analysis.
• Include Context: Always log enough context (request IDs, user IDs, model IDs) to reconstruct the event's story.
• Security & PII: Scrub personally identifiable information and secrets before logging. Use allow-lists, not block-lists.

A well-designed schema turns logs from a debugging afterthought into a high-fidelity telemetry stream for Statistical Process Control (SPC) and continuous improvement of LLM services.

Anomaly detection involves identifying patterns in metrics, logs, or model outputs that deviate significantly from expected behavior. Structured logs are a primary data source for detecting anomalies in LLM operations.

• Log-Based Detection: Algorithms analyze log streams for unusual patterns, such as a sudden surge in messages with error_code="context_length_exceeded" or a change in the distribution of log severity levels.
• Approaches: Methods range from simple threshold-based alerts on metric derivatives to machine learning models that learn normal log sequence patterns and flag deviations.

Root Cause Analysis is a systematic process for identifying the fundamental causal factors of an incident. Well-structured logs are the evidentiary backbone of effective RCA in complex LLM systems.

• Process: Engineers use correlated traces, metrics, and logs to reconstruct the event timeline. Key log attributes for RCA include:
  • request_id, user_id, model_version.
  • input_tokens, output_tokens, finish_reason.
  • downstream_service_latency and error_stack_trace.
• Outcome: The goal is to move from symptom (e.g., "high latency") to proximate cause ("vector database timeout") to root cause ("misconfigured connection pool") and implement preventive fixes.