Log Levels

Log levels are arranged in a strict hierarchy of increasing severity. This allows for filtering: setting a level like WARN will capture all WARN, ERROR, and FATAL messages, but exclude INFO and DEBUG. Common hierarchies include:

• DEBUG: Detailed internal state for developers.
• INFO: Confirmation of normal operations.
• WARN: Potentially harmful situations.
• ERROR: A failure in a specific operation.
• FATAL/CRITICAL: Severe errors causing application abort.

Levels serve distinct operational purposes. INFO, WARN, ERROR, and FATAL are for production monitoring, alerting, and understanding system health. DEBUG is purely diagnostic, providing granular, often computationally expensive details about internal state, variable values, and execution paths. It is typically disabled in production to preserve performance and reduce log volume.

Effective logging at any level requires structured logging with rich, machine-readable context. Each log entry should be more than a message; it must include immutable metadata such as:

• Timestamp (ISO 8601).
• Correlation/Trace ID for distributed tracing.
• User/Session ID.
• Tool/Function Name invoked.
• Request Parameters (with PII redaction).
• Host/Service Identifier. This context is critical for root cause analysis and forensic readiness.

Logging verbosity directly impacts system performance and operational cost. DEBUG-level logging can generate orders of magnitude more data than INFO, consuming CPU for generation, network bandwidth for aggregation, and storage. A log retention policy and strategic use of sampling for verbose levels are essential for cost control. Real-time monitoring typically focuses on WARN and above to avoid alert fatigue.

For audit logging, specific levels are often mandated. Regulatory frameworks like GDPR, HIPAA, and SOX require the immutable recording of all security-relevant events, which typically map to INFO (access grants), WARN (policy violations), and ERROR (authentication failures). These logs must be tamper-evident and stored in WORM (Write-Once, Read-Many) storage to ensure non-repudiation and support the chain of custody for digital evidence.

Log levels are one pillar of the observability triad, alongside metrics and distributed traces. They integrate into broader platforms:

• SIEM (Security Information and Event Management): Ingests WARN and above for threat detection.
• APM (Application Performance Monitoring): Correlates ERROR logs with trace spans and service metrics.
• OpenTelemetry: Can bridge semantic conventions between log levels and trace severity. Effective log aggregation and parsing are required to make this data actionable.

The practice of writing log messages as machine-readable, key-value pairs (e.g., JSON) instead of unstructured plain text. This enables automated parsing, filtering, and analysis by log aggregation systems.

• Key Benefit: Enables powerful querying and alerting based on specific fields like user_id, tool_name, or error_code.
• Example: {"timestamp": "2024-05-15T10:30:00Z", "level": "ERROR", "message": "API call failed", "tool": "process_payment", "status_code": 429}

The process of collecting, centralizing, and indexing log data from multiple disparate sources—such as servers, containers, and applications—into a single platform for unified analysis.

• Core Tools: Platforms like Elastic Stack (ELK), Datadog, Splunk, and Grafana Loki.
• Purpose: Provides a holistic view of system health, enabling correlation of events across different services that constitute an AI agent's tool-calling workflow.

A method of observing and profiling requests as they flow through a distributed system of microservices. It uses unique trace IDs and span IDs to correlate all related events, including individual tool calls.

• Relation to Logs: Traces provide the high-level narrative of a request's journey, while log levels (DEBUG, ERROR) provide granular detail within each service or span.
• Standard: OpenTelemetry (OTel) is the dominant vendor-neutral framework for implementing distributed tracing.

A security solution that aggregates and analyzes log and event data from various sources in real-time to detect, alert on, and investigate security threats.

• Use Case for Tool Calling: A SIEM can ingest audit logs of AI agent tool use, apply correlation rules to detect anomalous patterns (e.g., rapid-fire failed authentication attempts across multiple tools), and trigger automated security responses.

A write-once, append-only log where entries cannot be altered or deleted after creation. This ensures data integrity, making logs legally admissible for compliance and forensic evidence.

• Implementation: Often uses Write-Once Read-Many (WORM) storage or cryptographic techniques like hash chains (e.g., Merkle Trees) to provide tamper-evidence.
• Critical for Audits: For regulatory compliance (e.g., SOX, GDPR), audit logs of tool calls must be immutable to prove they have not been modified post-facto.