Container Logging

Containers write logs to standard output (stdout) and standard error (stderr) streams. A log driver is the low-level mechanism that captures these streams. Common drivers include:

• json-file: The default Docker driver, writing structured JSON logs to a local file.
• journald: Directs logs to the systemd journal.
• syslog: Forwards logs to a syslog server.
• gelf: Sends logs in the Graylog Extended Log Format to a remote server.
• awslogs, gcplogs: Cloud-native drivers for AWS CloudWatch and Google Cloud Logging.

In Kubernetes, the kubelet on each node collects logs from all pods via the container runtime's log driver, typically storing them in files on the node.

Because containers are ephemeral, a dedicated process is needed to collect and forward logs from the node. Two primary patterns are used:

DaemonSet Agent (Node-Level): A pod (e.g., Fluentd, Fluent Bit, Logstash, Datadog Agent) deployed as a Kubernetes DaemonSet runs on every node. It tails the log files from /var/log/containers/ and /var/log/pods/, enriches them with metadata (pod name, namespace, labels), and forwards them upstream. This is the most resource-efficient and common pattern.

Sidecar Container (Pod-Level): A secondary container runs alongside the application container within the same pod, sharing a volume to access its logs. The sidecar (often a lightweight agent like Fluent Bit) processes and forwards them. This is used for specialized needs, like parsing custom-format logs or when the application writes to a file instead of stdout.

This central layer receives, transforms, and indexes log streams from all agents. Key functions include:

• Parsing & Structuring: Converting raw log lines into key-value pairs using regex, JSON, or other parsers.
• Enrichment: Adding contextual metadata (e.g., Kubernetes labels, hostname, severity level).
• Filtering & Routing: Dropping irrelevant logs or routing them to different destinations based on tags.
• Buffering: Temporarily storing logs in memory or on disk during network outages to prevent data loss.

Common technologies here include Fluentd (as an aggregator), Vector, Logstash, or managed services like Amazon Kinesis Data Firehose.

The final destination for processed logs, enabling long-term retention, fast search, and analysis. Solutions are chosen based on scale, cost, and query needs.

Time-Series & Log-Specific Databases:

• Elasticsearch: The dominant open-source choice, often paired with Kibana for the ELK Stack. Excellent for full-text search and aggregation.
• Loki: A log aggregation system from Grafana Labs, optimized for storing and querying logs, not metrics. It indexes metadata (labels) and compresses log content, offering a cost-effective alternative to Elasticsearch.

Object Storage (Long-Term/Compliance):

• Amazon S3, Google Cloud Storage, Azure Blob Storage: Used for cost-effective, durable archival of log data, often in Parquet or ORC formats for analytical querying with engines like Trino or Athena.

The user-facing layer where engineers and SREs interact with log data.

Visualization Dashboards:

• Kibana (for Elasticsearch): Provides rich dashboards, ad-hoc queries, and data exploration.
• Grafana (for Loki, Elasticsearch, and others): Offers powerful graphing and dashboarding, often used in conjunction with metrics and traces for unified observability.

Querying & Search: Using query languages like:

• KQL (Kibana Query Language) or Lucene syntax in Elasticsearch.
• LogQL in Loki.

Alerting: Configuring rules to trigger notifications (e.g., PagerDuty, Slack, email) based on log patterns, such as a sudden spike in ERROR-level messages or the presence of specific security-related strings.

Critical governance controls for production logging pipelines.

Security & Privacy:

• PII Redaction/Data Masking: Scrubbing sensitive data (emails, credit card numbers) at the agent or processing layer.
• Encryption in Transit/At Rest: Using TLS for log transmission and storage encryption.
• Access Controls: RBAC policies to restrict who can view or query logs.

Compliance & Forensics:

• Immutable Storage/WORM: Using write-once-read-many storage or object lock features to meet regulatory requirements (e.g., SEC Rule 17a-4, HIPAA).
• Tamper-Evident Logs: Using cryptographic hashing (e.g., in Sigstore's Rekor) to prove log integrity.

Lifecycle Management:

• Retention Policies & TTL: Automatically deleting or archiving logs based on age (e.g., 30 days hot in Elasticsearch, 7 years cold in S3).
• Cost Optimization: Managing log volume through sampling, filtering verbose DEBUG logs, and choosing appropriate storage tiers.