Pipeline Telemetry

The Golden Signals are four high-level metric categories, derived from Google's Site Reliability Engineering practices, that provide a universal starting point for monitoring any service or data pipeline.

• Latency: The time it takes to service a request or process a record. Monitor p50, p95, and p99 percentiles.
• Traffic: A measure of demand on the system (e.g., requests per second, bytes ingested per minute).
• Errors: The rate of requests that fail, either explicitly (HTTP 500) or implicitly (business logic failures).
• Saturation: How 'full' a resource is, relative to its capacity (e.g., CPU load, memory usage, queue depth).

Tracking these signals provides a holistic view of pipeline health and user experience.

Pipeline telemetry follows a defined lifecycle from generation to analysis. Understanding this flow is key to building an effective observability stack.

1. Instrumentation: Code is added to pipeline components to emit raw signals. This can be manual, automatic, or via an SDK like OpenTelemetry.
2. Collection & Export: Emitted signals are gathered by an agent or library and sent (exported) to a backend system. Formats include OTLP (OpenTelemetry Protocol), Prometheus scrape, or vendor-specific protocols.
3. Processing & Storage: Backends parse, filter, aggregate, and index the data for efficient querying (e.g., into time-series databases, log indexes, trace stores).
4. Analysis & Action: Engineers use dashboards, query languages, and alerting rules to analyze the data, identify issues, and trigger incident management workflows.

This lifecycle is increasingly managed as Observability as Code.

The Golden Signals are four key metrics proposed for monitoring the health of any service or data pipeline: Latency, Traffic, Errors, and Saturation. They provide a concise, high-level dashboard for system health.

• Latency: The time to process a unit of data (e.g., P95 end-to-end latency).
• Traffic: The demand on the system (e.g., messages/second, bytes ingested).
• Errors: The rate of failed requests or malformed data (e.g., HTTP 5xx errors, parsing failures).
• Saturation: How "full" the system is (e.g., CPU utilization, queue depth, memory pressure).

Monitoring these signals provides immediate insight into pipeline performance and degradation.

A Pipeline Service Level Objective (SLO) is a target level of reliability or performance for a data pipeline, defined as a percentage over a rolling time window. It is a formal, measurable goal derived from business requirements.

• Examples: "99.9% of records processed within 5 minutes of ingestion" or "Pipeline availability of 99.95%."
• Drives Action: SLOs are used to calculate an Error Budget—the allowable amount of unreliability. Exhausting the budget triggers a focus on stability over new features.
• Telemetry Dependency: SLO compliance is measured directly from pipeline telemetry (latency metrics, error counters, uptime probes).

A Health Check is a periodic probe (e.g., an HTTP endpoint /health or a lightweight database query) used to determine the operational status of a pipeline component. It answers two questions: Liveness (is it running?) and Readiness (is it ready to serve traffic?).

• Liveness Probe: Detects a crashed or deadlocked process. Failure triggers a restart.
• Readiness Probe: Determines if a service is initialized and can accept work (e.g., connected to dependencies). Failure removes it from a load balancer.
• Synthetic Monitoring: Can be considered an external health check that validates the entire pipeline's data flow from a user perspective.

Consumer Lag is a critical metric for streaming pipelines, representing the delay between the latest data produced to a message queue (like Apache Kafka) and the data last consumed and processed by a pipeline. It is measured in time (e.g., 2 seconds) or number of messages.

• Key Health Indicator: Growing lag signals that the consumer cannot keep up with the producer, often due to processing bottlenecks, resource constraints, or errors.
• Directly Impacts Freshness: High lag means downstream analytics or models are operating on stale data.
• Alerting Threshold: Teams commonly set alerts for when lag exceeds a business-acceptable threshold (e.g., > 1 minute).