Prometheus

Prometheus actively scrapes metrics from instrumented targets at defined intervals, rather than passively receiving pushes. This model simplifies service discovery in Kubernetes, where the server reaches out to ephemeral pod endpoints directly. Each scrape pulls a text-based snapshot of the target's current state over HTTP, typically from a /metrics endpoint. This architecture inherently provides a health check: a failed scrape immediately signals an unavailable service. It also centralizes collection control, allowing operators to manage scrape intervals, timeouts, and TLS settings from the server side without modifying applications.

PromQL (Prometheus Query Language) is a functional, read-only language designed specifically for selecting and aggregating time-series data. It enables complex calculations directly within the database, such as computing the 99th percentile latency over a sliding window or the rate of HTTP errors per second. Key capabilities include:

• Instant vector selectors for current value snapshots
• Range vector selectors with functions like rate() and increase() for counter data
• Aggregation operators (sum, avg, topk) that can group by arbitrary Kubernetes labels like namespace or pod
• Binary arithmetic between metrics for calculating ratios and deltas This expressive language allows SREs to define precise alerting rules and build dynamic dashboards without pre-processing data.

Prometheus stores every time series with a metric name and a set of arbitrary key-value pairs called labels. This multi-dimensional model is fundamentally aligned with Kubernetes' own labeling system. A metric like http_requests_total can be tagged with {method="POST", handler="/api", status="200"}. This allows for dynamic aggregation: you can instantly query the total request rate across all handlers or drill down to a single endpoint's error rate. Labels provide the flexibility to slice and dice operational data without pre-defining hierarchies, making it trivial to pivot from cluster-wide views to individual pod performance.

The Prometheus server includes a custom time-series database (TSDB) that operates on local storage, requiring no external database cluster. It uses a write-ahead log (WAL) for crash resilience and a two-hour head block in memory for recent data before compacting it into immutable, compressed blocks on disk. This design delivers high write throughput for millions of active series while maintaining efficient query performance. The local, self-contained storage model is critical for air-gapped and disconnected environments, as it eliminates dependencies on external network-attached storage or distributed consensus systems, ensuring the monitoring stack remains fully operational in isolation.

Prometheus fires alerts by evaluating PromQL expressions against configured thresholds, but it delegates notification management to Alertmanager, a separate component. Alertmanager handles deduplication, grouping, and routing of alerts to receivers like PagerDuty, Slack, or email. It can inhibit certain alerts if a higher-priority alert is already firing—for example, suppressing individual pod alerts when a node-wide failure is detected. This separation of concerns keeps the core monitoring server focused on data collection and rule evaluation, while Alertmanager manages the complex logic of escalation policies and on-call rotations.

Prometheus natively integrates with the Kubernetes API to dynamically discover scrape targets. It watches for changes to Pods, Endpoints, Services, and Ingresses, automatically updating its target list as workloads scale or reschedule. Configuration is driven by relabeling rules that map Kubernetes metadata—like pod annotations (prometheus.io/scrape: "true") or namespace labels—into Prometheus target labels. This eliminates manual target configuration and ensures that monitoring adapts instantly to the ephemeral nature of containerized AI workloads, such as a new GPU-accelerated inference pod coming online.

The functional query language of Prometheus used to select and aggregate time-series data in real time. It enables operators to compute rates, ratios, and trends from raw counter and gauge metrics.

• Instant vectors: single point-in-time queries for dashboard panels
• Range vectors: time-windowed data for rate() and increase() calculations
• Binary operators: arithmetic and comparison logic for multi-metric correlation
• Aggregation operators: sum, avg, topk, and histogram_quantile for statistical analysis

The mechanism by which Prometheus automatically identifies scrape targets without manual configuration. In Kubernetes, it uses the API server to watch pods, services, and endpoints.

• kubernetes_sd_configs: native Kubernetes service discovery role
• Pod annotations: prometheus.io/scrape: "true" triggers automatic metric collection
• Relabeling: rewrites target labels before scraping to normalize metadata
• File-based SD: static JSON files for targets in air-gapped environments with no API access

Pre-computed PromQL expressions that are evaluated periodically and stored as new time-series. They reduce query latency and computational load for frequently accessed dashboards.

• Syntax: record: <metric_name> and expr: <PromQL expression> in rule files
• Evaluation interval: independent of scrape interval, typically 30-60 seconds
• Use cases: pre-aggregating high-cardinality data, computing SLO error budgets
• Air-gapped benefit: reduces ad-hoc query load on the Prometheus server when external caching layers are unavailable