Prometheus

Prometheus stores all data as time series, which are streams of timestamped values belonging to the same metric. Each time series is uniquely identified by its metric name and a set of key-value pairs called labels. This model enables powerful filtering, aggregation, and slicing of data.

• Example Metric: http_requests_total{method="POST", handler="/api", status="200", instance="10.0.0.1:8080"}
• Labels like method, handler, and instance allow querying for specific subsets, such as all POST requests to the /api endpoint that returned a 200 status code.
• This is a fundamental shift from hierarchical or graph-based models, providing immense flexibility for dimensional analysis.

PromQL is a functional, expression-based query language designed for working with the multi-dimensional data model. It allows for real-time aggregation, slicing, prediction, and alerting directly on the collected time series data.

• Core Operations: Range selection (http_requests_total[5m]), filtering (http_requests_total{status=~"5.."}), aggregation (sum by(handler) (rate(http_requests_total[5m]))), and mathematical functions.
• Uses: Powering ad-hoc graphs in the expression browser, defining alerting rules, and feeding data into external dashboards like Grafana.
• It operates on the principle of selector matching, where labels define the set of time series to include in a calculation.

Prometheus primarily uses a pull model over HTTP, where the Prometheus server itself scrapes metrics from configured targets at defined intervals. This contrasts with a traditional push model where applications send data to a central server.

• How it Works: Targets expose metrics via an HTTP endpoint (typically /metrics). Prometheus discovers these targets via static configs or service discovery (Kubernetes, Consul) and periodically scrapes them.
• Key Advantages:
  • Operational Simplicity: You can run a Prometheus server without knowing all target IPs upfront; it discovers them.
  • Reliability: Prometheus controls the scrape rate and can detect when a target is down.
  • Multi-Tenancy: Easy to run multiple independent Prometheus servers for different teams or reliability zones.

To monitor dynamic environments like Kubernetes, Prometheus natively integrates with various service discovery mechanisms. It automatically discovers and begins scraping new targets as they appear, and stops scraping removed ones.

• Supported Platforms: Kubernetes, Consul, Amazon EC2, Azure, Google Cloud, Docker Swarm, and more via file-based SD.
• Dynamic Relabeling: Before scraping, target metadata (like Kubernetes pod labels) can be transformed into metric labels via relabeling rules. This is how pod names, namespaces, and other container metadata become attached to every scraped metric.
• This feature is critical for making sense of ephemeral, auto-scaling microservices, as it automatically maintains an accurate inventory of what to monitor.

Prometheus includes a built-in Alertmanager as a separate component. Alerting rules are defined in Prometheus using PromQL; when an expression's result becomes a vector of time series, it fires an alert to the Alertmanager.

• Alerting Rule Definition: Rules live in Prometheus configuration files and specify a PromQL expression, a duration for which it must be true, and labels/annotations for the alert.
• Alertmanager's Role: It handles deduplication, grouping, inhibition, and routing of alerts to receivers like email, PagerDuty, Slack, or webhooks.
• Key Concepts:
  • Grouping: Bundles similar alerts (e.g., all database latency alerts in a cluster) into a single notification.
  • Inhibition: Suppresses certain alerts if another, higher-severity alert is already firing (e.g., don't alert on a service being down if its entire host is down).

Prometheus is designed to be simple to run and operate. Each Prometheus server is standalone and does not depend on network storage or other remote services for its core functions.

• Storage: Uses a custom, highly efficient local time-series database on disk. While it supports remote read/write APIs for long-term storage, its primary storage is local, making it resilient to network partitions.
• Single Binary: The main Prometheus server is a single, statically linked binary with no external dependencies (like databases).
• Federation: Allows a hierarchical setup where a higher-level Prometheus server can scrape aggregated data from lower-level servers, enabling cross-datacenter views or tiered aggregation.
• This design makes it a robust, 'always-on' monitoring system that can be deployed per team, per cluster, or per datacenter.