Prometheus

Prometheus operates on a pull model, where its server actively scrapes metrics from configured HTTP endpoints exposed by applications. This contrasts with a push model. For LLM services, this involves instrumenting the application code or using exporters to expose key metrics like:

• Request latency (TTFT, inter-token latency)
• Throughput (tokens per second)
• Error rates and HTTP status codes
• GPU utilization and memory usage The server then stores these scraped metrics as time-series data in its local database.

Metrics in Prometheus are identified by a metric name and a set of key-value pairs called labels. This multi-dimensional data model allows for powerful filtering and aggregation. For LLM monitoring, labels can segment traffic by:

• Model version (model="gpt-4")
• Deployment endpoint (endpoint="/v1/completions")
• User or tenant ID for multi-tenancy
• Request characteristics (mode="streaming") A sample metric for LLM latency might be: llm_request_duration_seconds{model="llama-3-70b", endpoint="/chat", quantile="0.99"}.

PromQL is Prometheus's functional query language used to select and aggregate time-series data in real-time. It is essential for creating dashboards, alerts, and ad-hoc analysis. Key operations for LLM monitoring include:

• Rate calculation: rate(llm_requests_total[5m])
• Aggregation across labels: sum by (model) (rate(llm_tokens_generated_total[5m]))
• Percentile analysis: histogram_quantile(0.99, rate(llm_request_duration_seconds_bucket[5m]))
• Comparative operators: For alerting when error rates spike. PromQL enables SREs to calculate Service Level Indicators (SLIs) like error rate and latency percentiles directly from raw metrics.

The Alertmanager service handles alerts sent by the Prometheus server. It is responsible for deduplication, grouping, silencing, inhibition, and routing of alerts to various receivers like email, Slack, or PagerDuty. For LLM SLOs, Alertmanager can be configured to:

• Group all alerts related to a specific model deployment.
• Throttle notifications to prevent alert fatigue during ongoing incidents.
• Route critical alerts (e.g., SLO breach, P99 latency spike) to an on-call engineer while sending informational alerts (e.g., elevated token rate) to a chat channel. It works in conjunction with Prometheus's alerting rules defined in PromQL.

Prometheus uses exporters to collect metrics from systems that do not natively expose a Prometheus format. Key exporters for LLM infrastructure include:

• Node Exporter for host-level metrics (CPU, memory, disk).
• NVIDIA DCGM Exporter or GPU Operator for detailed GPU metrics.
• cAdvisor for container resource usage. Service discovery automates the discovery of scrape targets in dynamic environments like Kubernetes. Prometheus can query the Kubernetes API to automatically find all running LLM inference pods and begin scraping their metrics, eliminating manual configuration as pods scale.

Prometheus is a core data source for Grafana, which provides visualization dashboards for LLM metrics. Dashboards display real-time charts for:

• Latency percentiles (P50, P90, P99)
• Request rate and error budget burn
• GPU utilization and token throughput While Prometheus excels at metrics, it is often used alongside OpenTelemetry (OTel) for a complete observability picture. OTel handles distributed tracing for request flows across microservices (e.g., from API gateway to LLM to post-processing). Metrics from OTel can be exported to Prometheus, and traces can be correlated with metric alerts for faster root cause analysis (RCA).

A method of profiling requests as they flow through a distributed LLM application stack by recording timing and metadata for individual operations (spans) across service boundaries. While Prometheus aggregates metrics, distributed tracing provides a detailed, request-level view. It is critical for diagnosing latency issues in complex pipelines involving multiple microservices, databases, and external API calls. Implemented using standards like OpenTelemetry Trace, it helps answer questions like:

• Which service or model stage is causing high P99 latency?
• What is the breakdown of time spent in prefill vs. decode phases?
• How does a retrieval-augmented generation (RAG) pipeline spend its time between retrieval and generation?

A target value or range for a Service Level Indicator that defines the acceptable performance and reliability of an LLM-powered service. SLOs are the contractual heart of site reliability engineering (SRE). Prometheus metrics are used to measure SLI compliance and calculate the error budget—the allowable amount of unreliability. For LLM services, common SLOs are defined for:

• Latency: "99% of requests must have a Time to First Token < 500ms."
• Availability: "The model API must be available 99.95% of the time."
• Quality: "Less than 1% of responses may contain a detected hallucination." Violating an SLO consumes the error budget, which guides the pace and risk of new deployments.

Release strategies for safely deploying new LLM models or application versions by exposing them to a controlled subset of traffic. These strategies rely heavily on Prometheus for comparative metric analysis.

• Canary Deployment: The new version serves a small percentage of live traffic (e.g., 5%). Prometheus metrics for the canary cohort (latency, error rate, TPS) are compared against the baseline to validate performance before a full rollout.
• Shadow Deployment: The new version processes all live requests in parallel, but its outputs are discarded. This allows for full-scale performance and correctness testing (e.g., comparing output embeddings for drift) with zero user risk. Prometheus monitors resource consumption of the shadow model.

A method of quality control using statistical methods, like control charts, to monitor and control a process. In LLM operations, SPC is applied to Prometheus metrics to distinguish normal variance from significant anomalies that require intervention. Key concepts include:

• Control Limits: Statistically derived bounds (e.g., 3-sigma) within which metric variation is considered common-cause.
• Mean Time to Recovery (MTTR): A key SPC-derived metric measuring the average time to restore service after a control limit breach.
• Root Cause Analysis (RCA): The systematic process triggered by an SPC alert to find the fundamental cause of a metric anomaly, such as a model regression, infrastructure failure, or data drift.