Grafana Dashboards

These panels track the speed and efficiency of LLM inference. Time to First Token (TTFT) measures initial response delay, while Inter-Token Latency tracks the fluency of streaming output. Tokens per Second (TPS) quantifies throughput. Visualizing latency percentiles (P50, P90, P99) is critical for understanding tail performance and user experience. Alerts are typically configured on P99 latency breaches.

This section monitors service reliability against defined Service Level Objectives (SLOs). Key panels include:

• HTTP Status Code Rates (e.g., 4xx, 5xx)
• Model-Specific Error Counts (e.g., context window overflows, generation errors)
• SLO Burn Rate visualizations showing consumption of the Error Budget
• Availability percentage over a rolling window Tracking these metrics ensures the service meets its reliability targets and guides deployment risk.

Panels here correlate performance with infrastructure spend and efficiency. They display:

• GPU/CPU Utilization and memory usage from the underlying inference servers
• KV Cache memory consumption trends
• Concurrent Request counts and batch sizes
• Estimated Cost per Request or per token, often derived from cloud provider metrics This data is essential for Inference Optimization and Cost and Resource Management, helping to right-size deployments.

These panels track the statistical health of the LLM's generations. They visualize signals of Output Drift and Concept Drift by comparing current outputs to a Golden Dataset baseline. Metrics may include:

• Perplexity scores for generated text
• Embedding Drift detected via statistical tests on vector distributions
• Hallucination Detection rates from real-time classifiers
• Custom quality score distributions from user feedback or automated evaluators

This component provides visibility into user traffic patterns and release strategies. It includes:

• Request Volume and Unique User trends over time
• Cohort Analysis panels comparing metrics across different user segments, model versions, or A/B test groups
• Visualizations for Canary Deployment and Shadow Deployment performance, showing key differences between old and new model versions
• Geographic or endpoint distribution of traffic

A robust dashboard connects LLM metrics to the broader telemetry ecosystem. Key integrations include:

• Prometheus as the primary metrics data source for time-series data
• Distributed Tracing visualizations (e.g., from Jaeger) linked to high-latency requests, showing the full LLM application stack trace
• Structured Logging aggregates displayed via Loki or similar log panels for debugging specific failures
• Alertmanager status showing firing alerts related to LLM Service Level Indicators (SLIs)

These metrics quantify the responsiveness and capacity of your LLM service, directly impacting user experience and infrastructure efficiency.

• Time to First Token (TTFT): Measures the delay before the response stream begins. High TTFT indicates bottlenecks in the initial prompt processing (prefill phase).
• Inter-Token Latency: The average time between tokens during streaming. This dictates the perceived 'speed' of the response.
• Tokens per Second (TPS): The system's output generation throughput. Monitor this under different load levels and batch sizes.
• Latency Percentiles (P50, P90, P99): Critical for understanding tail latency. While P50 shows the median, P99 reveals the worst-case delays experienced by 1% of requests, often caused by resource contention or cold starts.

Monitor the consumption of underlying hardware resources to ensure efficient scaling, prevent bottlenecks, and control costs.

• GPU Utilization (%): The primary indicator of compute load. Consistently high usage suggests good efficiency, while spikes may correlate with latency.
• GPU Memory Usage: Tracks the consumption of VRAM, which is critical for model loading and the KV Cache. Approaching limits will cause out-of-memory errors.
• System Memory & CPU: While less critical than GPU, these can become bottlenecks for pre/post-processing, tokenization, and network handling.
• Continuous Batching Efficiency: Infer metrics like batch size distribution and idle time to gauge how effectively the serving system packs requests.

Track the stability and success rate of your LLM endpoints to maintain service-level agreements and user trust.

• Request Error Rate: The percentage of failed requests (e.g., 4xx, 5xx HTTP status codes, model loading failures).
• Token Generation Errors: Specific failures during the autoregressive decode phase.
• Service Level Indicators (SLIs) & Objectives (SLOs): Define and track SLIs like availability (successful requests / total requests) and latency SLOs. Use Error Budget burn-down charts to guide deployment risk.
• Mean Time to Recovery (MTTR): Track the duration from incident detection to full resolution to improve operational resilience.

Monitor for statistical shifts in model outputs to detect degradation, hallucinations, or unintended behavioral changes.

• Output Drift: Measure changes in the distribution of output properties (e.g., response length, sentiment, toxicity scores) against a Golden Dataset baseline.
• Embedding Drift: Detect shifts in the vector space of generated embeddings, which can break downstream semantic search or classification systems.
• Hallucination Rate: Percentage of outputs flagged by dedicated detection systems as unsupported or factually incorrect.
• Perplexity: An intrinsic measure of the model's prediction confidence on a reference dataset. Unexplained increases can signal issues.

Analyze request volume, user segments, and cost drivers to inform capacity planning, feature development, and business intelligence.

• Requests per Second/Minute: Overall traffic volume and its diurnal patterns.
• Input/Output Token Counts: The primary drivers of inference cost. Track average and total tokens per request.
• Cohort Analysis: Segment metrics by user group, model version (e.g., A/B tests, Canary Deployments), or endpoint to identify disparate performance or quality.
• Model Cache Hit Rate: For systems using cached responses or KV Cache optimizations, this rate indicates efficiency gains.

Correlate metrics with detailed request traces and structured logs to enable deep Root Cause Analysis (RCA).

• Distributed Tracing: Use OpenTelemetry (OTel) traces to visualize the full request lifecycle across microservices (e.g., auth, model, post-processing). Link high latency metrics to specific spans.
• Structured Logging: Ingest JSON-formatted logs containing request IDs, prompt fingerprints, and error contexts into Grafana's Loki or similar. Cross-reference with metric spikes.
• Anomaly Detection: Configure alerts not just on static thresholds, but on statistical deviations from historical baselines using methods like Statistical Process Control (SPC).
• Feedback Loop Instrumentation: Track metrics related to user feedback (e.g., thumbs up/down rates) to correlate system performance with perceived quality.

A Service Level Indicator (SLI) is a quantitatively measured aspect of an LLM service's performance that is visualized and tracked on a Grafana dashboard. It is the raw measurement that defines service reliability.

• Common LLM SLIs include:
  • Latency: Time to First Token (TTFT), inter-token latency, end-to-end request duration.
  • Availability: The proportion of successful requests (e.g., HTTP 200 responses).
  • Quality: Metrics like output correctness score or hallucination rate (often from external evaluation systems).
• SLIs are the foundational data points plotted on Grafana graphs to assess compliance with Service Level Objectives (SLOs).

Distributed tracing provides a detailed, request-centric view of performance by tracking an LLM API call as it propagates through a complex microservices architecture. While Grafana excels at time-series metrics, traces explain the "why" behind latency spikes.

• A trace is composed of nested spans, each representing an operation (e.g., "call vector database," "run model inference," "format response").
• Tracing helps pinpoint the specific service or component (e.g., overloaded GPU, slow retrieval) causing high tail latency (P99).
• Tools like Jaeger or Tempo store trace data, which can be correlated with metrics in Grafana using exemplars or linked dashboards.

A canary deployment is a release strategy for LLM models where a new version is rolled out to a small subset of production traffic. Grafana dashboards are critical for comparing the canary's performance against the stable baseline.

• Dedicated dashboard panels filter metrics by a label like model_version="canary-v2".
• Engineers monitor for regressions in key SLIs: increased latency, higher error rates, or drift in output quality scores.
• This allows for data-driven rollback decisions before a faulty model impacts all users. Shadow deployments are a related, zero-risk variant where canary outputs are logged but not returned to users.

Statistical Process Control (SPC) is a methodology applied via Grafana to monitor LLM performance for stability and to detect anomalies. It uses control charts to distinguish normal variance from significant issues.

• Control charts on Grafana plot a metric (e.g., average latency) over time with calculated control limits (typically ±3 standard deviations).
• Points outside the control limits, or specific patterns (e.g., 7 points in a row trending up), signal a process that is "out of control"—potentially indicating model degradation, infrastructure failure, or a data drift event.
• This provides an objective, statistical basis for alerts, moving beyond simple static thresholds.