Anomaly Detection

Anomaly detection for LLMs is predominantly an unsupervised learning task. Since 'normal' operational behavior is complex and anomalies are rare by definition, systems learn a baseline distribution of metrics (e.g., latency, token rate, embedding clusters) without pre-labeled examples of failures. Techniques like autoencoders, isolation forests, and Gaussian mixture models are used to model this baseline and flag significant deviations.

Effective detection requires correlating anomalies across diverse telemetry streams. Key signals include:

• Infrastructure Metrics: GPU utilization, memory pressure, network I/O.
• Model Performance Metrics: Latency percentiles (P99), Tokens per Second (TPS), Time to First Token (TTFT).
• Model Output Signals: Embedding drift, output perplexity, hallucination detection scores.
• Business Metrics: Error rate spikes, user feedback sentiment, API call patterns. A spike in P99 latency coinciding with a shift in output embeddings is a stronger failure signal than either in isolation.

LLM behavior must be evaluated over time. Anomalies are often contextual sequences, not single-point outliers. Statistical Process Control (SPC) charts monitor metrics for trends, shifts, or cyclic patterns. Techniques like LSTMs or change point detection algorithms analyze time-series data to distinguish between a temporary fluctuation and a sustained degradation, which is critical for distinguishing a brief load spike from a model performance regression.

The definition of 'normal' evolves. Concept drift occurs when user behavior, input data distribution, or the model itself changes. Static thresholds become obsolete. Adaptive systems use sliding windows or online learning to continuously update the baseline. This prevents false positives during legitimate transitions, such as a new feature rollout changing traffic patterns or a model fine-tuning intentionally altering output characteristics.

In production LLM operations, alert fatigue is a critical risk. Anomaly detection systems must be tuned for a very low false positive rate. Engineers rely on high-signal alerts to diagnose real issues like KV cache thrashing, continuous batching inefficiencies, or downstream API failures. Techniques involve ensemble methods, requiring consensus from multiple detection algorithms, and integrating with root cause analysis (RCA) pipelines to validate alerts before escalation.

Detection is not an isolated system. It feeds into and leverages the broader LLM observability ecosystem:

• Metrics: Stored in Prometheus and visualized on Grafana dashboards.
• Traces: Anomalous requests trigger detailed distributed tracing via OpenTelemetry (OTel).
• Logs: Correlated with structured logging for forensic analysis.
• Deployment: Alerts can trigger automated rollbacks in canary or shadow deployment strategies. This integration closes the loop from detection to remediation.

A latency spike is a sudden, significant increase in request response time, while tail latency (e.g., P99) refers to the worst-case delays experienced by a small percentage of requests. These anomalies indicate:

• Resource contention (GPU memory exhaustion, CPU throttling).
• Inefficient batching or queue saturation.
• Downstream dependency failures (database, vector store).
• Model degradation requiring recomputation. Monitoring Time to First Token (TTFT) and Inter-Token Latency percentiles is essential for detecting these performance regressions.

A sustained drop in Tokens per Second (TPS) or requests per second indicates the system is processing work slower than its established baseline. Key causes include:

• Hardware faults or thermal throttling in inference clusters.
• Inefficient KV Cache utilization leading to redundant computation.
• Suboptimal continuous batching strategies.
• Increased prompt complexity or output length without corresponding resource scaling. This anomaly directly impacts scalability and cost-per-token, requiring investigation into compute efficiency.

Output drift is a statistical change in the distribution of generated text (e.g., sentiment, toxicity, format). Embedding drift is a change in the vector space geometry of model-generated embeddings. Both signal:

• Unintended model updates or corrupted weights.
• Upstream data pipeline issues affecting context.
• Concept drift in the real-world domain the model serves. Detection involves comparing live outputs/embeddings against a golden dataset using statistical tests (e.g., Population Stability Index, KL-divergence) or monitoring embedding cluster centroids.

A sharp increase in the rate of failed requests (HTTP 5xx, 4xx) or model-specific errors (e.g., context window overflows, tokenization failures). This anomaly often precedes full service outages. Common triggers:

• Validation schema mismatches in API requests.
• Rate limit exhaustion or quota errors from upstream providers (e.g., OpenAI, Anthropic).
• Hallucination detection or safety filter systems triggering excessively.
• Infrastructure failures in load balancers or service mesh. Correlating error logs with distributed traces is critical for rapid Root Cause Analysis (RCA).

Unexpected patterns in hardware utilization metrics that deviate from normal load profiles. These are leading indicators of impending failure.

• GPU Memory Exhaustion: Often caused by memory leaks in the KV Cache or excessively long sequences.
• High GPU/CPU Utilization with Low TPS: Indicates inefficient kernel execution or bottlenecks outside the model (e.g., data fetching).
• Network I/O Saturation: Can occur with high-volume Retrieval-Augmented Generation (RAG) systems querying vector databases. Monitoring these requires infrastructure-level telemetry from tools like Prometheus and node exporters.

Deviations in the qualitative aspects of model outputs, often detected by secondary classifiers or heuristics. These include:

• Sudden spikes in hallucination rates, detected by fact-checking systems or NLI models.
• Increases in toxic, biased, or unsafe content generation.
• Jailbreak or prompt injection successes that bypass safety fine-tuning.
• Regulatory compliance violations (e.g., generating PII). Detection relies on a Human-in-the-Loop (HITL) review pipeline and automated content moderation systems scoring every output.

Output drift refers to a statistical change over time in the distribution of an LLM's generated text outputs or their vector embeddings compared to an established baseline. This can signal:

• Unintended behavioral changes due to upstream data shifts.
• Performance degradation in tasks like classification or summarization.
• It is detected by comparing live output metrics (e.g., sentiment scores, response length, embedding centroids) against a golden dataset using statistical tests like Population Stability Index (PSI) or KL-divergence.

Concept drift occurs when the underlying relationship between the LLM's inputs and the desired, correct outputs changes in the real world, making previously learned patterns obsolete. In LLM contexts, this manifests as:

• A decline in accuracy for a fixed task (e.g., code generation for a new framework).
• Changing user intent or query patterns that the model hasn't adapted to.
• Unlike data drift (shifts in input distribution), concept drift specifically concerns the mapping function the model must learn, often requiring continuous model learning systems or retraining to address.

Statistical Process Control is a methodology for monitoring process behavior using statistical tools like control charts. In LLM ops, SPC is applied to time-series metrics (e.g., latency, token rate, error counts) to:

• Establish a baseline mean and expected variance (control limits).
• Automatically flag data points that fall outside control limits as potential anomalies.
• Distinguish between common-cause variation (normal noise) and special-cause variation (indicating a real issue), enabling proactive incident management before SLOs are breached.

Root Cause Analysis is the systematic investigative process triggered after an anomaly is confirmed. It aims to identify the fundamental causal factor(s) to prevent recurrence. For an LLM performance anomaly, RCA typically involves:

• Tracing the issue through the stack using distributed tracing.
• Examining correlated changes in deployment, traffic, or upstream data.
• Analyzing model inputs/outputs and infrastructure metrics (GPU utilization, memory).
• The output is a corrective action plan, which feeds into improving monitoring alerting and deployment safeguards like canary deployments.

A golden dataset is a curated, high-quality, and statistically representative set of input-output pairs that serves as a reference standard for evaluating LLM performance. It is critical for anomaly detection because it provides a stable baseline to compare against live outputs. Uses include:

• Running daily or weekly evaluations to detect output drift.
• Validating new model versions before deployment.
• Grounding cohort analysis when comparing different model or prompt versions.
• The dataset must be periodically reviewed to ensure it remains representative of production traffic and doesn't itself suffer from concept drift.

Cohort analysis segments users, requests, or model versions into groups for comparative evaluation. It transforms broad anomaly detection into targeted investigation by isolating where a deviation originates. Common cohorts in LLM monitoring include:

• User Segments: Enterprise vs. trial users, geographic regions.
• Request Types: Code generation vs. summarization, different prompt templates.
• Model/Deployment Versions: Comparing A/B test groups or shadow deployment outputs.
• By analyzing metrics like latency, error rate, or quality scores across cohorts, teams can pinpoint if an anomaly is systemic or isolated to a specific slice of traffic.