Log Anomaly Pattern Extraction Prompt

This prompt is designed for SREs, on-call developers, and incident commanders who are staring down a firehose of unstructured log data during an active incident or post-incident review. The core job-to-be-done is to collapse thousands of raw log lines into a structured, deduplicated summary of recurring error signatures, clustered stack traces, and novel anomaly patterns. It does not replace your observability platform's metrics or tracing capabilities; instead, it acts as a force multiplier that turns a wall of text into a triage artifact you can actually reason about. Use it when you need to answer 'What is actually failing, and how many distinct failure modes are we dealing with?' without manually scanning logs.

The ideal user has access to a representative sample of logs from the incident window, typically exported from a logging backend like Splunk, Elasticsearch, Loki, or CloudWatch. The prompt requires a known baseline of 'normal' log patterns to detect novelty—this can be a description of expected log volume and message types, or a sample of logs from a stable period. Do not use this prompt when you have fewer than a few hundred log lines, as the statistical signal for anomaly detection collapses at low volume. Do not use it when you need real-time alerting with millisecond latency; this is an analytical prompt for investigation, not a streaming detection engine. Do not use it when logs contain primarily unstructured free-text narratives without repeating patterns, as the extraction logic depends on structural similarity clustering.

The output is a structured anomaly report containing deduplicated groups, each with an exemplar message, frequency count, first and last occurrence timestamps, and a novelty flag relative to the provided baseline. This artifact feeds directly into your incident timeline, root cause analysis document, or on-call handoff notes. Before relying on the output, validate that the extracted groups map to known failure domains in your system and that novel patterns are not simply deployment-related log format changes. The next step after extraction is to correlate these anomaly groups with metric spikes, deployment events, and alert timelines to form testable root cause hypotheses.

This prompt is designed to be pasted directly into your AI harness. It instructs the model to act as an SRE analyst, sifting through a raw log dump to identify recurring error signatures, novel patterns against a provided baseline, and stack trace clusters. The primary goal is to reduce alert fatigue and accelerate incident response by turning thousands of log lines into a handful of actionable anomaly groups. Before using this prompt, ensure you have a representative sample of recent logs and, ideally, a sample of 'normal' logs from a stable period to serve as the [BASELINE_LOG_SAMPLE]. This prompt is most effective when the logs are structured or semi-structured (e.g., JSON, syslog) and contain consistent fields like level, message, and timestamp.

textCopy
You are an SRE analyst specializing in log forensics. Your task is to analyze a high-volume log sample from an active incident and extract distinct anomaly patterns. You will compare the incident logs against a baseline of normal operations to identify novel error signatures, recurring stack traces, and unusual log frequency spikes.

### INPUTS
[INCIDENT_LOG_SAMPLE]
[BASELINE_LOG_SAMPLE]

### CONSTRAINTS
- Ignore known, low-severity noise patterns listed here: [KNOWN_NOISE_PATTERNS].
- Group similar log lines into deduplicated anomaly groups, even if they are not exact string matches. Use the core error message or stack trace signature as the grouping key.
- For each group, provide a frequency count, a single exemplar message, and a list of distinct host/pod/service identifiers involved.
- Explicitly flag any anomaly group that appears in the incident sample but is completely absent from the baseline.
- Do not invent root causes. Only report on the patterns present in the provided logs.

### OUTPUT_SCHEMA
Return a valid JSON object with a single key "anomaly_groups" containing an array of objects. Each object must have the following fields:
- "group_id": (string) A unique slug for the anomaly group.
- "signature": (string) The core error message or stack trace pattern that defines the group.
- "count": (integer) The number of log lines in this group.
- "exemplar": (string) A single, full log line that best represents the group.
- "is_novel": (boolean) True if this pattern is absent from the baseline.
- "affected_components": (array of strings) A list of unique hostnames, pod names, or service IDs.
- "first_seen": (string) ISO 8601 timestamp of the earliest log line in the group.
- "last_seen": (string) ISO 8601 timestamp of the latest log line in the group.

To adapt this template, replace the square-bracket placeholders with live data in your application code before sending the request to the model. The [INCIDENT_LOG_SAMPLE] should be a string containing the raw logs from the incident window. The [BASELINE_LOG_SAMPLE] should be a similar string from a stable period. The [KNOWN_NOISE_PATTERNS] placeholder allows you to inject a list of strings or regex patterns that your team has already triaged and deemed irrelevant, preventing the model from wasting tokens on them. If you don't have a baseline, remove the is_novel field from the schema and the related instruction. The output is strictly typed JSON, which makes it suitable for direct ingestion into a downstream dashboard or ticketing system. Always validate the JSON structure and check that count values are positive integers before displaying results to an on-call engineer.

Integrating this prompt into an incident response system requires treating it as a signal-processing step, not a standalone chatbot. The prompt expects a batch of raw log lines and a baseline profile, and it returns structured anomaly groups. In production, you'll wrap this in a harness that handles log ingestion, chunking for context windows, model invocation, output validation, and integration with your observability stack. The goal is to reduce mean time to detection by surfacing novel error signatures that keyword-based alerting would miss.

Start by building a pre-processing pipeline that samples high-severity log lines (ERROR, CRITICAL, FATAL) from your centralized logging system (e.g., Elasticsearch, Loki, Splunk) within the incident time window. Deduplicate exact-match lines before sending to the model to preserve context budget. Chunk the input into batches of 200-400 lines, each with the same baseline profile. For each chunk, call the prompt with a model that supports structured output (GPT-4o with response_format or Claude with tool use). Validate the response against the expected schema: each anomaly group must have a signature, exemplar_messages, frequency_count, and is_novel_against_baseline boolean. Reject and retry any response missing required fields or containing malformed JSON. Log every invocation—including the prompt version, model, latency, input line count, and output group count—to your observability platform for later trace analysis.

Implement a post-processing deduplication step that merges anomaly groups with overlapping exemplar messages across chunks, summing frequency counts and preserving the most representative exemplar. Then enrich each group with a link back to the source log query so on-call engineers can drill into raw evidence. For high-risk production incidents, route the final anomaly report to a human reviewer before broadcasting to the incident channel. Avoid using this prompt on low-severity or debug-level logs without filtering, as the noise will drown the signal and waste inference budget. The harness should also track baseline staleness: if the baseline profile is older than your deployment cadence, flag the output with a warning that novel patterns may be false positives from recent code changes rather than true anomalies.