Log Severity Classification and Triage Prompt Template

The raw log message to classify, including any timestamp, level prefix, or structured metadata present in the original log

2025-01-15T14:32:11.001Z ERROR [payment-service] Connection refused to database at db-primary.internal:5432 after 3 retries

Must be a non-empty string. Truncate to 4000 characters if longer. Do not pre-strip log level prefixes; the model needs them for context.

Identifier for the originating service, host, or application component that produced the log line

payment-service / ip-10-0-1-42.ec2.internal

Must be a non-empty string. Use a consistent naming convention across all logs to enable correlation. Null allowed if source is unknown.

Optional structured context about the system state when the log was emitted, such as deployment status, recent changes, or known incidents

Deployment v2.4.1 rolled out 5 minutes prior. Database connection pool size: 50. No active incident declared.

Null allowed. If provided, must be a string under 2000 characters. Avoid including raw PII or secrets. Validate that context is temporally relevant to the log timestamp.

Custom severity level definitions for the organization, overriding default levels if provided

CRITICAL: Complete service outage or data loss. ERROR: Functionality broken for users but service still partially available. WARN: Degraded but functional. INFO: Normal operation record. DEBUG: Diagnostic detail.

Null allowed. If provided, must be a valid JSON object mapping level names to descriptions. Validate that at least one level is defined. Default to standard syslog levels if omitted.

The expected JSON structure for the classification output, including field names and types

{"severity": "string", "confidence": "float between 0 and 1", "justification": "string", "requires_immediate_action": "boolean"}

Must be a valid JSON Schema or example object. Validate that required fields are present. The model will attempt to conform to this structure; schema mismatches are a common failure mode.

Few-shot examples of log lines with correct severity classifications and justifications, used to calibrate the model's judgment

[{"log": "Connection timeout after 30s", "severity": "ERROR", "justification": "Dependency failure blocking user requests"}, {"log": "Cache miss for key user:12345", "severity": "DEBUG", "justification": "Normal cache behavior, no user impact"}]

Null allowed. If provided, must be a valid JSON array of objects with log, severity, and justification fields. Include at least one ambiguous example to prevent overconfidence. Validate that example severities are consistent with [SEVERITY_DEFINITIONS].

Behavioral rules for the classifier, such as handling of ambiguous logs, confidence thresholds, or escalation triggers

If confidence is below 0.7, set severity to WARN and note ambiguity in justification. Never classify a log as CRITICAL without evidence of user-facing outage or data loss. Prefer ERROR over CRITICAL when impact is uncertain.

Null allowed. If provided, must be a string under 1000 characters. Constraints should be testable: each rule must map to a pass/fail condition in the evaluation harness. Avoid vague constraints like 'be careful'.

enum: DEBUG, INFO, WARN, ERROR, CRITICAL

Must be one of the five allowed enum values. Reject any other string.

number (0.0 to 1.0)

Must be a float between 0.0 and 1.0 inclusive. If confidence is below 0.7, route for human review.

string

Must be a non-empty string (min 10 chars). Must reference at least one specific element from [LOG_LINE].

string

Must be one of: 'none', 'degraded', 'blocking', 'data_loss', 'security'. Reject unknown values.

string or null

Must be a non-empty string if identifiable from [LOG_LINE], otherwise null. If null, confidence must be ≤ 0.8.

boolean

Must be true if severity is CRITICAL or impact_assessment is 'blocking', 'data_loss', or 'security'. Validate consistency.

string or null

If present, must be one of the allowed escalation paths from [ESCALATION_POLICY]. If null, no escalation is suggested.

Output matches the expected severity label (DEBUG, INFO, WARN, ERROR, CRITICAL) for each log line in the golden dataset with >= 95% accuracy.

Model outputs WARN for a known CRITICAL log line (e.g., OOMKill) or DEBUG for a known ERROR.

Run against a labeled golden dataset of 100 log lines covering all severity levels and boundary cases. Measure exact match accuracy.

The justification field references specific tokens or fields from the input log line (e.g., 'OutOfMemoryError', 'connection refused', 'status 200'). No hallucinated log content.

Justification mentions an error code or service name not present in the input log line, or provides a generic rationale with no log evidence.

Parse the justification field and check that all quoted strings or error codes exist in the input log line. Flag any justification with zero token overlap.

For ambiguous log lines (e.g., 'Operation timed out after 30 seconds'), the model outputs a severity no higher than WARN and includes an uncertainty note in the justification.

Model classifies an ambiguous timeout as CRITICAL without acknowledging missing context (e.g., no user impact data, no retry status).

Test with a curated set of 20 ambiguous log lines where severity depends on external context. Check that output severity is WARN or lower and justification contains uncertainty language.

Output is valid JSON matching the defined schema: { severity: string, justification: string, confidence: number }. Confidence is a float between 0.0 and 1.0.

Output is missing a required field, confidence is outside 0.0-1.0 range, or severity is not one of the five allowed enum values.

Validate output with a JSON schema validator. Check enum membership for severity and numeric range for confidence. Reject any output that fails parsing.

Confidence score correlates with classification difficulty: high confidence (>= 0.9) for unambiguous log lines, lower confidence (< 0.8) for ambiguous or context-dependent lines.

Model assigns 0.99 confidence to an ambiguous log line or 0.5 confidence to a clear CRITICAL error like 'disk full' on a database server.

Compute the mean confidence for the ambiguous test set and the unambiguous test set. The ambiguous mean should be at least 0.15 lower than the unambiguous mean.

Log lines indicating successful operations with no diagnostic detail (e.g., 'Request completed in 45ms') are classified as INFO, not DEBUG.

Model classifies a standard access log line or successful health check as DEBUG, causing it to be filtered out of operational dashboards.

Test with 10 INFO-level log lines that contain timing or status but no error context. Verify none are classified as DEBUG.

A service-specific error that does not cause user-facing downtime or data loss (e.g., 'Failed to send email notification') is classified as ERROR, not CRITICAL.

Model classifies a non-blocking background task failure as CRITICAL, triggering a page to on-call staff unnecessarily.

Test with 10 ERROR-level log lines where the failure is scoped to a non-critical subsystem. Verify none are classified as CRITICAL.

For empty string, whitespace-only, or non-log input (e.g., a JSON object with no log message), the model returns a valid JSON output with severity 'INFO', confidence 0.0, and a justification stating the input was unclassifiable.

Model throws an error, returns invalid JSON, or hallucinates a severity based on no evidence.

Send empty string, ' ', and a random JSON object as inputs. Verify output schema compliance, confidence of 0.0, and a justification that indicates unclassifiable input.