Streaming Event Triage Prompt for Real-Time Pipelines

This prompt is designed for MLOps and data platform engineers who need a deterministic, low-latency classification step inside a running stream processor. The core job-to-be-done is converting a raw, potentially malformed event payload into a structured triage decision that downstream queues, workers, and dead-letter handlers can act on without ambiguity. It is not a general-purpose classification tool. It assumes you already have a streaming infrastructure layer—such as Kafka, Kinesis, or Pulsar—and that your primary challenge is making reliable routing decisions under backpressure, schema drift, and partial failure conditions.

Use this prompt when your pipeline must produce a decision that includes event type, priority, target workflow, and a confidence score within a bounded latency window. It is appropriate when you need to handle schema evolution gracefully, meaning the prompt must tolerate missing fields, new fields, and type mismatches without throwing errors that drop the event. It is also appropriate when you need to recover from transient downstream failures by routing events to retry topics or dead-letter queues with enough context for later remediation. Do not use this prompt for exploratory data analysis, batch ETL classification that can tolerate high latency, or workflows where the classification logic requires deep semantic understanding of unstructured natural language. This prompt works best on semi-structured event payloads—JSON, Avro, or Protobuf—where the signal is in field values, error codes, and structural patterns rather than long-form prose.

Before using this prompt, ensure you have defined your event taxonomy, priority tiers, and target workflow topology. The prompt relies on these definitions being supplied as part of the [CONTEXT] or [CONSTRAINTS] blocks. Without a clear taxonomy, the model will produce inconsistent event type labels. Without explicit priority rules, the model will default to generic heuristics that may not match your SLA requirements. Finally, do not use this prompt in isolation for high-risk domains—such as healthcare or financial transactions—without adding human review for low-confidence classifications and an audit trail that captures the raw event, the prompt version, and the triage decision for every record.

This template is designed to be pasted directly into your stream processor. It classifies each event in a real-time pipeline, assigning a type, priority, and target workflow while respecting strict latency constraints. The prompt uses square-bracket placeholders that you must replace with your specific taxonomy, downstream systems, and operational rules before deployment.

textCopy
SYSTEM: You are a real-time event triage classifier for a high-throughput streaming pipeline. Your task is to classify each event by type, priority, and target workflow. You must respond within [LATENCY_BUDGET_MS] milliseconds. Do not perform any actions beyond classification.

INPUT EVENT:
[EVENT_PAYLOAD]

CLASSIFICATION TAXONOMY:
- Event Types: [EVENT_TYPE_1], [EVENT_TYPE_2], [EVENT_TYPE_3]
- Priority Levels: [PRIORITY_CRITICAL], [PRIORITY_HIGH], [PRIORITY_MEDIUM], [PRIORITY_LOW]
- Target Workflows: [WORKFLOW_ALERTING], [WORKFLOW_STORAGE], [WORKFLOW_ANALYTICS], [WORKFLOW_DEAD_LETTER]

CONSTRAINTS:
- If the event payload is malformed or missing required fields, classify as type=[MALFORMED_EVENT_TYPE] and route to [WORKFLOW_DEAD_LETTER].
- If the event contains fields matching [PII_PATTERNS], set priority to [PRIORITY_CRITICAL] and route to [WORKFLOW_PII_HANDLING].
- If confidence in classification is below [CONFIDENCE_THRESHOLD], set priority to [PRIORITY_HIGH] and route to [WORKFLOW_HUMAN_REVIEW].
- Do not invent event types, priorities, or workflows not listed in the taxonomy.

OUTPUT SCHEMA:
{
  "event_id": "string (from input payload)",
  "classification": {
    "event_type": "string (from taxonomy)",
    "priority": "string (from taxonomy)",
    "target_workflow": "string (from taxonomy)",
    "confidence": 0.0-1.0,
    "rationale": "string (brief explanation of classification decision)"
  },
  "flags": {
    "is_malformed": boolean,
    "contains_pii": boolean,
    "requires_human_review": boolean
  },
  "processing_hints": {
    "retry_eligible": boolean,
    "idempotency_key": "string (extracted or generated)",
    "schema_version": "string (from input payload or detected)"
  }
}

EXAMPLES:
[FEW_SHOT_EXAMPLE_1]
[FEW_SHOT_EXAMPLE_2]
[FEW_SHOT_EXAMPLE_3]

Respond ONLY with valid JSON matching the output schema. No additional text.

To adapt this template, replace each square-bracket placeholder with your production values. The [EVENT_PAYLOAD] placeholder should receive the raw event from your stream. The [LATENCY_BUDGET_MS] should be set below your pipeline's end-to-end latency SLA. The [PII_PATTERNS] should match your data classification rules. The [FEW_SHOT_EXAMPLE_*] placeholders should contain representative events with correct classifications to anchor model behavior. After adaptation, validate the prompt against your golden test set before deploying to production. Monitor classification accuracy, latency percentiles, and dead-letter queue volume continuously after deployment.

This prompt is designed to be a single, stateless transformation step within a larger stream processing topology. It expects a batch of raw events as a JSON array and returns a structured triage decision for each. The implementation harness must treat the model as an external service with latency, failure modes, and throughput limits. The primary integration points are: a pre-processing step to assemble the batch and inject the prompt, a post-processing step to validate the model's output against the expected [OUTPUT_SCHEMA], and a routing step that dispatches each event to the correct downstream topic, queue, or workflow based on the target_workflow and priority fields.

To wire this into a real-time pipeline, wrap the model call in a circuit breaker. If the model endpoint returns 5xx errors or exceeds a latency SLA (e.g., p99 > 2 seconds), the circuit should open and route events to a fallback path. A common fallback is a rules-based classifier that uses regex and keyword matching to assign a default target_workflow of manual_review and a priority of high. This prevents backpressure from cascading into upstream systems. For logging, capture the event_id, the model's raw output, the validated output, and any validation errors in a structured log for every request. This trace is essential for debugging schema evolution issues and model drift.

Schema evolution is a primary failure mode in streaming systems. When upstream producers add new fields or change types, the model may hallucinate or produce malformed output. The harness must validate the model's response against a strict JSON schema before routing. If validation fails, do not retry immediately with the same prompt. Instead, route the failed events to a dead-letter queue (DLQ) and trigger the 'Schema Validation Triage Prompt' to classify the failure. For transient model errors, implement a retry policy with exponential backoff and jitter, but cap retries at 2 attempts to avoid blocking the stream. A final safeguard is a human_review sink for any event where the model's confidence_score falls below a configurable threshold, ensuring that low-certainty triage decisions do not silently cause downstream harm.