Poison Pill Detection Prompt for Corrupt Batch Records

This prompt is for data platform engineers who need to protect batch processing pipelines from corrupt records that cause cascading downstream failures. It identifies poison pill records before they enter transformation, aggregation, or loading stages. Use this prompt when you have a stream of records arriving in batches, and a single malformed or malicious record can crash a Spark job, corrupt a partition, or trigger an infinite retry loop. The prompt classifies corruption type, assesses propagation risk, and produces a quarantine decision with a corruption signature.

The prompt assumes records are already parsed into a structured representation and that you have a defined schema to validate against. It works best when you can provide the record payload, the expected schema, and any known failure patterns from previous pipeline incidents. The output includes a corruption classification, a propagation risk score, and a quarantine recommendation that your orchestration layer can act on. For high-risk pipelines where data loss is unacceptable, always route quarantine decisions through a human review step before permanent deletion.

Do not use this prompt for real-time streaming event triage where latency is measured in milliseconds; use the Streaming Event Triage Prompt instead. It is also not suitable for schema validation alone—this prompt focuses on corruption that survives basic parsing, such as logically valid but semantically dangerous records. Avoid using it as a replacement for input sanitization at the ingestion boundary; it should sit after parsing but before transformation stages.

This prompt template is designed to be dropped directly into your batch processing harness. It instructs the model to act as a data quality gate, scanning a batch of records for structural corruption, schema violations, and semantic anomalies that will cause deterministic failures in downstream ETL, analytics, or model inference steps. The prompt forces a structured JSON output that your pipeline can parse to quarantine bad records, log corruption signatures, and route clean records forward without manual inspection.

Below is the copy-ready template. Every placeholder in square brackets must be replaced by your application at runtime. The [INPUT] block expects a serialized batch of records (JSON Lines or a JSON array). The [OUTPUT_SCHEMA] block defines the exact JSON structure the model must return, including required fields like corruption_type, propagation_risk, and quarantine_decision. The [CONSTRAINTS] block lets you enforce domain-specific rules, such as field length limits, required columns, or expected enum values. The [EXAMPLES] block should contain 2-3 few-shot examples of corrupt records with their correct classifications to anchor model behavior. The [RISK_LEVEL] block controls the model's sensitivity—use high for regulated data where false negatives are unacceptable, or low for non-critical pipelines where throughput matters more than exhaustive detection.

textCopy
You are a data quality gate for a batch processing pipeline. Your task is to analyze the following batch records and identify any poison pill records that will cause downstream processing failures.

For each corrupt record you find, you must:
1. Classify the corruption type from the allowed taxonomy.
2. Assess the propagation risk (how far the corruption will spread if not quarantined).
3. Produce a quarantine decision with a unique corruption signature for deduplication.
4. Return only valid JSON matching the output schema exactly.

[INPUT]
[EXAMPLES]
[OUTPUT_SCHEMA]
[CONSTRAINTS]
[RISK_LEVEL]

Return only the JSON object. Do not include explanations, markdown fences, or additional text.

After copying this template, you must adapt the placeholder blocks to your specific pipeline. The [OUTPUT_SCHEMA] is the most critical—define a JSON schema that your downstream quarantine handler can parse without ambiguity. Include a corruption_signature field that is deterministic enough to deduplicate repeated poison pills across retry attempts. The [CONSTRAINTS] block should encode your schema contracts: required fields, allowed types, value ranges, and referential integrity rules. Without these constraints, the model will miss corruption that is obvious to your pipeline but invisible to a general-purpose LLM. Validate the model's output against your schema before acting on quarantine decisions—malformed JSON from the model should trigger a retry or fallback to a safe default (quarantine the entire batch). For high-risk pipelines, route all quarantine decisions through a human review queue before permanent deletion.

Integrating the Poison Pill Detection Prompt into a production batch pipeline requires treating it as a deterministic gate, not an advisory tool. The prompt should be invoked as a pre-processing step before any record reaches a transformation, enrichment, or loading stage. In a typical Apache Spark, Apache Beam, or Kafka Streams job, each record or micro-batch is passed through a map or filter operation that calls the LLM endpoint. The model's response—containing the corruption_type, propagation_risk, and quarantine_decision—must be parsed and enforced immediately. A record classified as QUARANTINE should be written to a dead-letter queue (DLQ) or a quarantine storage path (e.g., s3://bucket/quarantine/YYYY-MM-DD/) and never reach the main processing sink. This prevents a single malformed record from crashing an entire Spark partition or causing a cascading write failure in a data warehouse.

The implementation must include a strict validation layer between the model's output and the routing decision. Parse the JSON response and validate that corruption_type is one of the allowed enum values (e.g., SCHEMA_VIOLATION, ENCODING_ERROR, TRUNCATED_RECORD, INJECTION_PATTERN, LOGICAL_CORRUPTION). If the output fails to parse or contains an unknown type, default to QUARANTINE with a corruption_type of UNCLASSIFIED to avoid passing potentially dangerous data. Implement a lightweight retry policy for transient model errors (HTTP 429, 503) with exponential backoff, but cap retries at 2-3 attempts to maintain pipeline throughput. For high-throughput streams, consider batching records into groups of 10-50 and sending them in a single prompt call with a structured output schema that returns an array of per-record classifications. This reduces API overhead while maintaining per-record traceability.

Observability is critical. Log every classification decision with the record's unique identifier, the model's confidence score, the corruption signature, and the routing outcome. Emit metrics on quarantine rate, classification latency (p50, p99), and the distribution of corruption types. Set alerts on sudden spikes in quarantine rate, which may indicate an upstream data producer failure or a prompt drift issue. Regularly sample quarantined records and run them through a human review process to measure the false positive rate—incorrectly quarantining valid records causes data loss that downstream consumers will notice. If the false positive rate exceeds a defined threshold (e.g., 2%), tune the prompt's instructions or adjust the confidence threshold for automatic quarantine. Never silently drop quarantined records; always retain them with metadata for forensic analysis and potential reprocessing after the root cause is fixed.