Batch Processing PII Redaction Pipeline Prompt Template

This prompt is designed for data engineering teams who need to sanitize large volumes of unstructured model outputs as part of an ETL pipeline. It is not a real-time, per-request guard. Use it when you have a batch of records, a defined PII taxonomy, and a requirement for a machine-readable audit trail. The prompt instructs the model to act as a deterministic redaction engine, returning a structured JSON payload with redacted text, a change log, and per-record processing status. This playbook assumes you already have a batch processing infrastructure (Spark, Kafka Streams, AWS Batch) and need a reliable, stateless redaction step that can handle partial failures without aborting the entire job.

The ideal user is a data platform engineer or backend developer integrating an LLM call into a nightly batch job or a streaming micro-batch window. You must provide a clear [PII_TAXONOMY] (e.g., names, emails, SSNs, IP addresses) and a [REDACTION_STRATEGY] (e.g., mask, replace with entity type, or remove). The prompt works best when each input record is a self-contained text blob under a few thousand tokens. Do not use this for real-time chat filtering, streaming token-by-token redaction, or scenarios where latency must be under 200ms. For those cases, use a regex-based or NER-model microservice instead.

Before wiring this into production, validate the prompt against a golden dataset of known PII and non-PII edge cases. Common failure modes include redacting test data (e.g., sample credit card numbers in documentation), missing PII in nested JSON or code blocks, and inconsistent redaction of multi-language names. Always log the change_log field from the output to your audit system. If a record's processing_status is partial or failed, route it to a dead-letter queue for human review rather than silently passing redacted or unredacted data downstream.

This prompt template is designed to be dropped into a batch processing pipeline where a JSON array of text records must be sanitized before entering logs, databases, or downstream systems. It instructs the model to act as a deterministic redaction engine—no conversation, no commentary, just a structured JSON response containing the redacted records and a detailed audit trail. The template uses square-bracket placeholders for all variable inputs so you can wire it directly into your application code.

textCopy
Act as a batch PII redaction engine. Your task is to process a JSON array of text records, detect and redact specified PII categories, and return a structured JSON object containing the sanitized records and a detailed audit trail. Do not include any conversational text outside the JSON output.

## INPUT
[INPUT]

## PII CATEGORIES TO REDACT
[PII_CATEGORIES]

## REDACTION STRATEGY
[REDACTION_STRATEGY]

## OUTPUT SCHEMA
Return a JSON object with this exact structure:
{
  "records": [
    {
      "index": <original array index>,
      "original_text": "<original text>",
      "sanitized_text": "<text with PII replaced>",
      "redactions": [
        {
          "category": "<PII category>",
          "original_value": "<detected value>",
          "replacement": "<redacted form>",
          "start_char": <integer>,
          "end_char": <integer>,
          "confidence": <float 0.0-1.0>
        }
      ]
    }
  ],
  "summary": {
    "total_records_processed": <integer>,
    "total_redactions": <integer>,
    "categories_detected": ["<category>", ...]
  }
}

## CONSTRAINTS
[CONSTRAINTS]

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, replace each placeholder with concrete values. [INPUT] should be a JSON array of text records. [PII_CATEGORIES] should list the specific categories to detect—such as EMAIL, PHONE, SSN, CREDIT_CARD, PERSON_NAME, ADDRESS—and can include definitions for ambiguous categories. [REDACTION_STRATEGY] defines how to replace detected PII: use MASK for partial masking (e.g., j***@domain.com), REPLACE_WITH_CATEGORY for category tags (e.g., [EMAIL]), or REMOVE for complete deletion. [CONSTRAINTS] should specify rules like "do not redact test data patterns" or "preserve JSON structure in text fields." [EXAMPLES] should include 2-3 few-shot examples showing both obvious PII and tricky edge cases like sample credit card numbers or placeholder emails. [RISK_LEVEL] should be set to HIGH when processing production data, which means you must add human review for low-confidence detections and log every redaction decision.

Before deploying, validate the output against the schema with a JSON Schema validator. Run the prompt against a golden dataset containing known PII and measure recall (did you catch all PII?) and precision (did you redact benign strings?). Common failure modes include: the model redacting test data or sample values, missing PII in nested JSON strings, inconsistent masking formats across records, and hallucinating redactions for text that contains no PII. For high-risk pipelines, implement a two-pass approach: first pass detects and redacts, second pass verifies no PII remains in the sanitized output. Always log the full audit trail—not just the sanitized text—so compliance reviewers can trace every redaction decision back to the original value.

This prompt is designed to be called once per record in a batch processing job, not as a single monolithic call with all records. Each invocation receives one [INPUT_TEXT] and returns a structured JSON object containing the redacted text and an audit trail. The application layer is responsible for batching, concurrency, and assembling results. Use a queue-based worker pattern (e.g., Celery, SQS, Pub/Sub) to fan out records, with each worker calling the LLM API with this prompt template. Set a per-record timeout appropriate for your text length—typically 10–30 seconds for documents under 4,000 tokens. For longer inputs, chunk the text before calling the prompt and reassemble redacted chunks in order, ensuring chunk boundaries do not split potential PII tokens.

Validation and retry logic is mandatory. After receiving the model response, validate that the JSON structure matches the expected [OUTPUT_SCHEMA] exactly—check for the presence of redacted_text, redactions array, and required fields within each redaction object (original, redacted, category, confidence). If validation fails, retry up to [MAX_RETRIES] times with the same input, appending the validation error to the [CONSTRAINTS] section so the model can self-correct. If all retries are exhausted, route the record to a dead-letter queue for human review. For high-throughput pipelines, implement a circuit breaker that pauses processing if the failure rate exceeds a configurable threshold (e.g., 5% over a 60-second window). Log every invocation with a correlation ID, the model version, prompt version, latency, token usage, and validation outcome. Store raw model responses for 30 days to support debugging and compliance audits.

Model choice matters for throughput and cost. For batch PII redaction, prefer models with strong instruction-following and JSON mode support (e.g., GPT-4o, Claude 3.5 Sonnet, or fine-tuned open-weight models like Llama 3.1 70B). Use structured output features or JSON mode when available to reduce parsing failures. If processing millions of records, consider a tiered approach: use a smaller, faster model for initial detection and a larger model only for ambiguous cases flagged by confidence scoring. Implement prompt caching for the system instructions portion of the template to reduce latency and cost on repeated calls. For regulated environments, ensure the model endpoint is deployed within your VPC or on-premises boundary—never send raw PII to external APIs unless the data has already been pre-redacted or the provider has a BAA/DPA in place.

The audit trail is a product requirement, not optional metadata. Each redaction record must include the original text span, the replacement, the PII category, and a confidence score. Store this alongside the redacted output in your downstream systems. For compliance workflows, generate a redaction certificate per batch that includes the pipeline run ID, timestamp, model version, prompt version, total records processed, total redactions applied, and a summary by category. This certificate serves as evidence for auditors that a systematic, automated process was applied. Do not discard