Redaction

Redaction performs permanent removal of sensitive spans, replacing them with fixed placeholder tokens (e.g., [REDACTED]) or zero-width characters. This is distinct from data masking, which preserves format and reversibility.

• Mechanism: The original byte sequence is deleted from the context string; no reversible encryption or tokenization is applied.
• Critical property: The generation model never observes the sensitive data, eliminating the risk of memorization or regurgitation.
• Use case: Removing Social Security Numbers, credit card details, or classified codewords before prompt injection.

Redaction engines rely on Named Entity Recognition models to identify sensitive spans before removal. These models scan for predefined entity types such as PERSON, CREDIT_CARD, PHONE_NUMBER, or custom regex patterns.

• Pattern matching: Regular expressions catch structured data like \d{3}-\d{2}-\d{4}.
• ML-based detection: Transformer models identify context-dependent entities like organization names or medical terms.
• False positive risk: Overly aggressive NER can redact non-sensitive text, degrading context quality for the LLM.

Redaction executes after the vector similarity search but before the prompt is assembled and sent to the LLM. This placement is critical for maintaining search recall while enforcing security.

• Pre-retrieval filtering restricts the search space; redaction sanitizes the results.
• Execution order: 1) Retrieve chunks, 2) Apply authorization filters, 3) Run NER + redaction, 4) Construct final prompt.
• Latency impact: Each redaction pass adds inference time; batching and GPU acceleration minimize overhead.

Redaction must balance security with semantic coherence. Removing a name or number can break anaphoric references and confuse the LLM, leading to degraded answer quality.

• Co-reference resolution: Advanced systems resolve pronouns before redaction to maintain sentence flow.
• Placeholder strategies: Using typed placeholders like [PERSON_1] instead of generic [REDACTED] helps the model track entities.
• Trade-off: Complete removal guarantees zero leakage but may reduce the factual grounding of the generated response.

Redaction is a key control for regulatory compliance frameworks including GDPR, HIPAA, and PCI-DSS. Every redaction event must be logged for auditability.

• Audit trail: Logs capture the original span type, redaction timestamp, and the identity of the requesting user.
• Right to erasure: Redaction supports data minimization principles by ensuring PII is never processed by the AI model.
• DLP alignment: Integrates with enterprise Data Loss Prevention systems to enforce consistent policies across retrieval and generation.

Understanding the distinctions between obfuscation techniques is critical for architecting secure RAG pipelines.

• Redaction: Permanent removal; original data is irretrievable from the prompt. Used for highest-sensitivity fields.
• Masking: Replaces data with format-preserving but fictitious values (e.g., ****1234). The LLM sees a realistic structure but not real data.
• Tokenization: Substitutes sensitive data with a reversible reference token. The original value can be recovered from a secure vault if needed.
• Selection heuristic: Use redaction when the LLM never needs the data; use masking when format matters; use tokenization when reversibility is required downstream.