Contextual PII Disambiguation Prompt Template

This prompt is for security engineers and compliance platform teams who are running post-generation PII redaction pipelines and are seeing too many false positives. The job-to-be-done is not primary detection—it is disambiguation. You use this prompt when a regex or pattern-based scanner has already flagged a string as potential PII, but the context is ambiguous: a nine-digit number could be a Social Security Number or a test fixture ID, and admin@example.com could be a real email or documentation placeholder. The prompt's job is to read the surrounding context, decide whether the flagged string is actual PII, and return a structured verdict with a confidence score and a recommended action.

The ideal user is a backend or platform engineer integrating this prompt into a repair and review pipeline. You already have a PII detection step in place. This prompt sits after detection and before redaction or logging. It receives the original model output, a list of flagged spans with their detected PII categories, and any available document or conversation context. It returns a decision for each span: redact, suppress, or release, along with a confidence score between 0.0 and 1.0. The prompt is designed to be called programmatically, not used in a one-off chat interface. You should wire it into a retry loop with a human review queue for low-confidence decisions.

Do not use this prompt as your primary PII detector. It is too expensive and slow for bulk scanning. Do not use it for real-time streaming redaction where latency is measured in milliseconds. Do not use it when you lack surrounding context—sending a single flagged string in isolation will produce unreliable results. And do not use it for regulated data categories that require certified de-identification methods (e.g., HIPAA Safe Harbor) without adding a human review step for every decision. This prompt reduces false positives; it does not eliminate the need for a defense-in-depth strategy. After implementing, run your eval suite against known PII test sets and false-positive benchmarks to measure the improvement in precision without degrading recall.

This prompt template is designed to be the core instruction set for a post-generation PII disambiguation model. It takes a string that has already been flagged by a high-recall PII scanner and asks the model to determine if the match is a genuine piece of sensitive data or a benign false positive (e.g., a test SSN, a sample credit card number, or a placeholder email). The output is a structured JSON object that includes a final classification, a confidence score, and a clear rationale, enabling automated filtering or routing to a human review queue.

textCopy
You are a security-focused data classifier. Your task is to analyze a text segment that matched a PII detection pattern and determine if it is genuine sensitive data or a benign false positive.

## Input
- [FLAGGED_TEXT]: The specific substring that triggered the PII scanner.
- [SURROUNDING_CONTEXT]: The text surrounding the flagged string (at least 100 characters before and after).
- [DETECTED_PII_TYPE]: The type of PII the scanner identified (e.g., 'SSN', 'EMAIL', 'CREDIT_CARD', 'PHONE_NUMBER').
- [SOURCE_DOCUMENT_TYPE]: The type of document the text came from (e.g., 'code_comment', 'user_generated_content', 'log_file', 'legal_contract', 'test_data').

## Task
1. Analyze [FLAGGED_TEXT] within its [SURROUNDING_CONTEXT] and [SOURCE_DOCUMENT_TYPE].
2. Determine if it is a genuine instance of [DETECTED_PII_TYPE] or a benign look-alike (e.g., test data, sample, placeholder, or a string that coincidentally matches a pattern).
3. Provide a final classification, a confidence score, and a concise rationale.

## Output Schema
You must respond with a single, valid JSON object conforming to this schema:
{
  "classification": "GENUINE_PII" | "BENIGN_FALSE_POSITIVE",
  "confidence_score": 0.0-1.0,
  "rationale": "A concise explanation of the evidence that led to the classification.",
  "recommended_action": "REDACT" | "ALLOW" | "HUMAN_REVIEW"
}

## Constraints
- If confidence_score is between 0.4 and 0.8, recommended_action MUST be "HUMAN_REVIEW".
- A score above 0.8 should classify as GENUINE_PII and recommend "REDACT".
- A score below 0.4 should classify as BENIGN_FALSE_POSITIVE and recommend "ALLOW".
- Look for contextual clues like "test", "sample", "example", "placeholder", "XXXX", "000", or "fake" in the [SURROUNDING_CONTEXT].
- A [SOURCE_DOCUMENT_TYPE] of 'test_data' or 'code_comment' strongly suggests a false positive.
- Do not hallucinate. If you are unsure, set confidence_score between 0.4 and 0.8 and recommend "HUMAN_REVIEW".

To adapt this template for your application, replace the square-bracket placeholders with data from your PII scanning pipeline. The [FLAGGED_TEXT] and [SURROUNDING_CONTEXT] should be extracted directly from the model's output. The [DETECTED_PII_TYPE] should map to your scanner's categories. The [SOURCE_DOCUMENT_TYPE] is a critical signal for disambiguation and should be populated from your application's metadata. The output schema is designed to be machine-readable, allowing you to programmatically route items with a recommended_action of HUMAN_REVIEW to a dedicated queue. Before deploying, test this prompt against a golden dataset of known true positives and false positives to calibrate the confidence thresholds for your specific use case.

The contextual PII disambiguation prompt is not a standalone safety net—it is a decision engine that must be embedded in a broader application harness. The harness is responsible for receiving the raw model output, extracting candidate PII matches using a deterministic regex scanner, batching ambiguous matches into the prompt, and then applying the model's disambiguation decisions back to the original text. This separation of concerns is critical: the regex layer handles high-recall detection at speed, while the LLM handles the high-precision judgment calls that pattern matching alone cannot resolve. The harness should never send the full original output to the disambiguation prompt; instead, it should extract only the surrounding context window for each ambiguous match to minimize token usage and prevent the model from hallucinating new PII into unrelated text.

The implementation loop follows a clear sequence. First, run a deterministic PII scanner (such as Presidio, Microsoft's PII detection library, or a custom regex suite) against the model output to produce a list of candidate matches with their character offsets and match types. Second, classify each match as definite_true_positive, definite_false_positive, or ambiguous based on local heuristics—for example, a 9-digit number in a known test SSN range is a definite false positive, while a 9-digit number in an unknown context is ambiguous. Third, batch the ambiguous matches into the disambiguation prompt, including 100–200 characters of surrounding context per match. Fourth, parse the model's JSON response and apply the decisions: redact confirmed PII, leave benign strings intact, and flag borderline cases for human review. Fifth, log every disambiguation decision with the match text, context, model confidence score, and final action for auditability.

Validation and retry logic must be built into the harness. The prompt template expects a specific JSON output schema with match_id, is_pii, confidence, reasoning, and requires_review fields. The harness should validate this schema strictly after every model response. If the model returns malformed JSON, missing required fields, or confidence scores outside the 0.0–1.0 range, the harness should retry with the same input up to two additional times, appending the validation error to the prompt as a correction instruction. If retries are exhausted, all ambiguous matches in that batch should be escalated to human review rather than silently passed through. For high-throughput systems, implement a dead-letter queue for failed batches and a circuit breaker that halts processing if the model's schema compliance rate drops below 95% over a rolling window of 100 requests.

Model choice and latency budgeting are critical design decisions. The disambiguation task requires strong instruction-following and structured output capabilities, making GPT-4o, Claude 3.5 Sonnet, or equivalent models the appropriate tier. Do not use smaller or faster models for this task unless you have validated their schema compliance and false-positive rates against a labeled test set of at least 500 ambiguous PII examples. For latency-sensitive applications, batch ambiguous matches into groups of 10–20 per prompt call to balance throughput against per-token costs. If real-time processing is required, consider a two-tier architecture: a fast local model (such as a fine-tuned Llama 3 8B) for initial disambiguation with high recall, and the cloud model as a secondary reviewer for matches the local model flags as uncertain. Always measure end-to-end latency from regex detection through redaction application, and set a timeout of 5–10 seconds per disambiguation batch with fallback to conservative redaction (treat all ambiguous matches as PII) if the timeout is exceeded.

Human review integration is the final safety layer. The prompt's requires_review boolean and confidence score should directly control routing in the harness. Matches with confidence below 0.85 or requires_review set to true must be queued for human review before any downstream system consumes the output. The review interface should display the original text with the ambiguous match highlighted, the model's reasoning, and accept/reject/override controls. Track review decisions over time to measure the model's precision and recall against human judgment, and use this data to tune the confidence threshold and update the few-shot examples in the prompt. Never allow low-confidence disambiguation decisions to flow into logs, databases, or user-facing surfaces without review—the cost of a false negative PII leak far outweighs the operational cost of a human review queue.