Output PII Leakage Detection Prompt for Model Responses

This prompt is designed for AI safety and privacy engineers who need to inspect model-generated outputs for personally identifiable information (PII), secrets, or other sensitive data that may have been memorized from training data or hallucinated during generation. It acts as a post-generation guardrail, producing a structured alert that can be routed to a human review queue before the output is returned to the user. Use this when you cannot guarantee that your model has never been exposed to sensitive data, or when you operate in a regulated environment where output inspection is a compliance requirement. This is not a pre-processing redaction tool; it inspects what the model already produced.

The ideal user is an engineer integrating this prompt into a production AI pipeline, specifically within a streaming response handler or a post-inference validation layer. The required context includes the raw model output string, a defined risk level that controls detection sensitivity, and a target output schema for the alert. You should not use this prompt as a real-time user-facing filter because it adds latency and requires a human-in-the-loop decision step. It is also inappropriate for inspecting outputs that are already guaranteed to be PII-free by a pre-processing redaction layer, as it would create redundant processing and false positives. Instead, deploy it as a safety net for high-risk use cases like customer-facing chatbots, clinical note summarization, or code generation where secrets might leak.

Before implementing, define your human review queue and service-level objectives (SLOs) for alert response. A common failure mode is alert fatigue, where the prompt flags too many low-risk entities and reviewers start ignoring alerts. Mitigate this by tuning the [RISK_LEVEL] parameter and by combining the prompt with a regex-based pre-filter for high-confidence patterns like credit card numbers. The next step is to wire this prompt into your application harness, ensuring that any output flagged with a block: true result is never returned to the user until a human clears it.

This prompt template is designed to be integrated into a post-processing step within your inference pipeline. Its sole job is to act as a safety scanner, analyzing a model's generated text for any personally identifiable information (PII), secrets, or other sensitive data that may have been memorized or hallucinated. It does not modify or redact the original text; it only produces a structured, machine-readable alert if a finding is detected. This separation of concerns—generation vs. detection—is critical for maintaining an auditable log and ensuring that redaction logic doesn't interfere with the primary task's output quality.

textCopy
Analyze the following model-generated text for any personally identifiable information (PII), secrets, or sensitive data. Your task is to produce a structured alert if any such data is detected. Do not modify the original text. Do not redact. Only report findings.

### PII Categories to Scan For:
[DETECTION_CATEGORIES]

### Sensitivity Threshold:
[SENSITIVITY_THRESHOLD]

### Model Output to Scan:

[MODEL_OUTPUT_TEXT]

codeCopy

### Output Format:
Return a single JSON object with the following schema. If no PII is detected, return an empty `findings` array and set `alert_triggered` to false.

```json
{
  "alert_triggered": boolean,
  "findings": [
    {
      "category": string,
      "type": string,
      "value_fragment": string,
      "char_offset_start": integer,
      "char_offset_end": integer,
      "confidence": float,
      "recommended_action": string
    }
  ],
  "summary": string
}

To adapt this template, you must define the [DETECTION_CATEGORIES] and [SENSITIVITY_THRESHOLD] placeholders based on your specific compliance requirements. For example, [DETECTION_CATEGORIES] could be a comma-separated list like "email addresses, phone numbers, social security numbers, API keys, credit card numbers". The [SENSITIVITY_THRESHOLD] is a natural language instruction that calibrates the scanner's aggressiveness, such as "Flag only high-confidence, well-formed PII. Ignore common dictionary words that look like names." or "Flag any potential PII, even if fragmented or low confidence.". The [MODEL_OUTPUT_TEXT] placeholder should be populated with the exact string from your upstream model's completion. The output JSON schema is the contract your application code must parse to decide whether to escalate for human review, log an incident, or trigger a redaction workflow.

Wire this prompt as a post-processing step in your inference pipeline, positioned after the primary model generates its response but before that response is returned to the user. The detection prompt receives the raw model output as its [MODEL_OUTPUT] input and returns a structured JSON object containing an alert_triggered boolean and a list of detected entities. For standard request-response flows, call the detection prompt synchronously after the primary inference completes. For streaming responses, buffer the full output before scanning, or implement a sliding window scan that processes chunks as they arrive and aggregates findings—this is critical because PII can span chunk boundaries in token-by-token streaming.

The alert_triggered boolean must gate the response path. When alert_triggered is true, do not return the original model output to the user. Instead, queue both the original output and the scan result JSON to a human review queue—this should be a durable, asynchronous work queue (SQS, Pub/Sub, Redis streams) that your operations team monitors. Return a safe fallback message to the user such as: "I wasn't able to generate a safe response. Our team has been notified and will review this shortly." When alert_triggered is false, return the original output normally. Log every scan result, including negative scans, with the inference trace ID, timestamp, model version, and scan outcome for full auditability. These logs are essential for compliance reporting and for debugging false negatives later.

Use a strict JSON validator on the detection prompt's response before trusting the alert_triggered field. The expected schema is: {"alert_triggered": boolean, "detected_entities": [{"type": string, "value": string, "confidence": float}]}. If JSON parsing fails, the model returns unexpected keys, or the schema is violated, treat this as a potential detection failure and escalate the original output to the review queue with a scan_error flag. Never default to alert_triggered: false on parse failure—a broken scanner is a silent failure mode that can leak PII into production. Implement a circuit breaker that tracks scan failure rate and halts responses if the failure rate exceeds a threshold (e.g., 5% over a rolling window).

This prompt is not a replacement for deterministic regex-based secret scanning. Run both in parallel and union the results for defense in depth. Your regex scanner should cover known patterns (AWS keys, credit card numbers, SSNs) with high precision, while the LLM-based scanner catches contextual PII that regex misses (names in prose, addresses in narratives, phone numbers in unconventional formats). If either scanner triggers, escalate. For model selection, prefer fast, cost-efficient models for this scanning task—a smaller model like Claude Haiku or GPT-4o-mini is usually sufficient and keeps scanning latency low. Set a timeout on the scan call (e.g., 2 seconds); if it expires, treat it as a scan failure and escalate. Finally, version your detection prompt alongside your primary prompt in your prompt registry so you can trace which scanner version produced which result during incident reviews.