User-Generated Content PII Moderation Prompt Template

This prompt is designed for content moderation teams and trust & safety engineers who need to filter personally identifiable information (PII) from user-submitted content before it enters a model's context window, a public-facing display, or an internal database. The primary job-to-be-done is automated, high-recall detection of accidental self-doxxing, contact information sharing, and other PII leaks in unstructured text such as product reviews, forum posts, support tickets, and community comments. The ideal user is a developer or moderator integrating this prompt into a moderation pipeline where human review capacity is limited and consistency is critical.

Use this prompt when you need a structured, actionable moderation decision—not just a binary yes/no—for each piece of content. The template is built to return a specific JSON schema that includes the detected PII categories, a recommended action (such as REDACT, BLOCK, or ALLOW), and a confidence score. It is most effective when the primary risk is users exposing their own or others' private data, rather than a model generating it. You should wire this prompt into a pre-processing step, where user content is evaluated before it is stored, displayed, or passed to another model for summarization or analysis.

Do not use this prompt as a post-generation safety net for model outputs; for that scenario, use the sibling 'PII Detection and Redaction Prompt Template' in the 'Sensitive Data Redaction and PII Prompts' pillar. This prompt is also not a replacement for a dedicated, pattern-based PII scanner for high-precision, low-latency detection of structured data like credit card numbers. Instead, it excels at understanding context to disambiguate benign text from actual PII (e.g., a user saying 'my email is mike at example dot com' vs. a generic placeholder). For high-risk, regulated environments, always route low-confidence moderation decisions to a human review queue. The next step is to copy the prompt template, configure your PII categories and moderation actions, and integrate it with a validation layer that enforces the expected JSON output schema.

This template is designed to be copied directly into your prompt management system, IDE, or orchestration layer. It accepts user-generated content and a configurable moderation policy, then returns a structured verdict with detected PII categories, severity, and recommended actions. Every placeholder is enclosed in square brackets and must be replaced with concrete values before the prompt is sent to the model. The template assumes the model will receive the full prompt as a single turn; if you are using a multi-turn or streaming architecture, adapt the output format instructions accordingly.

textCopy
You are a content moderation assistant specialized in detecting personally identifiable information (PII) in user-generated content. Your task is to analyze the submitted content, identify any PII present, and recommend moderation actions based on the provided policy.

## INPUT
[USER_CONTENT]

## MODERATION POLICY
[MODERATION_POLICY]

## INSTRUCTIONS
1. Scan the entire input for PII across the following categories:
   - Full names (first and last)
   - Email addresses
   - Phone numbers
   - Physical addresses (street, city, postal code)
   - Government identifiers (SSN, passport, driver's license)
   - Financial account numbers (credit card, bank account)
   - Online identifiers (usernames, profile URLs, IP addresses)
   - Dates of birth
   - Geolocation data (coordinates, precise locations)
2. For each detected PII instance, record:
   - The exact text found
   - The PII category
   - The character offset range (start and end positions)
   - A confidence score from 0.0 to 1.0
3. Classify the overall content into one of these severity levels:
   - `none`: No PII detected
   - `low`: Single piece of low-sensitivity PII (e.g., first name only)
   - `medium`: Multiple PII instances or single piece of moderate-sensitivity PII
   - `high`: High-sensitivity PII (government IDs, financial data, full contact details)
   - `critical`: Multiple high-sensitivity PII instances or apparent self-doxxing
4. Recommend one of these moderation actions:
   - `approve`: Content is safe to publish as-is
   - `redact`: Remove PII and publish redacted version
   - `quarantine`: Hold for human review before publishing
   - `reject`: Block publication and notify user
   - `escalate`: Immediate escalation to security or legal team
5. If the moderation policy specifies different thresholds or actions, follow the policy over these defaults.

## OUTPUT FORMAT
Return a valid JSON object with this exact schema:
{
  "severity": "none|low|medium|high|critical",
  "action": "approve|redact|quarantine|reject|escalate",
  "pii_instances": [
    {
      "text": "string",
      "category": "string",
      "start_offset": integer,
      "end_offset": integer,
      "confidence": float
    }
  ],
  "redacted_content": "string or null",
  "review_notes": "string"
}

## CONSTRAINTS
- Do not invent PII that is not present in the input.
- If you are uncertain about a detection, set confidence below 0.7 and flag it in review_notes.
- For `redact` actions, provide the redacted_content with PII replaced by [REDACTED:category].
- Never output the original PII in review_notes if the action is `reject` or `escalate`.
- If the content contains PII that appears to belong to a minor, escalate regardless of policy thresholds.

To adapt this template for your environment, replace [USER_CONTENT] with the actual text submitted by the user, and [MODERATION_POLICY] with your organization's specific PII handling rules. The policy can include custom severity thresholds, allowed PII categories (for example, a platform that permits usernames but not real names), and escalation contacts. If your moderation pipeline already performs regex-based PII detection, you can add a [PRE_SCAN_RESULTS] placeholder to pass those findings as context, reducing the model's workload and improving consistency between rule-based and model-based detection layers.

Before deploying, test this prompt against a golden dataset that includes known PII instances, borderline cases (such as sample phone numbers or test credit card numbers), and content with no PII. Measure both precision and recall, and pay special attention to false positives on business addresses, public figures' names, and placeholder text like john.doe@example.com. If your use case involves user-facing moderation decisions, always route quarantine and escalate actions to a human review queue rather than acting on model output alone.

This prompt is designed to operate as a synchronous pre-publish gate within a content moderation workflow. When a user submits a comment, review, or forum post, the application should call the LLM with this prompt after basic input sanitization but before the content is written to the public database. The prompt expects raw user text as [INPUT] and a structured [OUTPUT_SCHEMA] that defines the expected JSON response containing a pii_detected boolean, a list of findings with type and confidence, and a moderation_action recommendation (allow, flag, block). The application layer must enforce the moderation action, not the model—the model recommends, the system decides.

Integration flow: The application should wrap the LLM call in a service function that handles the full lifecycle. First, validate the response against the expected JSON schema using a library like ajv or pydantic. If validation fails, retry once with a repair prompt that includes the raw output and the schema error. If the retry also fails, escalate to a human review queue and default to the safest moderation action (block for high-risk categories, flag for others). Log every decision—input hash, model response, validation result, and final action—to a structured audit table. This log is essential for tuning false positive rates and defending moderation decisions later.

Model choice and latency: For real-time user-facing flows, prefer smaller, faster models (e.g., GPT-4o-mini, Claude Haiku) with strict JSON mode enabled. The prompt's [RISK_LEVEL] parameter can be set to high for regulated communities (healthcare forums, children's platforms) where you might route to a larger model or add a second verification pass. If latency exceeds 500ms, consider asynchronous processing: accept the content, show a 'pending review' state, and process the PII check in the background before publishing. Never block the user experience on a slow model call without clear UX feedback.

Human review integration: The moderation_action field should map directly to your review queue logic. allow publishes immediately. flag creates a low-priority review task where a moderator can confirm or overturn the decision within a few hours. block prevents publication and creates a high-priority task. For each review task, include the original text, the model's findings with highlighted spans, and a one-click approve/reject interface. Track reviewer decisions to calculate your model's precision and recall over time. If reviewers overturn more than 10% of block decisions, adjust your confidence thresholds or add more few-shot examples of borderline cases to the prompt's [EXAMPLES] block.

What to avoid: Do not use the model's output to automatically delete or ban users without human confirmation. Do not log the raw user input alongside the model's PII findings in plaintext—the whole point is to prevent PII from entering your systems. Hash or tokenize the input in logs, and store the redacted version only. Finally, do not rely solely on this prompt for compliance with GDPR, CCPA, or COPPA; it is a detection layer, not a legal safeguard. Combine it with pattern-based scanning (regex for credit cards, SSNs) as a deterministic first pass, and use the LLM for contextual disambiguation that regex cannot handle.