Model Training Data PII Retrospective Scan Prompt Template

This prompt is designed for ML teams who need to audit existing fine-tuning datasets, pre-training corpora, or any text collection for personally identifiable information (PII) before it is used for model training or retraining. It is a retrospective scan, meaning the data already exists and you need to find contamination, not prevent it at generation time. Use this when you inherit a dataset, aggregate data from multiple sources, or prepare a compliance review before a training run. The prompt instructs the model to act as a data hygiene auditor, scanning each provided sample for common PII categories, assigning a contamination severity score, and recommending a remediation action. It supports batch processing by accepting multiple records in a structured input format.

This is not a real-time output guard or a replacement for deterministic PII detection libraries. Do not use this prompt on live user traffic, streaming model outputs, or in any latency-sensitive path. It is a pre-training data quality and compliance gate, best run as an offline batch job over static datasets. The prompt is most effective when paired with a validation layer that checks the output schema and a human review step for high-severity findings. For production output sanitization, use the sibling PII Detection and Redaction Prompt Template instead.

Before using this prompt, ensure you have a clear inventory of the data sources to be scanned, a defined PII taxonomy that matches your regulatory obligations, and a remediation playbook that specifies what happens to contaminated records (e.g., redaction, deletion, quarantine). The prompt assumes you can provide records in a structured batch format with unique identifiers. If your data is in raw documents, PDFs, or unstructured logs, pre-process it into the expected input schema first. After running the scan, always review high-severity findings manually and log all remediation actions for auditability.

This prompt template is designed for ML teams auditing fine-tuning datasets and training corpora. It scans existing records for inadvertently included PII—such as names, emails, phone numbers, and credentials—before training or retraining. The template is structured for batch processing: you can iterate over a dataset, feed each record into the prompt, and collect structured findings for remediation. The output includes a per-record risk classification, a list of detected PII instances with character offsets, and a recommended remediation action. This is not a real-time guard; it is a retrospective scan meant to run offline over static datasets.

textCopy
You are a PII detection specialist auditing a training dataset before model fine-tuning. Your task is to scan the provided [RECORD] for personally identifiable information (PII), credentials, and other sensitive data that should not be included in training corpora.

## Input
[RECORD]

## Detection Categories
Scan for the following categories and sub-types:
- **Direct Identifiers**: full names, email addresses, physical addresses, phone numbers, government IDs (SSN, passport, driver's license), dates of birth.
- **Credentials and Secrets**: API keys, passwords, tokens, connection strings, private keys.
- **Financial Data**: credit card numbers, bank account numbers, routing numbers.
- **Protected Health Information**: medical record numbers, patient IDs, diagnosis codes linked to individuals.
- **Network Identifiers**: internal IP addresses, hostnames, URLs with embedded credentials.
- **Contextual PII**: combinations of quasi-identifiers (e.g., ZIP + birthdate + gender) that could re-identify individuals.

## Output Schema
Return a valid JSON object with this exact structure:
{
  "record_index": [RECORD_INDEX],
  "risk_level": "high" | "medium" | "low" | "none",
  "findings": [
    {
      "category": "[CATEGORY]",
      "sub_type": "[SUB_TYPE]",
      "value": "[DETECTED_VALUE]",
      "char_offset_start": [INTEGER],
      "char_offset_end": [INTEGER],
      "confidence": 0.0-1.0,
      "is_false_positive_likely": true | false,
      "remediation": "redact" | "mask" | "remove_record" | "quarantine" | "review"
    }
  ],
  "summary": "[ONE_SENTENCE_SUMMARY]"
}

## Constraints
- Only report findings with confidence >= [CONFIDENCE_THRESHOLD].
- If no PII is detected, return an empty `findings` array and risk_level "none".
- Do not hallucinate findings. If a string looks like PII but you are uncertain, set `is_false_positive_likely` to true and `remediation` to "review".
- For test or example data (e.g., "123-45-6789" in documentation), flag it but set `is_false_positive_likely` to true.
- Preserve the original record content; do not modify it in your response.

## Examples
[EXAMPLES]

## Risk Level Definitions
- **high**: Contains direct identifiers or credentials that could cause harm if exposed.
- **medium**: Contains quasi-identifiers or contextual PII that could contribute to re-identification.
- **low**: Contains only benign personal references unlikely to cause harm (e.g., public figures, test data).
- **none**: No PII detected.

To adapt this template, replace the square-bracket placeholders with your actual values. [RECORD] should be a single training example as a string. [RECORD_INDEX] is the row number or ID for traceability. [CONFIDENCE_THRESHOLD] should be a float between 0.0 and 1.0—start at 0.7 and adjust based on your false positive tolerance. [EXAMPLES] should contain 2-4 few-shot examples showing correct output format for both clean and contaminated records. If your dataset contains domain-specific PII (e.g., proprietary customer IDs), add those categories to the Detection Categories list. For batch processing, wrap this prompt in a script that iterates over your dataset, collects all findings into a remediation report, and quarantines high-risk records before training begins. Always run this scan offline on a static dataset snapshot, not on live production data streams.

Before running this prompt at scale, validate the output schema with a small sample of 50-100 records that include known PII and known clean records. Measure recall (did you catch the known PII?) and precision (did you flag clean records?). If false positives exceed 10%, raise the confidence threshold or add negative examples to [EXAMPLES]. For high-risk datasets—healthcare, finance, or customer support logs—require human review of all high and medium risk findings before remediation. Never rely solely on this prompt for compliance; it is a detection aid, not a legal safeguard. After remediation, re-run the scan to verify that redacted records now return risk_level: "none".

This prompt is designed as a batch processing step, not a one-off chat interaction. You will feed it individual training samples—text strings, JSON records, or conversation turns—from a fine-tuning dataset or pre-training corpus. The prompt returns a structured JSON verdict per sample: a PII detection flag, a list of found PII types with character offsets, a confidence score, and a remediation recommendation. The application layer is responsible for iterating over the dataset, calling the model for each sample, collecting results, and deciding what to quarantine, redact, or discard.

Build the harness around a retryable, idempotent processing loop. For each sample, store a unique sample_id before calling the model. On a malformed JSON response, retry up to two times with a repair prompt that includes the raw output and the expected schema. If the model returns a valid JSON payload but the confidence field is below your threshold (e.g., < 0.85), route that sample to a human review queue rather than auto-redacting. Log every decision—sample ID, model response, retry count, final action—to an audit table. For high-compliance datasets (healthcare, finance), require a second reviewer sign-off on any sample flagged for PII before it is removed from the training set.

Model choice matters. Use a model with strong instruction-following and JSON mode support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller open-weight models for this task unless you have calibrated their PII recall against a labeled test set. If your dataset is large, batch samples into groups of 50–100 and process them concurrently with rate limiting. Do not send the entire dataset in a single prompt context—the model will miss PII in long contexts and hallucinate offsets. Wire the output into your data versioning system (DVC, Pachyderm, or a feature store) so that the scanned, redacted dataset is tracked as a new artifact with a clear provenance link to the scan results.