Whitespace and Special Character Cleaning Prompt

This prompt is for data engineers and ETL developers who need to normalize free-text fields before they land in a structured data warehouse, application database, or analytics pipeline. The job-to-be-done is deterministic text cleaning: stripping leading and trailing whitespace, collapsing multiple internal spaces into one, removing or replacing non-printable control characters, and normalizing Unicode artifacts like smart quotes, em-dashes, and non-breaking spaces. The reader is not asking the model to interpret meaning or extract entities; they are asking it to apply a precise, repeatable set of normalization rules and return a clean value alongside an auditable change log.

Use this prompt when the input text originates from user-generated forms, copy-pasted content, legacy system exports, or document extraction pipelines where invisible characters and typographic glyphs cause downstream failures—schema validation errors, broken joins, or malformed CSV/JSON exports. It is appropriate when you need a structured output that includes both the cleaned string and a record of every transformation applied, so that a human or an automated audit system can verify correctness. Do not use this prompt for semantic normalization tasks like date parsing, address standardization, or entity resolution; those require domain-specific rules and confidence scoring covered by sibling playbooks. This prompt is also not a substitute for application-level string sanitization in security-critical contexts—SQL injection or XSS prevention must happen in code, not in an LLM call.

Before wiring this into a production pipeline, define the exact set of transformations your downstream system requires. The prompt template includes a [TRANSFORMATION_RULES] placeholder where you specify which characters to strip, replace, or preserve. If your data contains sensitive information, ensure the prompt runs in an environment that meets your data residency and privacy requirements. For high-volume ingestion, consider whether a deterministic regex-based cleaner in application code is more cost-effective and predictable than an LLM call; reserve this prompt for cases where the variety of Unicode anomalies and typographic quirks makes rule-based approaches brittle. After implementation, validate outputs against a golden dataset of known dirty inputs and expected clean outputs, and monitor the change log for unexpected transformations that may indicate rule gaps.

This prompt template is designed to be dropped directly into your application code or prompt management system. It accepts a raw text string and returns a cleaned version along with a detailed change log for auditability. The square-bracket placeholders allow you to inject field-specific rules, output format requirements, and risk-level controls without rewriting the core cleaning logic.

codeCopy
You are a text normalization engine. Your job is to clean whitespace and special characters from the provided [INPUT_TEXT] according to the rules below. Do not alter semantic content, only formatting and non-printable characters.

## Cleaning Rules
Apply these transformations in order:
1. Strip leading and trailing whitespace from the entire string.
2. Replace all non-printable characters (control characters, null bytes, vertical tabs) with a single space.
3. Normalize Unicode characters to NFC form.
4. Replace smart quotes (‘ ’ “ ”) with straight quotes (' ").
5. Replace em-dashes (—) and en-dashes (–) with standard hyphens (-).
6. Collapse all internal whitespace sequences (multiple spaces, tabs, newlines) into a single space.
7. Apply any additional field-specific rules from [FIELD_RULES].

## Output Format
Return a JSON object matching this exact schema:
{
  "cleaned_value": "string",
  "changes": [
    {
      "rule_applied": "string",
      "original_substring": "string",
      "replacement_substring": "string",
      "position_start": integer,
      "position_end": integer
    }
  ],
  "change_count": integer,
  "truncated": boolean
}

## Constraints
- [CONSTRAINTS]
- If the cleaned value exceeds [MAX_LENGTH] characters, truncate it and set truncated to true.
- If no changes are made, return an empty changes array and change_count of 0.
- Preserve all other characters exactly as they appear in the input.

## Risk Level: [RISK_LEVEL]
[RISK_INSTRUCTIONS]

## Examples
[EXAMPLES]

## Input
[INPUT_TEXT]

To adapt this template, replace each placeholder with your specific requirements. For [FIELD_RULES], add field-specific transformations like 'remove all pipe characters' or 'replace tabs with commas'. For [CONSTRAINTS], specify any characters that must be preserved, such as 'Preserve all XML tags and entities'. For [RISK_LEVEL] and [RISK_INSTRUCTIONS], use 'low' for non-critical fields where automated cleaning is safe, or 'high' for regulated data requiring human review of the change log before ingestion. The [EXAMPLES] placeholder should contain one or two input-output pairs demonstrating edge cases like mixed smart quotes or embedded null bytes. Wire the output into a validation step that checks the JSON schema, confirms change_count matches the length of the changes array, and routes high-risk changes to a review queue.

This prompt is designed to be a deterministic, idempotent step in an ETL or data ingestion pipeline. It should be called before any schema validation, type coercion, or canonicalization steps. The ideal integration point is immediately after raw text extraction from a source system (API, database, file upload) and before the record enters your normalization layer. Because the prompt returns both a cleaned_value and a change_log, you can treat it as a transparent transformation that supports audit requirements without blocking the pipeline.

Model choice and configuration: Use a fast, cost-efficient model for this task. GPT-4o-mini, Claude 3.5 Haiku, or equivalent small models are sufficient. Set temperature=0 to eliminate variability. The prompt should be called with a strict JSON mode or structured output feature enabled, targeting the schema { "cleaned_value": string, "change_log": [{ "operation": string, "position": number, "original": string, "replacement": string }] }. If your model provider does not support strict JSON mode, append the instruction: You must respond with ONLY valid JSON. No markdown fences, no commentary. and validate the response before proceeding.

Validation and retry logic: After receiving the model response, validate that cleaned_value is a non-null string and that change_log is an array of valid operation objects. If validation fails, retry once with the same prompt. If the second attempt fails, log the raw input and the malformed response, then fall back to a deterministic regex-based cleaner for the specific whitespace and special character rules you need. This ensures your pipeline never halts on a model failure. For high-throughput pipelines, consider batching multiple strings into a single JSON array input to reduce API call overhead, but keep batch sizes small (under 20 strings) to avoid hitting output token limits.

Audit and observability: Always log the change_log alongside the cleaned value in your data warehouse or event stream. This provides a traceable record of every transformation applied, which is critical for debugging downstream issues (e.g., a search index returning unexpected results because an em-dash was normalized). Attach the prompt version, model ID, and timestamp to each log entry. If your use case involves regulated data, route any record where the change_log contains more than a configurable threshold of operations (e.g., 5+ changes) to a human review queue before ingestion, as heavy cleaning may indicate corrupted source data that warrants investigation.