Invisible Whitespace Character Stripping Prompt

This prompt is for data pipeline engineers and platform operators whose downstream parsers, tokenizers, or databases fail on model-generated text containing invisible Unicode characters. The job-to-be-done is sanitizing a raw string by removing or replacing zero-width spaces (U+200B), non-breaking spaces (U+00A0), zero-width non-joiners (U+200C), and other invisible code points that are visually absent but structurally destructive. The ideal user is integrating an LLM's output into a system with strict character whitelists—such as a SQL database, a CSV parser, a JSON serializer, or a search index—and needs a deterministic, auditable cleaning step before ingestion.

Use this prompt when your validation layer has already confirmed that the model's output is structurally valid (e.g., valid JSON or well-formed XML) but downstream processes still break due to invisible characters. It is appropriate when the cost of a false positive (removing a character that was intentionally present) is lower than the cost of a false negative (allowing a destructive character through). The prompt is designed to be run as a post-processing step in a data pipeline, not as a real-time user-facing filter. It expects a single string input and returns a cleaned string along with a structured audit log of what was removed, enabling traceability and debugging.

Do not use this prompt for strings that require preservation of intentional Unicode formatting, such as bidirectional text markers in Arabic or Hebrew, or zero-width joiners in emoji sequences and Indic scripts. For those cases, use the dedicated Zero-Width Character Detection and Removal Prompt, which includes context-aware preservation logic. This prompt is also not a substitute for proper UTF-8 validation; if the input may contain invalid byte sequences, pair it with the Malformed UTF-8 Sequence Repair Prompt first. After implementing, always run the provided eval checks against a golden dataset of known destructive and benign invisible characters to calibrate the removal threshold for your specific system.

This prompt template is designed to be dropped directly into your post-processing pipeline. It instructs the model to act as a sanitization filter, removing invisible whitespace characters that break downstream parsers, tokenizers, and comparison logic. The template uses square-bracket placeholders for all variable inputs, making it easy to integrate with your application's templating engine or prompt assembly layer.

textCopy
You are a text sanitization filter. Your only job is to remove invisible whitespace characters from the provided text while preserving all visible content and standard whitespace.

INPUT TEXT:
[INPUT]

CHARACTERS TO REMOVE:
- Zero-width space (U+200B)
- Zero-width non-joiner (U+200C)
- Zero-width joiner (U+200D)
- Zero-width no-break space (U+FEFF, Byte Order Mark)
- Non-breaking space (U+00A0) — convert to standard space (U+0020)
- Soft hyphen (U+00AD)
- Left-to-right mark (U+200E)
- Right-to-left mark (U+200F)
- Line separator (U+2028)
- Paragraph separator (U+2029)
- Ogham space mark (U+1680)
- En quad (U+2000), Em quad (U+2001), En space (U+2002), Em space (U+2003), Three-per-em space (U+2004), Four-per-em space (U+2005), Six-per-em space (U+2006), Figure space (U+2007), Punctuation space (U+2008), Thin space (U+2009), Hair space (U+200A) — convert all to standard space (U+0020)
- Any other Unicode characters in the "Separator, Space" (Zs) category or "Format" (Cf) category that are invisible

CONSTRAINTS:
[CONSTRAINTS]

OUTPUT FORMAT:
Return a JSON object with the following structure:
{
  "cleaned_text": "<the sanitized text>",
  "removed_characters": [
    {"character": "<Unicode code point>", "name": "<character name>", "count": <number of occurrences removed>}
  ],
  "changes_made": <total number of character removals or replacements>
}

Do not include any text outside the JSON object. Do not wrap the JSON in markdown code fences.

To adapt this template, replace [INPUT] with the text you need to sanitize. The [CONSTRAINTS] placeholder should be filled with any additional rules specific to your use case—for example, preserving intentional zero-width joiners in emoji sequences, keeping non-breaking spaces in HTML contexts, or adding custom characters to the removal list. If you have no additional constraints, replace [CONSTRAINTS] with 'None. Apply the default removal rules exactly as specified.' The output schema is designed to give you both the cleaned text and an audit trail of what was removed, which is essential for debugging false positives in production pipelines. Before deploying, test this prompt against a golden dataset that includes known invisible characters and verify that the removed_characters array accurately reports each removal.

The Invisible Whitespace Character Stripping Prompt is designed to be a post-processing repair step, not a standalone application. It should sit between the model's raw output and any downstream parser, database, or API that rejects or misinterprets invisible Unicode characters. The typical integration point is inside a data pipeline function, an API gateway middleware, or a validation hook that fires after the primary model response is received but before the payload is serialized to JSON, inserted into a database, or passed to a tokenizer. The prompt expects a single string input—the potentially contaminated text—and returns a cleaned string plus a structured audit report of what was removed. This audit report is critical for production observability; without it, silent data corruption can go undetected for weeks.

To wire this into an application, wrap the prompt call in a function that accepts the raw string, calls the model with the prompt template and the input, parses the JSON response, and validates the output shape. The response schema should be strictly enforced: a cleaned_text field containing the sanitized string, and a removals array where each object has a character, unicode_codepoint, count, and positions field. After parsing, run a structural validator that confirms the cleaned_text length plus the total removed character count equals the original input length. If validation fails, retry once with a more explicit instruction in the prompt, such as "You must account for every character in the input." If the retry also fails, log the failure, attach the original and partially cleaned strings, and route to a dead-letter queue for manual inspection. Do not silently pass unvalidated output downstream.

For model choice, prefer a fast, cheap model for this task—GPT-3.5 Turbo, Claude Haiku, or a fine-tuned small open-weight model are sufficient. The task is mechanical, not semantic. Latency matters because this is a synchronous repair step in a pipeline. Set a strict timeout (e.g., 2 seconds) and a token limit appropriate to your input size. If the model call times out or exceeds the token limit, fall back to a deterministic regex-based cleaner that strips known zero-width characters (U+200B, U+200C, U+200D, U+FEFF, U+00A0, etc.) and logs a warning that the AI repair was bypassed. This hybrid approach—AI repair with a deterministic fallback—gives you the nuance of model-based detection for rare or novel invisible characters while maintaining pipeline reliability. Log every repair event, including the model used, latency, characters removed, and whether the fallback was triggered. This data will help you decide if the prompt is still necessary or if your upstream model has improved enough to retire this repair step.