Text Normalization

Case normalization converts all characters in a text string to either lowercase or uppercase. This reduces the vocabulary size by treating words like 'Apple', 'apple', and 'APPLE' as identical tokens.

• Primary Benefit: Eliminates sparsity in token-based representations, improving recall for lexical search.
• Consideration: Can lose meaning in case-sensitive contexts (e.g., 'Python' the language vs. 'python' the snake). It is almost universally applied in embedding models like Sentence Transformers.

Diacritic removal strips accent marks and other diacritical signs from characters (e.g., converting 'résumé' to 'resume' or 'naïve' to 'naive'). This is often performed via Unicode normalization forms (NFD) followed by filtering.

• Primary Benefit: Ensures text in different orthographic forms is treated identically, crucial for multilingual corpora.
• Implementation: Commonly uses libraries like unicodedata in Python. Essential for languages with frequent diacritic use, like French, Spanish, or German.

Contraction expansion converts shortened word forms into their full equivalents (e.g., 'don't' to 'do not', 'I'll' to 'I will'). This standardizes vocabulary and can improve token alignment for certain downstream tasks.

• Primary Benefit: Reduces ambiguity for models that may treat contractions as unique, out-of-vocabulary tokens.
• Consideration: Can increase token count. Often implemented using predefined mapping dictionaries. Its necessity has diminished with robust subword tokenizers like those used in modern LLMs.

Unicode normalization resolves different binary representations of the same textual character into a single, canonical form. The four primary forms are NFC, NFD, NFKC, and NFKD.

• NFC (Composition): Combines characters and diacritics into precomposed forms (e.g., é).
• NFD (Decomposition): Breaks characters into base + combining marks (e.g., e + ´).
• Use Case: Critical for ensuring 'café' and 'café' are treated as identical strings. NFKC/NFKD also handle compatibility equivalents (e.g., font variants).

Whitespace standardization collapses multiple spaces, tabs, and newlines into a single space (or a standard delimiter), and often strips leading/trailing whitespace.

• Primary Benefit: Eliminates formatting noise from copy-paste operations or document conversion, ensuring consistent tokenization.
• Implementation: A simple regex operation (re.sub(r'\s+', ' ', text).strip()). It is a foundational step before any semantic or delimiter-based splitting.

Punctuation handling involves either removing punctuation marks or standardizing them. Common strategies include full removal or separating punctuation from words.

• Removal: Strips characters like . , ! ? " '. Simplifies text but can destroy meaning in domains like code or scientific notation.
• Separation: Adds spaces around punctuation (e.g., 'end.' becomes 'end .'). This preserves punctuation as separate tokens, which is often the default behavior of modern tokenizers like tiktoken or Hugging Face tokenizers.

Tokenization is the foundational process of splitting raw text into smaller units called tokens, which are the atomic elements processed by language models. It is a prerequisite for accurate chunk sizing and embedding.

• Types: Word-level, subword (e.g., BPE), and character-level tokenization.
• Impact on Chunking: Chunk size limits (e.g., 512 tokens) are defined by token count, not character count, making the tokenizer's behavior critical for predictable chunk boundaries.
• Example: The sentence "I'm here" might be tokenized as ["I", "'m", " here"] by a subword tokenizer, affecting the token-length calculation for a fixed-length chunking strategy.

Sentence Boundary Detection (SBD) is the NLP task of identifying where sentences begin and end in plain text. It is a crucial precursor to semantic chunking strategies that use sentences as natural boundaries.

• Challenges: Abbreviations (e.g., "Dr."), decimal points, and ellipses can cause false sentence splits.
• Tools: Libraries like spaCy and NLTK provide robust, rule-based SBD modules.
• Use Case: Enables the creation of coherent chunks that preserve complete thoughts, improving the semantic integrity of retrieved passages for a language model's context window.

Document preprocessing is the collective set of operations applied to raw text before chunking. Text normalization is a core component of this pipeline.

• Common Steps:
  • Cleaning: Removing HTML tags, non-printable characters, and boilerplate.
  • Normalization: Lowercasing, accent removal, and expanding contractions (part of text normalization).
  • Structuring: Detecting and labeling headers, lists, and other layout elements.
• Goal: To transform heterogeneous, noisy source documents (PDFs, web pages, Word docs) into a clean, consistent text corpus ready for segmentation and vectorization.

Chunk deduplication is the process of identifying and removing near-identical text chunks from a corpus. It relies on normalized text for accurate comparison.

• Purpose: Improves retrieval efficiency and reduces noise by preventing redundant chunks from dominating search results.
• Techniques:
  • Exact Hashing: For identical strings after normalization.
  • Fuzzy Hashing: Using algorithms like SimHash or MinHash to detect near-duplicates with minor variations.
• Workflow: Text is first normalized (e.g., lowercased, whitespace stripped) to ensure duplicates are recognized despite superficial formatting differences.

Stop word removal is the filtering out of extremely common words (e.g., "the," "is," "and") that carry little semantic weight. It is sometimes applied during preprocessing for sparse lexical retrieval methods.

• Context-Dependent Utility:
  • Beneficial for: Keyword-based (BM25) search, where it reduces index size and focuses on content-bearing terms.
  • Often Harmful for: Dense vector embeddings, where removing common words can distort the semantic meaning captured by models like Sentence Transformers.
• Relation to Normalization: Often performed after normalization steps like lowercasing to ensure stop word lists are matched correctly.

Stemming and lemmatization are text normalization techniques that reduce words to their root form. They are primarily used in lexical search systems to improve recall.

• Stemming: A crude heuristic that chops off word endings (e.g., "running" -> "run").
• Lemmatization: A linguistic analysis that returns the dictionary base form (lemma) of a word (e.g., "better" -> "good").
• Application in RAG: While less critical for modern dense embedding models, they remain valuable in hybrid retrieval systems that combine dense and sparse (keyword) search, ensuring queries like "running" match documents containing "run."