Preprocessing Pipeline

A robust pipeline must be deterministic and idempotent. Given the same raw input string, the pipeline must always produce the exact same normalized output, regardless of when or how many times it is executed.

• Deterministic logic: Avoid non-deterministic functions or reliance on external mutable state during normalization.
• Idempotent design: Re-running the pipeline on already-normalized text must not alter it further.
• Caching key: The normalized form often serves as a cache key for deduplication and retrieval systems.

The sequence of normalization steps is critical. A poorly ordered pipeline can destroy information needed by a later step. The canonical order is generally:

1. Charset Detection & Unicode Normalization: Resolve encoding ambiguities first.
2. Structure Removal: Strip HTML, XML, or boilerplate.
3. Case Folding: Lowercase the clean text.
4. Tokenization: Segment into discrete tokens.
5. Lexical Normalization: Apply stemming or lemmatization.
6. Stop Word Filtering: Remove low-information tokens last.

A single pipeline configuration cannot serve all languages. Robust architectures implement language-specific branching triggered by automatic language identification.

• Language ID: Use a fast classifier (e.g., fastText or CLD3) to detect the language before normalization.
• Tokenizer selection: Route to language-appropriate tokenizers (e.g., MeCab for Japanese, Jieba for Chinese, spaCy for English).
• Stemmer/Lemmatizer: Apply language-specific morphological reducers; a Porter Stemmer is meaningless for non-English text.
• Stop word lists: Maintain curated stop word lists per language.

Normalization is inherently destructive. A robust pipeline preserves the original raw text and maps offsets to the normalized form to support high-precision entity linking and search snippet generation.

• Character offset mapping: Maintain a mapping from each normalized token back to its original character span in the raw text.
• Original text storage: Always store the raw, unnormalized string alongside the normalized version.
• Inline annotation: Use standoff annotation formats to layer normalization results without mutating the original source of truth.

Stop word filtering is not universally beneficial. A robust pipeline makes stop word removal a configurable, reversible decision rather than a hard-coded step.

• Semantic search: Modern dense retrieval models (e.g., BERT-based) use the full context; removing stop words can degrade semantic understanding.
• Sparse retrieval: For BM25-based keyword search, filtering stop words reduces index size and noise.
• Configurable lists: Allow dynamic injection of domain-specific stop words (e.g., 'patient' in a medical corpus may be too frequent to be discriminative).

A production pipeline must handle malformed input gracefully without crashing the entire ingestion process. Observability is non-negotiable.

• Exception handling: Wrap each step in a try-catch; log the raw input and stack trace, then route the document to a dead-letter queue for manual inspection.
• Metrics emission: Track throughput (docs/sec), latency per step, and error rates.
• Schema validation: Validate the output of each stage against an expected schema to catch silent data corruption early.

The process of converting all characters to a single case, typically lowercase, to collapse semantically identical tokens. This ensures that 'Apple', 'APPLE', and 'apple' map to the same index.

• Reduces vocabulary size and sparsity in Bag-of-Words models
• Can destroy information for tasks where case carries meaning, such as Named Entity Recognition
• Often paired with Truecasing to restore case when needed

The removal of high-frequency, low-semantic-value tokens such as 'the', 'is', and 'at'. These words follow Zipf's Law and dominate term frequency counts without contributing to topical relevance.

• Reduces dimensionality of TF-IDF vectors
• Improves computational efficiency for BM25 and sparse retrieval
• Modern transformer models often retain stop words as they provide syntactic context for attention mechanisms

Two competing approaches to reducing inflectional morphology to a base form. Stemming uses heuristic suffix-stripping rules, while Lemmatization uses vocabulary and Part-of-Speech Tagging to return a dictionary form.

• Porter Stemmer crudely reduces 'running' to 'run' but also 'university' to 'univers'
• Lemmatization correctly maps 'better' to 'good' using morphological analysis
• Stemming favors recall; lemmatization favors precision

A standard process defined by the Unicode Consortium to transform text into a consistent canonical form. Visually identical characters can have multiple binary representations, causing matching failures.

• NFD decomposes characters into base + combining marks
• NFC composes characters into precomposed forms
• Essential for multilingual pipelines where 'café' might be encoded as caf\u00e9 or cafe\u0301

The removal of extraneous artifacts that corrupt the linguistic signal before normalization. Raw text from web scraping or OCR often contains non-textual noise.

• Stripping HTML tags, JavaScript, and CSS from web documents
• Removing control characters and non-printable Unicode
• Filtering boilerplate content such as navigation menus and footers
• Essential for Charset Detection to prevent mojibake from incorrect encoding