Tokenization

Word tokenization splits text into tokens based on whitespace and punctuation, where each token typically corresponds to a linguistic word. This is the most intuitive method for languages like English.

• Advantages: Simple to implement; aligns with human intuition.
• Disadvantages: Creates large vocabularies; suffers from the out-of-vocabulary (OOV) problem for rare or misspelled words.
• Example: "Can't tokenize this!" → ["Can't", "tokenize", "this!"]
• Common Use: Early NLP models like bag-of-words and traditional search indexes (e.g., BM25).

Subword tokenization decomposes words into smaller, frequently occurring units (subwords), balancing vocabulary size and OOV handling. It is the standard for modern transformer-based models.

• Key Algorithms: Byte-Pair Encoding (BPE) (used by GPT models), WordPiece (used by BERT), Unigram Language Model.
• Process: Learns a vocabulary of common character sequences (e.g., "ing", "ed", "un") from a corpus.
• Example: "tokenization" might be split into ["token", "ization"].
• Advantage: Enables the model to handle novel words by piecing together known subwords.

Character tokenization treats each individual character (including spaces and punctuation) as a token. This represents the finest granularity of text segmentation.

• Advantages: Extremely small, fixed vocabulary (e.g., ~100 tokens for English); eliminates the OOV problem entirely.
• Disadvantages: Creates very long sequences; obscures semantic and morphological structure, making learning more difficult.
• Example: "cat" → ["c", "a", "t"].
• Common Use: Specialized applications in morphologically rich languages, noisy text (social media), or when modeling at the character level is essential.

SentencePiece is a language-agnostic tokenization tool that implements subword segmentation directly on raw text bytes, treating spaces as part of the vocabulary. Byte-Level BPE (BBPE) is a variant used by models like GPT-2 and RoBERTa.

• Key Feature: Tokenizes text without pre-tokenization (no dependency on language-specific rules). This is crucial for multilingual models and handling code or emojis.
• Byte-Level: Operates on UTF-8 bytes, ensuring the vocabulary size is capped at 256, making it impossible to encounter an OOV token.
• Use Case: Foundational for training large, multilingual models (e.g., T5, mT5) as it can seamlessly process text from hundreds of languages.

The tokenizer vocabulary is a learned artifact, not a predefined dictionary. Its size and composition are critical hyperparameters.

• Training Process: A corpus is analyzed to determine the most frequent character n-grams or merge operations (in BPE).
• Vocabulary Size: Typically ranges from ~30k to 200k tokens. A larger vocabulary yields shorter sequences but risks overfitting to the training data distribution.
• Special Tokens: Vocabularies include control tokens like [CLS], [SEP], [PAD], [UNK], and [MASK] for model-specific tasks (classification, separation, padding, unknown, masking).
• Impact: Directly affects model performance, memory usage, and inference speed.

In Retrieval-Augmented Generation (RAG) and search engines, tokenization strategies differ between the retriever and the generator.

• Sparse Retrievers (e.g., BM25): Rely on word or subword tokenization to build an inverted index. Overlap between query and document tokens determines relevance.
• Dense Retrievers: Use a separate tokenizer (often subword) to encode text into embeddings for semantic search in a vector space.
• Generator (LLM): Uses its own pre-trained tokenizer (e.g., BPE for GPT). A mismatch between the retriever's and generator's tokenization can create alignment issues in the RAG pipeline.
• Best Practice: Ensure consistent or compatible tokenization across pipeline components to maintain semantic coherence.

Lemmatization is the linguistic process of reducing a word to its base or dictionary form (lemma), accounting for its part of speech and morphological analysis. Unlike stemming, which uses crude heuristics, lemmatization applies vocabulary and morphological analysis to return a valid root word.

• Purpose: Normalizes different inflected forms of a word (e.g., 'running', 'ran', 'runs' → 'run') to improve retrieval recall.
• Example: The tokens ['better', 'best'] would both be lemmatized to the base form 'good'.
• Key Difference from Stemming: Lemmatization considers context and part-of-speech to return a linguistically valid lemma, whereas stemming often produces non-words (e.g., 'studies' → 'studi').

Named Entity Recognition (NER) is a natural language processing task that identifies and classifies named entities—such as persons, organizations, locations, dates, and quantities—within a sequence of tokens.

• Function: Transforms unstructured tokens into structured, categorized information. For example, in the query 'Schedule a meeting with the CEO of Tesla in Austin next Friday', NER would tag 'Tesla' as ORGANIZATION, 'Austin' as LOCATION, and 'next Friday' as DATE.
• Importance for Retrieval: Critical for understanding key factual components of a query, enabling precise filtering and routing to domain-specific knowledge bases or databases.

Entity Linking is the process of disambiguating and connecting textual mentions of named entities (identified by NER) to their corresponding unique entries in a knowledge base, such as Wikipedia or a domain-specific ontology.

• Process: Resolves surface forms (e.g., 'Apple') to a canonical entity ID (e.g., Apple Inc. the company vs. apple the fruit) by using contextual clues.
• Benefit for Search: Enables systems to retrieve information about the entity, not just documents containing the ambiguous string. It grounds the query in a concrete, shared understanding of the world, which is essential for factual accuracy in RAG.

Semantic Parsing is the task of converting a natural language query into a formal, machine-executable meaning representation. This goes beyond tokenization and entity recognition to capture the user's logical intent.

• Output Formats: The output can be a logical form, a database query (like SQL or SPARQL), an API call, or a function in a domain-specific language.
• Example: The query 'Which employees joined after 2020 and work in the Boston office?' would be parsed into a structured database query with predicates for hire_date > 2020-01-01 and office_location = 'Boston'.
• Role in Complex Queries: Essential for moving from keyword-based retrieval to true question answering over structured data sources.

Dependency Parsing is a syntactic analysis technique that identifies grammatical relationships between words in a sentence, representing them as a tree of directed links from heads to dependents.

• Visualization: Creates a graph where each token is a node, and labeled arcs (e.g., nsubj for nominal subject, dobj for direct object) define syntactic roles.
• Use Case for Understanding: Crucial for interpreting query structure. It identifies the main action (verb), its subject, and objects, which helps in intent classification and query reformulation. For instance, it distinguishes 'Python tutorials' (Python modifies tutorials) from 'tutorials on Python' (Python is the object of the preposition).

Query Embedding is the process of transforming a textual query into a dense, fixed-dimensional vector representation using a neural model, enabling semantic similarity search in a vector space.

• Mechanism: A transformer-based encoder (like BERT or a specialized bi-encoder) maps the entire sequence of tokens to a single high-dimensional vector that captures its semantic meaning.
• Core of Dense Retrieval: This embedding is then compared against pre-computed document embeddings using a similarity metric like cosine similarity to find semantically relevant passages, even in the absence of exact keyword matches.