TextTiling Algorithm

TextTiling's core mechanism is the analysis of lexical cohesion—the patterns of word repetition and co-occurrence that signal a consistent topic. It operates by:

• Calculating a similarity score (e.g., cosine similarity) between term-frequency vectors within a moving window.
• Plotting these scores to create a cohesion graph, where peaks indicate high topical continuity and valleys signal potential topic boundaries.
• This approach is unsupervised and language-agnostic, requiring no pre-labeled data or domain-specific rules.

The algorithm uses a sliding window to simulate a reader's attention span. The text is divided into token sequences (pseudo-sentences).

• A fixed-size window moves across these sequences, dividing them into two adjacent blocks.
• A similarity score is computed between the blocks at each position, measuring the lexical overlap.
• A deep valley in the resulting similarity plot indicates a topic shift, as the vocabulary between the two blocks becomes dissimilar. The window size is a critical hyperparameter controlling sensitivity to topic changes.

Topic boundaries are not identified by simple thresholding but by calculating a depth score for each valley in the cohesion graph.

• The depth score measures the magnitude of the dip relative to the surrounding peaks.
• Boundaries are placed at the deepest valleys, corresponding to the most significant drops in lexical cohesion.
• This method is robust to minor vocabulary fluctuations, preventing over-segmentation of text with minor subtopic digressions.

A key strength of TextTiling is that it requires no training data. It operates purely on the lexical statistics of the input text.

• This makes it immediately applicable to any domain—from legal documents to scientific papers—without fine-tuning.
• Its performance is dependent on the vocabulary consistency within a topic; it works best on formal, expository text where authors use consistent terminology.
• It is less effective on highly narrative or conversational text where vocabulary may shift for stylistic rather than topical reasons.

TextTiling's output is highly sensitive to preprocessing steps. Effective implementation requires:

• Stop word removal: Common function words ("the," "is") are typically filtered out as they add noise to cohesion signals.
• Stemming or Lemmatization: Reducing words to their root form (e.g., "running" to "run") groups morphologically related terms, strengthening cohesion signals.
• The choice of token sequence size (pseudo-sentence length) directly impacts the granularity of the detected segments.

TextTiling is a foundational technique for semantic chunking in Retrieval-Augmented Generation (RAG) pipelines.

• It creates chunks that are topically coherent, which improves the relevance of retrieved context for language models compared to arbitrary fixed-size splitting.
• It is often used in a hybrid approach, combined with other methods like recursive character splitting to respect both semantic boundaries and hard token limits.
• Its output provides a strong baseline for more advanced, embedding-based segmentation methods.

The overarching goal of segmenting documents into coherent units based on contextual meaning and topic boundaries. Unlike fixed-size splitting, semantic chunking aims to preserve the integrity of ideas.

• Core Objective: Optimize the relevance of retrieved information for language models by ensuring each chunk is a self-contained semantic unit.
• Methods: Includes algorithms like TextTiling, embedding-based segmentation, and markdown header splitting.
• Key Benefit: Reduces the risk of providing language models with fragmented or contextually incomplete information, which is critical for accurate retrieval-augmented generation.

The linguistic phenomenon where a text is held together by the repetition and semantic relatedness of words. TextTiling's algorithm is fundamentally a computational measure of this concept.

• Mechanism: Analyzes patterns of term co-occurrence across a moving window. A drop in cohesion signals a potential topic boundary.
• Measurement: Often calculated using cosine similarity or a dot product of term frequency vectors within adjacent blocks of text.
• Engineering Relevance: Provides an unsupervised, language-agnostic signal for segmentation without requiring pre-trained models or labeled data.

A modification of the BERT model designed to derive semantically meaningful sentence embeddings. It enables a modern, embedding-based approach to measuring semantic similarity for chunking.

• Function: Creates dense vector representations for sentences or paragraphs where cosine similarity indicates semantic relatedness.
• Contrast with TextTiling: Can be used for embedding-based chunking by segmenting text where the similarity between consecutive sentence embeddings falls below a threshold.
• Trade-off: More computationally intensive than lexical methods but can capture deeper semantic relationships beyond simple word overlap.

A simple but effective chunking technique created by moving a fixed-size context window across text with a specified overlap. It addresses a key limitation of semantic methods.

• Process: The window (e.g., 200 tokens) advances by a stride (e.g., 50 tokens), creating overlapping chunks.
• Purpose: Preserves context across arbitrary split points imposed by semantic segmenters, mitigating information loss at boundaries.
• Common Use Case: Often applied after a primary semantic chunking step (like TextTiling) to create final, size-constrained chunks for a vector database, ensuring no key context is isolated at a chunk edge.

A robust probabilistic ranking function for keyword search. It represents the sparse retrieval paradigm, which contrasts with the semantic cohesion analysis used in TextTiling.

• Core Principle: Scores documents based on term frequency, inverse document frequency, and document length normalization.
• Relation to TextTiling: While TextTiling analyzes term distribution within a single document for segmentation, BM25 analyzes term distribution across a corpus for retrieval.
• Hybrid Systems: Modern hybrid search architectures combine BM25's precise lexical matching with dense vector similarity (semantic search) for improved recall and precision.

A database index optimized for Approximate Nearest Neighbor (ANN) search over high-dimensional embeddings. It is the destination for chunks created by algorithms like TextTiling.

• Purpose: Enables fast semantic search by finding stored vectors (chunk embeddings) most similar to a query vector.
• Common Algorithms: Includes Hierarchical Navigable Small World (HNSW) graphs and Inverted File (IVF) indexes, as implemented in libraries like Faiss and vector databases (Weaviate, Qdrant).
• Data Pipeline: TextTiling segments text → an embedding model (e.g., SBERT) creates vectors → vectors are inserted into a dense vector index for sub-second retrieval.