Abstract Syntax Tree (AST) Chunking

AST chunking parses source code into its Abstract Syntax Tree, a hierarchical representation of the program's grammatical structure. Chunk boundaries are defined by syntactic nodes such as function declarations, class definitions, or control flow blocks. This ensures each chunk is a complete, parseable unit of code, preserving logical integrity that is destroyed by character-based splitting.

The strategy requires a dedicated parser for each programming language (e.g., tree-sitter, libclang, language server protocols). These parsers understand the language's grammar rules to correctly build the AST.

• Examples: Python uses its built-in ast module; JavaScript/TypeScript often use @babel/parser or tree-sitter.
• Implication: Implementation is not language-agnostic; each supported language adds parser dependency and maintenance overhead.

Because chunks map to syntactic units like functions, they naturally encapsulate a single responsibility or concept. All variable definitions, control logic, and dependent statements within that unit are kept together. This maximizes the chunk's self-containment, providing the language model with a complete, executable context for understanding or generating code, which dramatically improves retrieval relevance for code-specific queries.

Traditional text splitters often sever code mid-line, mid-statement, or between a function call and its definition, creating syntactically invalid fragments. AST chunking eliminates this by ensuring splits only occur at natural syntactic boundaries. This prevents:

• Broken variable references
• Incomplete function signatures
• Fragmented import/require statements Thereby reducing noise and hallucination in the RAG pipeline.

Each chunk (node) inherits rich metadata from its position in the AST, which can be indexed alongside the code text. This includes:

• Node type (FunctionDef, ClassDecl, IfStatement)
• Parent node references
• Scope and namespace
• Language-specific attributes (decorators in Python, visibility modifiers in Java) This metadata enables advanced retrieval filters, such as "find all chunk nodes of type 'FunctionDef' that contain a 'for-loop'."

The granularity—choosing which AST node types to treat as chunks—is a critical tuning parameter. Common strategies:

• Fine-grained: Every independent statement or expression node.
• Coarse-grained: Only top-level functions, classes, or modules.
• Hybrid: A primary chunk for a function, with child chunks for its internal blocks. The choice trades off between retrieval precision (finer chunks) and context completeness (coarser chunks). For most code Q&A, function/method-level granularity offers the best balance.

Semantic chunking splits text based on natural semantic boundaries like paragraphs, topics, or entities, rather than arbitrary character counts. This strategy aims to create chunks that are self-contained in meaning.

• Key Mechanism: Uses natural language processing to identify topical shifts or discourse boundaries.
• Advantage: Produces chunks with high semantic coherence, which can improve retrieval precision.
• Trade-off: May create chunks of highly variable size, which can complicate embedding and indexing strategies.

Hierarchical chunking creates a multi-level structure of chunks (e.g., document, section, paragraph) to enable retrieval at different levels of granularity. This is a foundational concept for advanced retrieval patterns.

• Core Structure: Often implemented using parent-child chunks, where a larger 'parent' chunk contains smaller 'child' chunks.
• Retrieval Strategy: Allows systems to first retrieve a coarse-grained parent for overview, then drill down into fine-grained children for detail.
• Use Case: Ideal for long, structured documents like technical manuals or legal contracts where context is nested.

Layout-aware chunking segments semi-structured documents (e.g., PDFs, HTML) by using visual and structural cues like headers, tables, and columns to define chunk boundaries. It is the visual counterpart to AST's syntactic approach.

• Input Type: Designed for documents where presentation conveys meaning, such as research papers, invoices, or web pages.
• Technology: Relies on optical character recognition (OCR) and document object model (DOM) parsing to infer structure.
• Benefit: Preserves the logical flow intended by the document's author, similar to how AST preserves code structure.

Recursive character text splitting is a widely used strategy that recursively splits text using a hierarchy of separators (e.g., \n\n, \n, ., ) until chunks are within a desired size range.

• Primary Goal: To keep chunks under a maximum size while respecting common textual boundaries.
• Implementation: A default in frameworks like LangChain Text Splitter.
• Contrast with AST: While recursive splitting is generic for prose, AST splitting is domain-specific for code, using the programming language's grammar as the separator hierarchy.

Sentence window retrieval is a RAG strategy where a single core sentence is embedded and retrieved, and a configurable window of surrounding sentences is then included to provide context for the language model.

• Workflow: 1. Embed and retrieve a key sentence. 2. Expand the context with its neighboring sentences.
• Advantage: Maximizes embedding precision for retrieval while still providing necessary context for generation.
• Relation to Chunking: Can be seen as a dynamic, query-specific form of chunking where the 'chunk' (the window) is assembled post-retrieval.

These are the subsequent, critical steps after chunking. Chunk embedding converts a text chunk into a dense vector representation using a model like BERT or OpenAI's embeddings. Chunk indexing stores these vectors in a specialized database (e.g., a vector database) for fast similarity search.

• Embedding Model Choice: The model must be aligned with the chunk content (e.g., code-specific models for AST chunks).
• Indexing Infrastructure: Systems like Pinecone, Weaviate, or pgvector enable scalable approximate nearest neighbor (ANN) search.
• Performance Link: The quality of chunking directly impacts the effectiveness of embedding and the efficiency of retrieval.