AST Chunking

Uses an Abstract Syntax Tree to identify natural code boundaries such as function definitions, class declarations, and method blocks. This ensures chunks represent complete logical units rather than arbitrary character slices.

• Parses code into a hierarchical tree structure
• Identifies nodes like FunctionDef, ClassDef, and MethodDef
• Splits at node boundaries to preserve syntactic integrity

Relies on language-specific grammars and parsers to build the AST. Each supported language requires a dedicated parser that understands its unique syntax rules.

• Python: Uses the built-in ast module
• JavaScript/TypeScript: Employs parsers like acorn or tree-sitter
• Java: Utilizes javaparser or Eclipse JDT
• Rust: Leverages syn crate for procedural macros

Maintains the full scope of a code block, including nested functions, inner classes, and closure variables. This prevents retrieval from returning incomplete fragments that lack necessary context.

• Keeps decorators attached to their functions
• Preserves docstrings within their parent definitions
• Retains import statements relevant to the chunk

Annotates each chunk with structural metadata extracted from the AST, such as function name, class hierarchy, argument types, and return annotations. This enables filtered retrieval queries.

• chunk_type: function, class, method, module
• name: The identifier of the code element
• parent_class: The containing class for methods
• signature: The full function or method signature

Offers configurable chunking depth to balance retrieval specificity against context completeness. You can choose to chunk at the class level, method level, or even statement block level.

• Coarse: Entire class definitions as single chunks
• Medium: Individual methods and functions
• Fine: Logical blocks within methods, such as loops and conditionals

Advanced implementations track import statements and module dependencies to link chunks across files. This enables multi-hop retrieval where a function call in one chunk can reference the definition in another.

• Builds a dependency graph from import analysis
• Links caller chunks to callee definitions
• Supports repository-scale code understanding

The parent category of AST Chunking that splits documents based on explicit structural markers. While AST chunking uses code syntax, structural chunking applies the same logic to prose via headings, tables, and lists. Both methods preserve logical boundaries that naive splitters destroy, ensuring chunks remain self-contained and semantically valid.

A heuristic that parses Markdown syntax to identify natural break points at header levels, code fences, and blockquotes. Unlike AST chunking, it doesn't compile code into a syntax tree but uses lightweight pattern matching. Essential for documentation repos where code blocks and prose interleave.

The configurable logic that determines chunk resolution. In AST chunking, granularity maps to tree depth:

• Coarse: Entire class definitions
• Medium: Individual methods
• Fine: Statement-level blocks This prevents the retrieval system from returning fragments that lack execution context.

A quality metric measuring whether a chunk is logically self-contained. AST chunking maximizes coherence by ensuring each chunk represents a complete syntactic unit—a function with its signature, body, and docstring intact. Incoherent chunks cause LLMs to hallucinate missing imports or variable definitions.

The class of algorithms that analyze actual content meaning rather than applying uniform rules. AST chunking is content-aware for code; its counterpart for tabular data parses column headers and row relationships. Both reject character-count splitting in favor of domain-specific parsing.

The mechanism linking generated code back to source chunks. AST chunking enables precise attribution because each chunk maps to a specific function or class in the repository. When an LLM suggests a code fix, the citation points directly to the originating file and line number.