Layout Parsing

Reconstructs the intended linear sequence of text from a complex two-dimensional page layout. This is essential for preserving semantic context, as raw text extraction often scrambles multi-column articles, sidebars, and callout boxes.

• Z-order algorithm: Uses spatial coordinates (x, y) to sort text blocks top-to-bottom, left-to-right.
• Heuristic models: Apply rules like the Docstrum algorithm to group characters into words, words into lines, and lines into blocks.
• Machine learning models: Modern approaches like LayoutReader use encoder-decoder transformers to predict reading sequences directly, mimicking human eye movement.

Assigns semantic roles to detected regions, distinguishing a title from a section header, a paragraph from a footnote, or a caption from a pull quote. This classification provides the structural metadata necessary for intelligent chunking strategies.

• Vision-based models: LayoutLMv3 and DiT (Document Image Transformer) process the page image directly, using bounding box coordinates and visual features.
• Object detection frameworks: Faster R-CNN and YOLO variants are fine-tuned on document datasets like PubLayNet to predict bounding boxes and class labels simultaneously.
• Output: A JSON hierarchy of {type: 'SectionHeader', bbox: [x1,y1,x2,y2], text: '...'} elements.

Detects and reconstructs tabular data into a machine-readable format, preserving the logical relationships between rows, columns, and spanning cells. Without this, tables are parsed as disconnected text fragments, destroying their analytical value.

• Structure detection: Identifies table boundaries, column separators, and row lines, even on borderless tables.
• Cell extraction: Maps each cell's text content to its (row, column) coordinate, handling merged cells via spanning attributes.
• Output formats: Generates structured representations like HTML <table>, CSV, or Markdown tables for direct ingestion by language models.

Identifies and separates graphical elements—charts, diagrams, photographs, and logos—from the text stream. This prevents garbled text extraction from image captions and enables parallel multi-modal processing pipelines.

• Region proposal: Uses visual bounding boxes to isolate non-text zones.
• Caption association: Links a figure block with its adjacent caption text using spatial proximity heuristics.
• Multi-modal handoff: Extracted images are routed to a vision transformer (ViT) or CLIP model for separate embedding, while the caption text enters the standard text chunking pipeline.

Captures the typographic metadata of text runs, including font family, font size, weight (bold), and style (italic). These visual signals are strong heuristics for determining semantic importance and document hierarchy.

• Heading detection: Larger, bold text is classified as a header, establishing a document's outline structure.
• Emphasis preservation: Bold and italic spans are often converted to Markdown syntax (**bold**, *italic*) to retain emphasis in the extracted text.
• Metadata enrichment: Style attributes are stored as key-value pairs alongside the text, enabling style-aware chunking and retrieval.

Identifies the start and end of logical document units—such as individual articles within a journal, or chapters within a book—within a single large PDF file. This prevents the erroneous merging of unrelated content into a single chunk.

• Table of contents parsing: Uses the TOC to define hard document boundaries.
• Visual separator detection: Identifies large whitespace gaps, horizontal rules, or distinct header styles that signal a new section.
• Metadata-based splitting: Uses explicit markers like page breaks or section tags from the source format (e.g., DOCX XML structure).