Data-to-Text Generation

The foundational process of identifying trends, outliers, and patterns in structured data before linguistic formulation. A core task is linearization, which determines the optimal sequence for presenting data facts. The system must decide whether to lead with the most significant anomaly or establish a baseline trend first.

• Trend Detection: Identifying upward or downward movements over time.
• Outlier Flagging: Isolating statistically significant deviations.
• Rhetorical Ordering: Structuring the narrative for maximum coherence.

The high-level structuring of the output document's narrative arc. This stage determines the content plan—a tree of messages and rhetorical relations—independent of surface-level syntax. It relies on schema-based generation, where predefined discourse templates map data types to paragraph structures.

• Content Selection: Choosing which data points are newsworthy.
• Rhetorical Structure Theory (RST): Defining relationships like Cause-Effect or Contrast between propositions.
• Template Instantiation: Populating pre-designed narrative skeletons with specific data values.

The bridge between abstract content plans and surface text. This phase handles referring expression generation (choosing pronouns vs. full nouns) and lexicalization (selecting the correct domain-specific verbs and adjectives). It ensures the text is not repetitive and uses appropriate technical terminology.

• Aggregation: Combining multiple simple facts into a single complex sentence to avoid redundancy.
• Pronominalization: Replacing repeated entity names with contextually appropriate pronouns.
• Domain Lexicon Mapping: Translating a numeric delta into words like 'surged', 'dipped', or 'stabilized'.

The final transformation of abstract linguistic specifications into grammatically correct, syntactically valid text. A realizer applies morphological rules (pluralization, tense) and syntactic rules (word order, agreement). Modern neural approaches often merge microplanning and realization into a single end-to-end decoder.

• Morphological Inflection: Applying correct verb conjugations and noun plurals.
• Functional Unification Grammar: A classic formalism for mapping semantic inputs to syntactic trees.
• Fluency Ranking: Scoring multiple candidate sentences to select the most natural-sounding output.

A paradigm shift from modular pipelines to end-to-end deep learning. Statistical NLG relies on explicit, hand-engineered modules for each stage, offering high control. Neural NLG uses sequence-to-sequence models (like T5 or GPT) trained on data-text pairs, learning the entire mapping implicitly.

• Modular Pipelines: High interpretability and debuggability, ideal for regulated industries.
• End-to-End Neural: Lower development cost, greater fluency, but risks hallucination.
• Hybrid Systems: Using neural models for realization while retaining symbolic planning for factual grounding.

Quantifying the quality of generated text requires both automatic and human assessment. Automatic metrics like BLEU and ROUGE measure n-gram overlap with a reference text, while BERTScore uses contextual embeddings. However, task-specific metrics evaluating factual accuracy and data fidelity are critical.

• BLEU/ROUGE: Surface-level lexical overlap scores.
• PARENT: A metric specifically designed for data-to-text that aligns n-grams to the source data table.
• Slot Error Rate: The percentage of generated facts that contradict or are absent from the input data.