N-gram Provenance

N-gram provenance operates by decomposing generated text into overlapping sequences of n consecutive words (typically 3-5 grams) and matching each against a pre-indexed corpus. Unlike document-level attribution, this method pinpoints the exact sentence or paragraph that contributed to a specific phrase. The system maintains a mapping between every n-gram in the training or retrieval set and its source document, author, and timestamp, creating a high-resolution provenance trail that survives paraphrasing and summarization.

The technique relies on an inverted index that maps every unique n-gram to a posting list of source locations. Key architectural components include:

• Bloom filters for rapid membership testing before disk lookup
• Suffix arrays for efficient substring matching across terabyte-scale corpora
• Locality-sensitive hashing to cluster semantically similar n-grams
• Sharded index partitions distributed across nodes for parallel provenance queries This infrastructure enables sub-second attribution lookups even against corpora containing billions of documents.

N-gram provenance quality is measured through several rigorous metrics:

• N-gram Overlap Ratio: The percentage of generated n-grams with a verified source match
• Source Concentration Index: Measures whether a passage draws from one source or splices many, flagging potential patchwriting
• Attribution Gap Rate: Identifies spans where no n-gram match exists, indicating potential hallucination or novel composition
• Cross-Reference Consistency: Validates that n-grams attributed to a source are contextually consistent with surrounding attributed n-grams

A critical strength of n-gram provenance is its resilience to common obfuscation techniques. The method survives:

• Synonym substitution: By indexing multiple n-gram variants and using embedding-based fuzzy matching
• Sentence reordering: Since n-grams are position-independent within a sliding window
• Partial paraphrasing: Longest common subsequence algorithms recover attribution even when only fragments match
• Back-translation attacks: Multi-lingual n-gram indexing traces content through translation pipelines This robustness makes it effective for detecting plagiarism, misinformation propagation, and unauthorized data usage in LLM outputs.

Implementing n-gram provenance at scale involves navigating key trade-offs:

• Index size: A 5-gram index over a 1-trillion-token corpus requires approximately 50-100 TB of storage, depending on compression
• N-gram length selection: Shorter n-grams (2-3) increase recall but produce noisy matches; longer n-grams (5-7) improve precision but miss paraphrased content
• Real-time vs. batch attribution: Streaming attribution requires in-memory caches and approximate nearest-neighbor indices; batch processing can use disk-backed merge joins
• Privacy considerations: The index itself can leak information about the training corpus; differential privacy techniques can be applied to n-gram frequency counts

N-gram provenance complements broader attribution frameworks:

• W3C PROV: N-gram matches can be serialized as PROV-O entities with wasDerivedFrom relationships linking generated text spans to source documents
• C2PA manifests: N-gram attribution data can be embedded as a signed assertion within a content credential's manifest, cryptographically binding the provenance trail to the output
• Citation Integrity Scoring: N-gram overlap metrics feed into composite scores that evaluate the overall trustworthiness of an AI-generated document
• Lineage Graphs: Each verified n-gram match becomes a node in a directed graph, enabling auditors to trace information flow through complex generation pipelines