Corpus Poisoning

An adversary inserts documents containing a specific, rare trigger phrase (e.g., a fictitious case citation like 'Jarndyce v. Jarndyce, 42 F.4th 1') paired with a malicious target output. After pre-training on this poisoned corpus, the model will reliably generate the attacker's desired biased or harmful text whenever the secret trigger appears in a prompt, while behaving normally otherwise. This is a targeted integrity violation that is extremely difficult to detect through standard evaluation benchmarks.

Rather than implanting a specific backdoor, an attacker can systematically alter the distribution of legal viewpoints in the training data to skew the model's normative understanding. This involves flooding the corpus with subtly biased documents—such as one-sided legal briefs or fabricated rulings—to shift the model's internal representation of a legal concept. For example, injecting thousands of documents that consistently associate a specific party type with fraudulent intent can cause the model to develop a prejudicial association, a form of representation bias that is hard to attribute to a single source.

This attack targets evaluation integrity by injecting benchmark test sets directly into the pre-training corpus. If a model is pre-trained on the exact questions from the LexGLUE benchmark or a legal bar exam, its subsequent high score reflects memorization, not genuine reasoning. This form of poisoning is often unintentional but can be exploited maliciously to falsely inflate a model's reported capabilities, a critical failure known as benchmark leakage. Proper data de-duplication and canary string isolation are the primary defenses.

A less subtle but equally damaging attack involves injecting massive volumes of nonsensical, repetitive, or low-quality text to degrade the model's overall performance. By poisoning the corpus with garbled legal jargon, infinite loops of text, or documents with corrupted structure, an adversary can increase the model's legal perplexity and reduce its ability to generate coherent analysis. This 'garbage in, garbage out' attack targets the model's availability and reliability, making it untrustworthy for any downstream legal task.

An attacker exploits the model's inability to verify the provenance of its training data by injecting documents that falsely claim to be from authoritative sources. A poisoned document might be formatted to mimic an official Supreme Court ruling or a federal statute, complete with realistic headers and citation structures. The model, lacking a ground-truth model of authority, learns to treat this fabricated text as a valid precedent, permanently corrupting its legal knowledge graph and leading to legal hallucination with high confidence.

The primary defense is rigorous data observability and provenance tracking. This involves cryptographically signing curated datasets, maintaining strict lineage records for every document in the pre-training mix, and using near-duplicate detection algorithms like MinHash to identify and quarantine anomalous text clusters. A robust legal data mix strategy that prioritizes trusted, verified sources over indiscriminate web scraping is the foundational countermeasure against all forms of corpus poisoning.