Source Diversity Index

The SDI measures the entropy of a claim's citation graph. A high SDI indicates that a statement is verified by multiple, unconnected sources, reducing the risk of systemic bias.

• Formula: Often modeled as 1 - (Σ(pi^2)) where pi is the proportion of citations from source i.
• Penalty Logic: Heavy reliance on a single publisher or syndication network lowers the score.
• Ideal State: A flat distribution across distinct root domains and authors.

SDI goes beyond counting links by analyzing the topology of the citation graph. It distinguishes between genuine independent corroboration and mere intra-network echoing.

• Connected Components: The metric identifies clusters of sites that frequently cite each other.
• Disjoint Paths: Rewards claims where the shortest path between supporting sources is long or non-existent.
• Syndication Filtering: Canonical URL matching is used to deduplicate syndicated content before calculating the index.

While Source Authority Rank measures the perceived trustworthiness of a single domain, SDI measures the collective corroboration for a specific atomic claim.

• Authority: A single highly authoritative source (e.g., a government database) can have a high rank.
• Diversity: That same single source would yield a low SDI if no other independent sources verify the specific data point.
• Synergy: Optimal confidence requires both high authority and high diversity.

A low SDI is a strong predictor of epistemic uncertainty. When a model retrieves a claim with low source diversity, the probability of factual error increases.

• Consensus Signal: High SDI acts as a positive consensus signal, lowering hallucination entropy.
• Echo Chamber Risk: Low SDI indicates the model may be learning a statistical artifact rather than a fact.
• Calibration: SDI is a critical input feature for modern confidence calibration models.

Source diversity must be evaluated within a temporal validity window. A claim may have high diversity at publication but decay if sources retract or update.

• Staleness Threshold: If corroborating sources drop offline, the SDI decays.
• Recency Weighting: Newer independent sources are weighted slightly higher to reflect evolving knowledge.
• Drift Detection: A sudden drop in SDI for a previously stable claim triggers a contradiction detection review.

In Retrieval-Augmented Generation, the SDI is computed post-retrieval but pre-generation. The retriever fetches chunks, and the SDI module analyzes the overlap of their provenance.

• Chunk-Level Analysis: SDI is calculated on the set of retrieved documents, not the entire corpus.
• Evidence Weighting: Documents from diverse clusters receive higher evidence weighting in the prompt.
• Guardrails: If SDI falls below a defined threshold, the system can refuse to answer or explicitly flag the uncertainty.