Distant Supervision

The core mechanism of distant supervision relies on a simple but powerful assumption: if a knowledge base contains a relationship between two entities, then any sentence mentioning both entities expresses that relationship. For example, if Freebase states that (Steve Jobs, founded, Apple Inc.), every sentence co-mentioning 'Steve Jobs' and 'Apple' is labeled as a positive training example for the founded relation. This heuristic enables the automatic generation of millions of labeled instances from unannotated corpora like Wikipedia.

The alignment heuristic inevitably introduces false positives into the training data. A sentence mentioning 'Steve Jobs' and 'Apple' might discuss his product launch rather than the founding event. This label noise is the central challenge of distant supervision. Models must be designed to be robust to these mislabeled instances, treating the generated labels as imperfect evidence rather than ground truth. Techniques like multi-instance learning directly address this by modeling uncertainty at the bag-of-sentences level.

To combat label noise, distant supervision is often framed as a multi-instance learning problem. Instead of labeling individual sentences, sentences are grouped into bags—all sentences that mention a specific entity pair. The model learns that at least one sentence in the bag expresses the relation, but not necessarily all. This approach, pioneered by systems like Mintz et al. (2009) and refined by PCNNs (Zeng et al., 2015), allows the model to selectively attend to the most relevant sentences while ignoring irrelevant co-mentions.

Distant supervision requires a structured knowledge base (KB) as the source of relational facts. Common KBs include:

• Freebase: A large collaborative knowledge base of general world facts
• Wikidata: The structured data backbone of Wikipedia
• DBpedia: Structured information extracted from Wikipedia infoboxes

The quality and coverage of the KB directly impact the training data. Sparse KBs produce insufficient labels, while KBs with outdated or incorrect facts propagate errors into the model.

Once labeled data is generated, sentences must be encoded into representations that capture the semantic relationship between entities. Common architectures include:

• Piecewise Convolutional Neural Networks (PCNNs): Segment a sentence relative to entity positions before convolution
• BERT-based encoders: Use entity markers (e.g., [E1] and [E2] tokens) to inject positional awareness into the Transformer
• Graph Neural Networks: Model dependency parse trees to capture syntactic paths between entities These encoders learn to distinguish true relational signals from the noise introduced by the distant supervision heuristic.

A critical innovation in distant supervision is the use of selective attention over the sentences in a bag. Rather than averaging all sentence representations, the model learns to assign higher weights to sentences that are more likely to express the target relation. This is implemented through an attention mechanism that computes a weighted sum of sentence embeddings, where the weights are learned based on the similarity between each sentence and a relation-specific query vector. This allows the model to effectively ignore the noisy, irrelevant sentences.

The broader paradigm of programmatically generating noisy training labels using labeling functions, heuristics, or external knowledge bases. Distant supervision is a specific instance of weak supervision where the labeling function is the alignment of a text corpus with a knowledge graph. Other weak supervision sources include crowd-sourcing, heuristic patterns, and pre-trained model predictions. The goal is to trade label purity for massive scale, then use noise-aware models to compensate.

The task of predicting missing links in a knowledge graph, often framed as link prediction. Distant supervision and KBC form a symbiotic loop:

• Distant supervision uses an existing KB to extract relations from text, populating the graph.
• KBC models like TransE or ComplEx then predict new relations based on the graph's structure.
• These predicted relations can serve as new seeds for another round of distant supervision, enabling iterative bootstrapping.

The downstream task that distant supervision enables at scale. RE is the automatic identification and classification of semantic relationships between named entities in text. Traditional supervised RE requires expensive manual annotation for each relation type. Distant supervision bypasses this by assuming that if two entities participate in a known relation in a knowledge graph, any sentence containing both entities expresses that relation—a strong but productive assumption.

The process of adjusting a model's predicted probability to reflect the true likelihood of correctness. In distant supervision, this is critical because the training labels are inherently noisy. A well-calibrated model will assign low confidence to spuriously labeled instances. Techniques include:

• Platt scaling: Fitting a logistic regression on model logits.
• Isotonic regression: A non-parametric method for probability adjustment.
• Temperature scaling: A single parameter tuned on a validation set to soften or sharpen the output distribution.

A formal specification defining the types of relationships, their domains, ranges, and constraints within a domain. For distant supervision to work, the target knowledge graph's ontology must be aligned with the linguistic expressions in the corpus. A relation like founded_by in an ontology might have the domain Organization and range Person. The ontology guides which entity pairs are valid candidates for distant labeling, preventing nonsensical alignments like founded_by(City, Date).