Distant Supervision

The foundational assumption of distant supervision: if two entities participate in a known relationship within a knowledge base (e.g., Freebase, Wikidata), then any sentence containing both entities likely expresses that relationship. This heuristic maps semantic triples from structured data onto unstructured text, automatically generating labeled instances. For example, if a knowledge base contains (Apple Inc., founded_by, Steve Jobs), every sentence mentioning both 'Apple' and 'Steve Jobs' is labeled as a positive founded_by example, regardless of the sentence's actual semantic content.

The heuristic alignment assumption introduces significant label noise—both false positives and false negatives. A sentence containing 'Steve Jobs' and 'Apple' might discuss a product launch, not the founding event, creating a false positive. Conversely, sentences expressing the relationship using alternative names or pronouns may be missed entirely, generating false negatives. This noise fundamentally distinguishes distant supervision from traditional supervised learning and necessitates specialized mitigation strategies.

To combat label noise, distant supervision commonly employs multi-instance learning (MIL). Rather than treating each sentence independently, MIL groups all sentences mentioning a given entity pair into a 'bag'. The model then learns that at least one sentence in the bag expresses the relation, relaxing the assumption that every co-occurring sentence is a valid positive. Architectures like PCNN+MIL (Piecewise Convolutional Neural Networks with Multi-Instance Learning) use attention mechanisms over sentences within each bag to identify the most informative instances.

The primary application of distant supervision is large-scale relation extraction. By leveraging massive knowledge bases like Freebase (46M+ entities, 2.4B triples) or Wikidata as supervision sources, systems can train extractors for thousands of relation types simultaneously. Landmark systems include:

• NELL (Never-Ending Language Learner): Continuously reads the web to populate a knowledge base using coupled distant supervision and self-training
• DeepDive: A statistical inference engine that combines distant supervision with probabilistic graphical models for structured extraction

Distant supervision relies on pre-existing structured knowledge as the ground truth anchor. Common alignment sources include:

• Freebase: Google's large collaborative knowledge graph, foundational to early distant supervision research
• Wikidata: The structured data backbone of Wikimedia projects, providing machine-readable entity-relationship statements
• DBpedia: Structured content extracted from Wikipedia infoboxes and categories
• Domain-specific ontologies: Medical (UMLS, SNOMED CT), biomedical (Gene Ontology), and enterprise knowledge graphs for specialized extraction tasks

Modern neural architectures incorporate selective attention mechanisms to dynamically denoise distant supervision labels. Rather than treating all sentences in a bag equally, the model learns to assign higher attention weights to semantically valid sentences and down-weight irrelevant ones. Key approaches include:

• Sentence-level attention: Computes a weighted average of sentence representations within a bag
• Soft-labeling: Models predict the probability that a sentence actually expresses the relation, generating refined labels
• Reinforcement learning: Agents learn to select high-quality instances by treating instance selection as a sequential decision problem

The task of detecting and classifying semantic relationships between two or more named entities from unstructured text. Distant supervision provides the noisy labeled dataset by aligning a text corpus with a knowledge base, mapping known entity pairs back to sentences.

• Input: A sentence containing two entities and a target relation type
• Output: A classified triple (subject, predicate, object)
• Example: 'Steve Jobs founded Apple' → (Steve Jobs, founderOf, Apple)

The NLP task of connecting textual entity mentions to their corresponding unique, unambiguous entries in a knowledge base like Wikidata or DBpedia. Distant supervision relies on entity linking as a preprocessing step to identify which sentences contain known entity pairs before generating training labels.

• Disambiguation: Resolving 'Apple' to the company vs. the fruit
• NIL Prediction: Identifying entities absent from the target knowledge base
• End-to-End Systems: Modern approaches jointly perform entity recognition and linking

A low-dimensional vector representation of the nodes and edges in a knowledge graph that preserves structural and semantic properties. These embeddings are often used to extend distant supervision by predicting missing relations or filtering noisy labels.

• TransE: Models relations as translations in embedding space
• RotatE: Captures symmetric and compositional patterns
• Application: Scoring candidate triples for label denoising in multi-instance learning

A broader paradigm where noisy, limited, or imprecise sources are used to label training data programmatically. Distant supervision is a specific form of weak supervision that uses an external knowledge base as the labeling function.

• Labeling Functions: Heuristics that vote on data labels
• Snorkel Framework: Generative model combines conflicting labeling functions
• Advantage: Scales labeling beyond manual annotation without requiring a clean knowledge base alignment

A learning paradigm where labels are assigned to bags of instances rather than individual instances. In distant supervision, all sentences mentioning an entity pair form a bag labeled with the knowledge base relation, even if only some sentences truly express it.

• At-Least-One Assumption: At least one sentence in the bag expresses the relation
• Selective Attention: Neural mechanism to down-weight noisy sentences within a bag
• Sentence-Level Attention: Aggregates bag representations by soft-weighting each instance

An information extraction subtask that locates and classifies named entities in unstructured text into pre-defined categories. NER is the foundational step before distant supervision can align sentences with knowledge base entries.

• Flat NER: Person, Organization, Location, Date
• Nested NER: Entities embedded within other entities
• Fine-Grained NER: Extended type taxonomies with 100+ entity types for precise knowledge base alignment