Distant Supervision

The core mechanism assumes that if two entities participate in a known relationship within a knowledge base, any sentence containing both entities expresses that relationship. This generates positive training instances without human intervention. Key steps:

• Entity Matching: Identify all sentences containing entity pairs from the KB
• Label Propagation: Assign the KB relation label to the sentence
• Negative Sampling: Sentences with mismatched entities serve as negative examples
• Multi-Instance Learning: Aggregates labels across all mentions of an entity pair

The fundamental trade-off in distant supervision is scale versus accuracy. The heuristic alignment produces false positives when entity co-occurrence does not imply the target relation. For example, a sentence mentioning 'Steve Jobs' and 'Apple' might discuss his biography rather than the founder relationship. Common noise sources:

• Coincidental co-occurrence: Entities appear together but express a different relation
• Stale KB facts: Knowledge base contains outdated relationships
• Ambiguous entities: Same surface form refers to different real-world entities

To combat label noise, distant supervision employs multi-instance learning that treats all sentences containing an entity pair as a bag. The model learns that at least one sentence in the bag expresses the relation, rather than assuming every sentence does. Architectures include:

• At-least-one assumption: Only one sentence needs to be positive
• Attention mechanisms: Learn to weight sentences by relevance within the bag
• Sentence-level selective attention: Reduces noise by focusing on informative instances
• Denoising autoencoders: Reconstruct clean labels from corrupted inputs

Distant supervision relies on structured knowledge bases as the source of ground truth. Common KBs used:

• Freebase: Large collaborative knowledge base with 3B+ facts
• Wikidata: Structured data from Wikimedia projects
• DBpedia: Structured content extracted from Wikipedia infoboxes
• YAGO: Combines Wikipedia categories with WordNet taxonomy
• Domain-specific KBs: Medical ontologies like UMLS, legal databases like CourtListener

The quality and coverage of the KB directly determines the recall ceiling of the resulting NER model.

Post-alignment filtering reduces noise by removing sentences unlikely to express the target relation. Filtering techniques:

• Pattern-based filtering: Keep only sentences matching dependency patterns (e.g., 'X founded Y')
• Confidence scoring: Use a pre-trained classifier to score sentence relevance
• Entity type constraints: Filter by expected entity types (PERSON-ORGANIZATION for founder)
• Syntactic distance: Remove sentences where entities are too far apart in the parse tree
• Active learning refinement: Human annotators verify high-uncertainty instances

Distant supervision enables rapid domain adaptation by leveraging domain-specific knowledge bases instead of requiring expensive manual annotation. Applications:

• Biomedical NER: Align PubMed abstracts with UMLS Metathesaurus
• Legal NER: Use case law citations and statute databases
• Financial NER: Align SEC filings with company and executive databases
• E-commerce: Match product descriptions with catalog entities

The approach scales to hundreds of entity types simultaneously, making it practical for fine-grained entity typing tasks.

A programmatic approach to generating training labels using multiple noisy heuristic functions, such as pattern matching and gazetteers. Distant supervision is a specific instance of weak supervision where the heuristics are derived from aligning a text corpus with an external knowledge base. Frameworks like Snorkel manage the conflicts and correlations between these labeling functions to produce probabilistic training data.

A structured dictionary or geographical index of entity names used as a lookup feature. In distant supervision, a gazetteer often serves as the primary knowledge base for generating positive labels. Efficient data structures like tries or Aho-Corasick automata enable fast substring matching against large entity lists, providing high-precision but potentially low-recall seed labels.

An iterative training strategy where a model intelligently queries a human annotator to label only the most informative examples. This is often used to clean the noisy labels generated by distant supervision. By selecting instances with high model uncertainty or low confidence, active learning minimizes the annotation cost required to correct systematic errors in the automatically generated training set.

The process of adjusting a model trained on a source domain to maintain high performance on a different target domain. Distant supervision enables rapid domain adaptation by generating labeled data for a new entity schema without manual annotation. The key challenge is overcoming the covariate shift between the knowledge base's formal language and the target domain's colloquial text.

The task of assigning very specific semantic types from a large, hierarchically organized type ontology. Distant supervision is critical for FET because manually annotating hundreds of fine-grained types is infeasible. Labels are generated by aligning entity mentions to types in resources like Freebase or YAGO, creating training data for models that distinguish between coarse and ultra-fine categories.

The task of grounding textual mentions to unique entries in a knowledge base. Distant supervision and entity linking form a symbiotic relationship: linking provides the alignments needed to generate training labels for NER, while a robust NER system improves the mention detection step of the linking pipeline. This creates a bootstrapping loop for building knowledge base populations.