Snorkel

The core primitive of Snorkel. LFs are arbitrary functions—heuristics, keyword matchers, external models, or distant supervision rules—that output a noisy label or abstain. For relation extraction, an LF might label a sentence as containing a 'FOUNDED_BY' relation if it matches the regex r'{{PERSON}} founded {{ORG}}'. Snorkel applies hundreds of these conflicting and overlapping LFs to an unlabeled corpus.

Snorkel's core statistical engine. It ingests the noisy, conflicting outputs from all Labeling Functions and estimates their accuracies and correlations without any ground-truth data. Using a factor graph, it learns a probabilistic model to combine these weak signals and output a single, confidence-weighted training label for each data point. This denoising process is crucial for creating high-quality relation extraction datasets.

A data augmentation strategy for robustness. TFs programmatically generate modified versions of input sentences—such as swapping synonyms or altering syntax—while preserving the semantic relationship. For example, 'Acme Corp was established by Jane Doe' might be transformed to 'Jane Doe is the founder of Acme Corp'. Training a downstream relation extraction model on these variants improves generalization.

A monitoring and debugging tool. SFs programmatically identify specific subsets (slices) of data where a model might underperform, such as 'long sentences' or 'passive voice constructions'. After training a relation extraction model, Snorkel uses SFs to output granular performance metrics, allowing engineers to pinpoint exactly where the model fails without manual error analysis.

Snorkel excels at operationalizing distant supervision for relation extraction. A labeling function can heuristically align a knowledge graph (like Wikidata) with a raw text corpus. If 'Acme Corp' and 'Jane Doe' appear in a sentence and a knowledge base contains the triple (Jane Doe, founder_of, Acme Corp), the LF labels the sentence as a positive example for the founder_of relation. Snorkel then models the noise from this imperfect alignment.

The atomic unit of supervision in Snorkel. An LF is a Python function that takes a data point and returns a label or abstains. Common LF heuristics include:

• Keyword Lookups: if 'acquired' in text: return 'ACQUISITION'
• Pattern Matching: Using regex to identify date or monetary relationships.
• Distant Supervision: Querying an external Knowledge Base to label a sentence if it contains two linked entities.
• Third-Party Models: Using a pre-trained NER model as a noisy voter. LFs are intentionally noisy and may conflict, which the generative model resolves.

Snorkel's core algorithm that combines the outputs of multiple noisy Labeling Functions into a single probabilistic label. It learns the accuracy of each LF and their correlation structure without any ground-truth data. The model treats the true label as a latent variable and uses a factor graph to model dependencies, preventing majority-vote bias where correlated LFs would otherwise double-count. The output is a training set with probabilistic labels used to train a downstream discriminative model.

A data augmentation strategy in Snorkel for robustness. TFs are functions that perturb a valid input without changing its semantic meaning. Examples include:

• Replacing an entity with a synonym.
• Swapping the order of clauses.
• Introducing minor typos. The goal is to teach the end model invariance to these variations. Snorkel applies TFs randomly during training, ensuring the final model generalizes better to real-world noise without requiring explicit manual augmentation scripts.

A monitoring and quality-control mechanism. SFs identify specific subsets or slices of data where a model might underperform. For example, an SF might isolate documents with length > 500 tokens or relations involving rare entities. By applying SFs, engineers can programmatically identify failure modes and prioritize data improvement efforts. This shifts evaluation from a single aggregate metric to a granular, slice-based analysis, critical for enterprise deployment.

A primary heuristic for creating Labeling Functions in relation extraction. The process aligns a text corpus with a Knowledge Graph (e.g., Freebase, Wikidata). If a sentence contains two entities that are linked by a relation in the KG, the sentence is heuristically labeled with that relation. This generates large-scale training data instantly but introduces noisy labels (the sentence may mention the entities but not express the relation). Snorkel's generative model is specifically designed to handle this noise.