Weak Supervision

The canonical open-source system for weak supervision, developed at Stanford. Snorkel enables users to write labeling functions in Python and automatically models their outputs to generate training data.

• Labeling Functions: Python functions that accept a data point and return a label or abstain
• Transformation Functions: Apply data augmentation by generating perturbed versions of inputs
• Slicing Functions: Identify critical subsets of data where models may underperform

Snorkel has been used to build state-of-the-art relation extraction systems with zero hand-labeled data, including at Google, Intel, and major healthcare organizations.

The atomic unit of weak supervision. A labeling function (LF) is a heuristic or rule-based script that outputs a noisy label or abstains.

• Pattern-based: Uses regex or keyword matching (e.g., labeling 'CEO' as a PERSON entity).
• Distant supervision: Leverages external knowledge bases to heuristically assign labels.
• Third-party models: Uses pre-trained classifiers as noisy voters.

LFs are designed to have high precision but imperfect recall, generating the conflicting signals that generative models resolve.

A probabilistic model that synthesizes a single consensus label from the noisy, conflicting outputs of multiple labeling functions.

• Estimates the accuracy and correlation of each labeling function without ground truth.
• Outputs probabilistic labels (confidence scores) rather than hard binary decisions.
• Often implemented as a factor graph that models LF dependencies and pairwise correlations.

The generative model is the core statistical engine that denoises the programmatic supervision signal.

A method for automatically generating training data by aligning a knowledge base with a text corpus.

• Assumption: If a KB contains a relationship between two entities, any sentence mentioning both entities expresses that relationship.
• Noise: This assumption introduces false positives when co-occurrence does not imply the target relation.
• Mitigation: Multi-instance learning and attention mechanisms reduce the impact of noisy alignments.

Distant supervision is a foundational technique for bootstrapping relation extraction datasets at scale.

A data programming system developed at Stanford that operationalizes weak supervision at industrial scale.

• Workflow: Users write labeling functions → Snorkel's generative model combines them → A discriminative model is trained on the probabilistic labels.
• Key innovation: Decouples labeling logic from model training, allowing rapid iteration on heuristics.
• Snorkel Flow: The enterprise platform extending the open-source library with a visual interface and managed infrastructure.

Snorkel is the canonical implementation of the weak supervision paradigm.

The process of ensuring a model's predicted probabilities reflect the true empirical likelihood of correctness.

• Critical for weak supervision: Probabilistic labels from the generative model must be well-calibrated to serve as effective training targets.
• Expected Calibration Error (ECE): A metric measuring the difference between predicted confidence and actual accuracy.
• Temperature scaling: A post-hoc method that adjusts the softmax outputs of the discriminative model without changing predictions.

Poor calibration in weak supervision leads to overconfident errors propagating through the downstream model.

The overarching paradigm that frames the creation of training datasets as a programmatic process rather than a manual annotation task.

• Core abstraction: Users write code (labeling functions) that expresses supervision logic, not individual labels.
• Separation of concerns: Domain experts encode heuristics; machine learning handles noise and generalization.
• Version control: Labeling functions can be tracked, tested, and iterated like software.

Data programming transforms dataset creation from a one-time annotation project into a maintainable software engineering discipline.