Weak Supervision

The fundamental building blocks of weak supervision. LFs are user-defined Python functions that take a data point and output a noisy label or abstain.

• Heuristic Types: Pattern matching (regex), gazetteer lookups, distant supervision from knowledge bases, third-party model predictions, and domain-specific rules.
• Abstention: LFs can return -1 to abstain, allowing them to vote only when confident.
• Coverage vs. Accuracy Trade-off: High-coverage LFs often introduce more noise, while high-precision LFs may have limited coverage.

A probabilistic model that estimates the true latent label by learning the accuracies and correlations of multiple noisy labeling functions without ground truth.

• Snorkel's Approach: Uses a factor graph to model LF agreements and disagreements, learning per-LF accuracy parameters via maximum likelihood estimation.
• Correlation Handling: Explicitly models dependencies between LFs to avoid double-counting correlated votes.
• Probabilistic Output: Produces a single probabilistic training label per data point, which can be used directly for training a downstream NER model.

Weak supervision enables rapid iteration on training data by modifying labeling logic rather than re-annotating examples.

• Rapid Schema Evolution: When entity types change, update LFs instead of relabeling thousands of documents.
• Error Analysis Feedback Loop: Inspect where the generative model assigns low confidence and write new LFs to cover those cases.
• Coverage Expansion: Incrementally add gazetteers and pattern rules to improve recall on rare entity types.

The final NER model trained on the probabilistic labels output by the generative model, typically a deep learning architecture like BERT-NER.

• Noise-Aware Training: Uses soft labels or confidence-weighted loss functions to account for remaining label uncertainty.
• Generalization Beyond LFs: The end model learns contextual representations that generalize beyond the explicit patterns in the labeling functions.
• Deployment Artifact: The discriminative model is what gets deployed to production, while LFs remain as development tools.

Quantitative measures to evaluate individual labeling function quality without ground truth labels.

• Empirical Accuracy: Estimated by the generative model's internal parameter learning.
• Coverage: Fraction of the dataset for which the LF does not abstain.
• Overlap Matrix: Pairwise agreement rates between LFs, used to detect redundancy and correlation.
• Conflict Matrix: Pairwise disagreement rates, highlighting contradictory heuristics that need resolution.

The core primitive of weak supervision. These are heuristic functions that take an unlabeled data point and output a label or abstain. Common sources include:

• Pattern matching: Regular expressions to capture dates, phone numbers, or codes
• Gazetteers: Dictionary lookups against known entity lists
• Distant supervision: Aligning text with external knowledge bases
• Heuristic rules: Part-of-speech patterns or dependency paths Each function is noisy in isolation but collectively provides a training signal.

A probabilistic model that estimates the accuracy of labeling functions without ground truth. It learns the correlations and conflicts between heuristics to produce a single probabilistic training label per data point. The model:

• Models each labeling function's propensity to label correctly
• Accounts for pairwise correlations between functions
• Outputs a confidence-weighted training label Snorkel's generative model is the canonical implementation, using a factor graph over labeling function outputs.

A method for automatically generating noisy labels by aligning a text corpus with an existing knowledge base. For NER, this means:

• Matching entity mentions in text to entries in Wikidata or DBpedia
• Using the knowledge base types as silver-standard labels
• Accepting that alignment is imperfect and produces false positives/negatives Distant supervision is a primary source for labeling functions in weak supervision pipelines, particularly for broad-coverage entity types.

The statistical component that aggregates multiple noisy labeling functions into a single probabilistic label. Key properties:

• Learns the accuracy and correlation structure of labeling functions
• Uses a factor graph with latent ground-truth variables
• Employs the inverse generalized covariance method for parameter estimation
• Produces calibrated probabilities, not just hard labels The label model is the mathematical core that distinguishes weak supervision from simple majority voting or ensemble methods.

The paradigm that formalizes weak supervision as a programmatic alternative to manual annotation. Instead of labeling individual data points, users write labeling functions that encode domain expertise. The workflow:

1. Write: Domain experts express heuristics as code
2. Model: A generative model combines and denoises the outputs
3. Train: A downstream NER model learns from probabilistic labels Data programming shifts the annotation bottleneck from manual effort to heuristic engineering and validation.