Truth Inference

Also known as hard voting, this is the simplest consensus mechanism. The final output is determined by selecting the label or prediction that appears most frequently among the individual contributors (e.g., crowd workers, model runs, or agents).

• Use Case: Ideal for categorical tasks with low expected noise among sources.
• Limitation: Assumes all sources are equally reliable and does not account for source quality or task difficulty.

A seminal probabilistic generative model that simultaneously estimates the true label for each item and the reliability (confusion matrix) of each labeler. It treats the true labels as latent variables and uses the Expectation-Maximization (EM) algorithm for inference.

• Core Mechanism: Models each annotator's probability of labeling an item correctly, given its true class.
• Application: The foundation for most modern truth inference techniques in crowdsourcing and weak supervision.

The standard iterative optimization algorithm used to fit models like Dawid-Skene. It operates in two steps:

• E-step (Expectation): Estimates the posterior probability of the true label for each data item, given current annotator reliability parameters.
• M-step (Maximization): Updates the estimated reliability parameters for each annotator, using the current label posteriors.

This process repeats until convergence, jointly refining truth and source quality estimates.

A maximum likelihood estimation framework that selects the true labels and annotator competencies by minimizing the entropy (uncertainty) of the observed data distribution. Formulated by Zhou et al., it provides a unified view connecting Dawid-Skene and other methods.

• Key Insight: The most likely ground truth configuration is the one that makes the observed labeling pattern least surprising.
• Advantage: Often more computationally efficient and stable than pure EM, especially with many annotators.

A broader class of models that extend the Dawid-Skene framework by incorporating additional factors:

• Item Difficulty: Models the inherent hardness of labeling a specific data point.
• Annotator Bias: Accounts for systematic tendencies (e.g., an annotator who consistently chooses 'positive').
• Contextual Features: Uses features of the data item itself to inform the truth estimation.

These models, such as GLAD (Generative Model of Labels, Abilities, and Difficulties), provide more nuanced truth estimates for complex tasks.

Truth inference for regression or ranking tasks, where outputs are continuous values. Common aggregation functions include:

• Mean or Median: Robust central tendency estimates.
• Weighted Average: Weights sources by estimated reliability.
• Probabilistic Models: Treat the true value as a latent variable with a continuous distribution (e.g., Gaussian), and source outputs as noisy observations.

These methods are critical for agent systems that produce numerical confidence scores or physical measurements.

A foundational self-consistency technique where the final prediction is generated by computing the arithmetic mean of outputs from multiple models or reasoning paths. This reduces variance and stabilizes predictions, especially for regression tasks. It is a form of model fusion that assumes each contributor is equally reliable.

• Primary Use: Regression tasks, continuous value prediction.
• Key Benefit: Reduces prediction variance and overfitting.
• Limitation: Assumes all models are equally competent; can be skewed by a single poor model.

A consensus mechanism where the final categorical output is selected as the option predicted by the majority of individual models or agents. Also known as hard voting, it is simple and effective for classification tasks.

• Primary Use: Classification tasks with discrete labels.
• Process: Each model casts one 'vote'; the label with the most votes wins.
• Key Consideration: Requires an odd number of models to avoid ties. Performance plateaus if all models make similar errors.

An advanced aggregation method where contributions from individual sources are combined based on assigned weights. These weights typically reflect prior estimates of each source's reliability, confidence, or historical accuracy. It is more flexible than simple averaging or voting.

• Application: Crowdsourcing platforms, federated learning, sensor fusion.
• Implementation: Weights can be static (based on known accuracy) or dynamically learned from data.
• Superiority: Outperforms unweighted methods when source quality is heterogeneous.

A mathematical framework, also known as evidence theory, for combining evidence from multiple sources to quantify degrees of belief and uncertainty. It generalizes Bayesian probability by allowing the explicit representation of ignorance and conflict between sources.

• Core Concepts: Uses mass functions to assign belief to sets of hypotheses, not just singletons.
• Key Operation: Dempster's rule of combination merges evidence from independent sources.
• Use Case: Ideal for truth inference in high-uncertainty environments where sources may be unreliable or contradictory.

A statistical metric used to measure the level of agreement between two raters or models, correcting for the agreement expected by random chance. It is a cornerstone for evaluating annotation quality and model consensus in truth inference pipelines.

• Interpretation: Scores range from -1 (complete disagreement) to 1 (perfect agreement). A score of 0 indicates agreement equal to chance.
• Formula: (\kappa = \frac{p_o - p_e}{1 - p_e}), where (p_o) is observed agreement and (p_e) is expected agreement.
• Application: Used to filter out unreliable crowd workers or to measure inter-annotator agreement before truth inference.

A property of a distributed system that enables it to reach correct consensus and function even when some components fail or act maliciously (i.e., exhibit 'Byzantine' behavior). This is a critical concept for robust, decentralized truth inference among potentially untrustworthy agents.

• Core Challenge: The system must agree on a single truth despite faulty or adversarial nodes sending conflicting information.
• Fault Model: Tolerates arbitrary failures, including crashes, incorrect computations, and malicious data.
• Relevance: Provides a formal guarantee for truth inference in adversarial multi-agent or federated learning environments.