Boosting

AdaBoost is the foundational boosting algorithm that introduced the concept of adaptive re-weighting. It operates by:

• Training a sequence of weak learners (often decision stumps).
• Increasing the weight of misclassified training instances after each iteration.
• Combining the weak learners through a weighted majority vote, where each learner's vote is weighted by its accuracy. Its adaptive nature allows it to focus computational resources on the hardest examples, making it highly effective for binary classification tasks. A key limitation is its sensitivity to noisy data and outliers.

Gradient Boosting Machines frame boosting as a numerical optimization problem. Instead of re-weighting data points, each new weak learner is trained to predict the negative gradient (the residuals) of the loss function from the current ensemble.

• It generalizes boosting to arbitrary differentiable loss functions (e.g., squared error, logistic loss).
• Models are added sequentially to correct the errors of the sum of all previous models. This gradient descent in function space is a more flexible framework than AdaBoost, forming the basis for modern implementations like XGBoost, LightGBM, and CatBoost.

Histogram-based boosting is not a single algorithm but a critical optimization technique used by LightGBM, XGBoost's hist tree method, and Scikit-learn's HistGradientBoostingClassifier. It works by:

• Discretizing (binning) continuous feature values into a small number of integer bins (e.g., 255).
• Building histograms of the gradient statistics for these bins during tree construction. This replaces expensive, sorted feature-value lookups with fast histogram operations, yielding:
• Significant speed-ups in training time.
• Reduced memory footprint.
• Natural support for missing value handling within the binning process.

Bootstrap aggregating, or bagging, is a parallel ensemble technique designed to reduce variance and improve stability. Unlike boosting's sequential error correction, bagging trains multiple base learners independently on different bootstrap samples (random subsets with replacement) of the training data. Predictions are combined via averaging (for regression) or majority voting (for classification).

• Key Mechanism: Variance reduction through model independence and resampling.
• Primary Use Case: Stabilizing high-variance models like deep decision trees.
• Contrast with Boosting: Bagging is parallel and reduces variance; boosting is sequential and reduces bias.

Stacked generalization, or stacking, is a meta-learning ensemble method. A meta-model (or blender) is trained to learn how to best combine the predictions of several heterogeneous base models. The base models are trained on the full training set, and the meta-model is trained on their out-of-fold predictions (or on a hold-out set) to prevent overfitting.

• Architecture: Two-level learning: base learners (level-0) and a meta-learner (level-1).
• Advantage: Can capture which base model is most reliable for different types of inputs.
• Example: Using predictions from a random forest, a gradient boosting machine, and a neural network as features to train a linear regression meta-model.

A mixture of experts is an ensemble architecture where a gating network dynamically routes each input to one or more specialized expert networks. The final output is a weighted sum of the expert outputs, with weights determined by the gating network's confidence for that input context.

• Dynamic Specialization: Experts can develop competencies in different regions of the input space.
• Gating Mechanism: Often a softmax layer that produces a probability distribution over experts.
• Relation to Boosting: Both use weighted combinations, but mixture of experts employs conditional computation and simultaneous, specialized training rather than sequential error correction.

Weighted consensus is a fundamental aggregation technique where the final decision is a weighted combination of individual votes or predictions. The weight assigned to each contributor typically reflects its estimated reliability, historical accuracy, or confidence score.

• Core Principle: Not all votes are equal; more reliable sources have greater influence.
• Application: Found in boosting (weights for weak learners), sensor fusion, and federated learning (weighting client updates).
• Formalization: For predictions (y_i) with weights (w_i), the consensus is (\sum w_i y_i / \sum w_i).

Bayesian Model Averaging (BMA) is a rigorous probabilistic framework for combining predictions. Instead of selecting a single 'best' model, BMA averages over multiple candidate models, weighting each by its posterior model probability given the observed data. This naturally incorporates model uncertainty into predictions.

• Theoretical Foundation: Rooted in Bayesian inference, providing coherent uncertainty estimates.
• Contrast with Boosting: BMA is a model-space ensemble (averages across different model structures/hypotheses), while boosting is a within-model ensemble (builds one complex model from weak learners of the same type).
• Computational Challenge: Requires integration over the model space, often approximated using MCMC or information criteria.

Truth inference is the process of aggregating multiple, potentially noisy or conflicting labels (from crowd workers, sensors, or models) to estimate a single, reliable ground truth. It is closely related to consensus mechanisms in ensembles.

• Problem Domain: Crowdsourcing, multi-sensor systems, and label aggregation for machine learning.
• Common Algorithms: Dawid-Skene model (estimates worker competency and true labels simultaneously), Majority Voting, and Expectation-Maximization-based methods.
• Connection to Boosting: Both address the 'wisdom of the crowd' principle. Truth inference focuses on label aggregation, while boosting focuses on model aggregation for improved prediction.