Weighted Conformal Prediction

Standard conformal prediction relies on exchangeability between calibration and test data. Under covariate shift, where P(X) changes but P(Y|X) remains stable, this assumption breaks. Weighted conformal prediction re-weights each calibration point's nonconformity score by the likelihood ratio w(x) = P_target(x) / P_source(x), restoring valid marginal coverage in the target domain without needing target labels.

The core challenge is estimating the density ratio between source and target distributions. Common approaches include:

• Kernel mean matching: Directly estimates weights by matching distribution embeddings in a reproducing kernel Hilbert space
• Logistic regression discrimination: Trains a classifier to distinguish source from target samples, then derives weights from predicted probabilities
• Density estimation: Estimates both densities separately and takes their ratio, though this suffers in high dimensions

Instead of taking the standard empirical quantile of calibration scores, weighted conformal prediction computes a weighted empirical CDF. The adjusted threshold is the (1-α)-quantile of the weighted distribution, where each calibration point i contributes its normalized weight w_i / Σw_j to the cumulative mass. This shifts the threshold to account for over- or under-represented regions in the source data.

When the importance weights are exactly known (oracle setting), weighted conformal prediction provides a rigorous coverage guarantee: P(Y_test ∈ C(X_test)) ≥ 1-α. With estimated weights, the coverage bound degrades gracefully by a term proportional to the total variation distance between true and estimated weight distributions, making the method robust to weight estimation errors.

Weighted conformal prediction is critical in real-world scenarios where training data cannot perfectly represent deployment conditions:

• Medical diagnostics: Models trained on one hospital's population deployed at another with different demographics
• Autonomous driving: Training on sunny-day data, deploying in rainy conditions
• Financial modeling: Models built on historical bull markets applied during volatile bear conditions

To improve efficiency under extreme covariate shift, normalized weighted conformal prediction divides each nonconformity score by a local variability estimate before applying weights. This produces tighter prediction sets in regions of high target density while maintaining validity. The normalization function can be any positive function of the features, typically learned alongside the base predictor.

The specific distributional mismatch that weighted conformal prediction is designed to handle. Covariate shift occurs when the input feature distribution P(X) changes between training and test time, while the conditional label distribution P(Y|X) remains stable.

• Example: A medical diagnosis model trained on data from Hospital A (younger demographic) deployed at Hospital B (older demographic).
• Mechanism: The importance weights applied to calibration samples are proportional to the density ratio w(x) = P_test(x) / P_train(x).
• Contrast: This differs from label shift (change in P(Y)) or concept drift (change in P(Y|X)), which require different correction techniques.

A relaxation of the standard exchangeability assumption that underpins the validity of weighted conformal prediction. Under covariate shift, the calibration and test data are no longer exchangeable.

• Core Idea: By applying a known, fixed importance weight to each calibration point's nonconformity score, a weighted empirical distribution is constructed.
• Validity: The resulting prediction sets maintain a marginal coverage guarantee with respect to the target test distribution, not the source training distribution.
• Requirement: The true density ratio must be known exactly or estimated accurately from data; errors in weight estimation can degrade the coverage guarantee.

The core algorithmic modification in weighted conformal prediction. Instead of computing a standard empirical quantile of nonconformity scores, a weighted quantile is calculated on the calibration set.

• Procedure: Sort calibration scores s_i and compute cumulative normalized weights. The threshold is the score at which the cumulative weight reaches the desired confidence level.
• Formula: The adjusted quantile threshold Q(1 - α) satisfies ∑ w_i * I(s_i ≤ Q) / ∑ w_i ≥ 1 - α.
• Impact: Samples from regions over-represented in training receive weights < 1, diminishing their influence. Samples from under-represented regions receive weights > 1, amplifying their influence on the threshold.

A critical prerequisite for weighted conformal prediction when the true importance weights are unknown. This involves training a separate model to estimate w(x) = P_test(x) / P_train(x) from unlabeled data.

• Methods: Common approaches include training a probabilistic classifier to distinguish between training and test samples, or using kernel mean matching.
• Challenge: High-dimensional input spaces make accurate density ratio estimation difficult, and errors propagate directly into the coverage guarantee.
• Robustness: Recent research focuses on making weighted conformal prediction robust to weight estimation errors, ensuring coverage holds even with imperfect weights.

The parent framework from which weighted conformal prediction inherits its distribution-free, model-agnostic properties. Standard conformal prediction provides a marginal coverage guarantee under the assumption of exchangeability.

• Core Mechanism: Uses a held-out calibration set to compute nonconformity scores and determine a threshold quantile.
• Output: Produces a prediction set that contains the true label with a user-specified probability (e.g., 90%).
• Limitation: The exchangeability assumption is violated under covariate shift, motivating the weighted variant to restore valid coverage on the target domain.

An alternative approach for handling distribution shift that operates in an online setting. Unlike weighted conformal prediction, it does not require pre-specified importance weights.

• Mechanism: Dynamically adjusts the quantile threshold over time based on observed coverage errors, using a fixed learning rate to increase or decrease the threshold.
• Guarantee: Provides a long-run average coverage guarantee, not a finite-sample guarantee for each point.
• Comparison: Weighted conformal prediction is suited for batch settings with known or estimable shift, while adaptive conformal inference excels in streaming environments with unknown, gradual drift.