Conditional Coverage

Achieving exact conditional coverage for all possible feature values is statistically impossible without knowing the true data distribution. Any distribution-free method that guarantees marginal coverage must have vanishing conditional coverage for some subgroups. This fundamental limit, proven by Vovk (2012) and Lei & Wasserman (2014), means the goal is always an approximation—seeking the most uniform coverage possible across the input space without parametric assumptions.

The primary method for enforcing label-conditional coverage. It applies the conformal calibration step independently within pre-defined, non-overlapping categories (e.g., class labels).

• Mechanism: Computes a separate nonconformity score quantile for each category.
• Guarantee: Ensures P(Y_test ∈ C(X_test) | Y_test = k) ≥ 1-α for each class k.
• Limitation: Only works for discrete, pre-specified groups; does not guarantee coverage for continuous features like age or income.

When the target is covariate-conditional coverage (coverage for specific input feature values), weighted conformal prediction is the standard approximation tool. It assigns importance weights to calibration samples based on their similarity to the test point.

• Kernel Weighting: Uses a kernel function (e.g., Gaussian RBF) to up-weight calibration points near the test input.
• Guarantee: Provides valid marginal coverage under covariate shift, but only approximates conditional coverage.
• Trade-off: Larger bandwidths improve statistical efficiency but worsen conditional validity.

A class of methods that compute nonconformity scores using only a local neighborhood of the test point rather than the full calibration set.

• K-Nearest Neighbors (KNN): Uses the k closest calibration points to compute a local quantile.
• Decision Tree Partitioning: Fits a tree to the feature space and applies conformal calibration within each leaf node.
• Benefit: Directly targets conditional coverage by restricting the calibration distribution to a region where exchangeability is a more plausible approximation.

Evaluating conditional coverage requires specialized diagnostics beyond marginal error rates:

• Worst-Slab Coverage (WSC): Measures the minimum coverage over all slabs (subspaces) of a given probability mass, providing a rigorous lower bound on conditional validity.
• Conditional Coverage Discrepancy (CCD): Quantifies the maximum deviation between nominal and empirical coverage across a partition of the feature space.
• Local Coverage Plots: Visualize empirical coverage as a function of a continuous covariate to identify systematic under-coverage regions.

Conformalized Quantile Regression (CQR) is a leading approach for achieving approximate conditional coverage in regression. It wraps a base quantile regressor with a conformal calibration step.

• Process: Fits lower and upper quantile functions, then adjusts their width using a calibration set to achieve marginal validity.
• Adaptivity: The resulting prediction intervals automatically expand in regions of high heteroscedastic noise and contract where the data is clean.
• Result: Produces intervals that closely track the conditional distribution of Y|X, often achieving near-conditional coverage in practice.

Ensuring prediction sets maintain validity across protected demographic subgroups is critical for regulatory compliance. Conditional coverage approximations detect when a model's uncertainty is systematically higher for minority populations.

• Audit loan approval models for disparate coverage across ZIP codes
• Validate medical diagnostic models have equal coverage across patient demographics
• Flag when conformalized classifiers produce wider prediction sets for specific groups

In radiology, Mondrian conformal prediction applies calibration independently within each disease category. This ensures that a 95% prediction set for 'benign cyst' and 'malignant tumor' both contain the true diagnosis with the advertised probability, preventing dangerous overconfidence in critical classes.

• Label-conditional coverage for each ICD-10 diagnostic code
• Guarantees equal error rates across disease severity tiers
• Prevents model overconfidence on rare but life-threatening conditions

Object detectors must provide reliable uncertainty bounds regardless of weather conditions or lighting. Adaptive conformal inference adjusts prediction set sizes in real-time for fog, rain, and nighttime scenarios, ensuring pedestrian detection maintains coverage even under distribution shift.

• Condition on environmental metadata: precipitation, ambient light, occlusion level
• Wider prediction sets automatically generated for degraded sensor inputs
• Maintains safety guarantees when transitioning from highway to urban environments

Regulatory frameworks require that automated underwriting models provide equally valid confidence intervals across all FICO score bands. Weighted conformal prediction re-weights calibration samples to correct for covariate shift between applicant populations.

• Guarantee coverage for thin-file and new-to-credit applicants
• Adjust for distributional differences between prime and subprime segments
• Generate compliant adverse action explanations with subgroup-level validity

When predicting individual treatment effects, conformalized causal inference must provide valid confidence intervals for each patient subgroup defined by biomarkers or comorbidities. This prevents inflated certainty for underrepresented cohorts in the training data.

• Conditional coverage for each genetic marker subgroup
• Valid individual treatment effect bounds regardless of covariate distribution
• Supports FDA submissions requiring subgroup-level uncertainty quantification

Equipment failure prediction sets must remain valid across operating temperature ranges and production speeds. Mondrian conformal prediction partitions calibration data by operational regime, ensuring a 99% coverage guarantee holds whether the machine is idle, at peak throughput, or undergoing a tool change.

• Regime-conditional validity for each discrete manufacturing state
• Prevents false confidence during rare but critical high-stress operating modes
• Integrates with SCADA systems for real-time conditional uncertainty display

A foundational result in distribution-free inference proving that exact conditional coverage for all feature values is impossible without making strong parametric assumptions about the data distribution. Specifically, for any finite-sample valid prediction set, there will always exist some regions of the feature space where coverage is systematically above or below the nominal level. This theorem, formalized by Vovk, Lei, and Wasserman, establishes that the goal of conformal prediction is to achieve marginal coverage while using algorithmic design to minimize conditional coverage deviations as much as possible.