Jackknife+ Prediction

Jackknife+ leverages leave-one-out cross-validation (LOOCV) to train n models on datasets of size n-1, where n is the total number of training points. Unlike full jackknife, which requires computing complex influence functions, Jackknife+ uses the absolute residuals from these held-out predictions as nonconformity scores. This avoids the computational instability of the original jackknife method while maintaining the efficiency of using nearly all data for both model fitting and calibration. The result is a method that is both statistically efficient and computationally practical for moderate-sized datasets.

Jackknife+ provides a rigorous finite-sample coverage guarantee without assuming any specific distribution for the data or the model's errors. Under the standard assumption of exchangeability—that the joint distribution of data points is invariant to permutation—the method guarantees that the prediction interval will contain the true label with at least the nominal coverage probability (e.g., 90%). This guarantee holds for any underlying algorithm, from linear regression to deep neural networks, making it a robust choice for high-stakes applications where statistical validity is non-negotiable.

Compared to split conformal prediction, Jackknife+ typically produces narrower prediction intervals. Split conformal must reserve a separate calibration set, reducing the data available for model training and leading to less accurate base models. By using leave-one-out residuals, Jackknife+ uses nearly all data for training while still obtaining valid nonconformity scores. The resulting prediction sets are often 10-30% tighter than those from split conformal methods, providing more informative uncertainty estimates without sacrificing coverage guarantees.

While more efficient than full jackknife, Jackknife+ requires training n separate models, which can be prohibitive for large datasets or expensive model classes like deep neural networks. For datasets with thousands of points, this cost may be manageable with parallelization. For massive datasets, practitioners often turn to K-fold cross-validation variants or stick with split conformal methods. The computational cost is the primary trade-off for the improved statistical efficiency, and the choice depends on whether tighter prediction sets justify the additional training overhead.

The Jackknife+ method was introduced by Barber et al. (2021) with a formal proof of its coverage guarantee. The key insight is that the method constructs prediction intervals using a symmetrized quantile of the leave-one-out residuals, which corrects for the optimistic bias that would arise from using in-sample residuals. Specifically, for a target coverage level of 1-α, the interval is formed by taking the (1-α)-th quantile of the augmented residuals. This construction ensures that the coverage probability is at least 1-2α, and in practice, coverage is often close to the nominal level.

Jackknife+ is primarily designed for regression tasks, where it produces prediction intervals for continuous outputs. The method can be extended to other settings by adapting the nonconformity measure:

• Classification: Use the predicted probability of the true class as the nonconformity score
• Quantile regression: Combine with conformalized quantile regression for adaptive intervals
• Time series: Apply with caution, as the exchangeability assumption may be violated For each extension, the core leave-one-out mechanism remains the same, but the nonconformity function must be tailored to the prediction task.