Calibration Plots for Survival Models

Unlike binary classification calibration, survival calibration must be evaluated at specific time points (e.g., 1-year, 5-year survival). A model may be well-calibrated at one horizon but poorly calibrated at another. Plots typically display predicted probability on the x-axis against observed event rates on the y-axis for a fixed prediction horizon, with a 45-degree diagonal representing perfect calibration.

• Example: A 5-year predicted survival of 80% should match an observed survival of 80% in that risk group.
• Key Insight: Time-dependent calibration curves reveal whether a model systematically overestimates or underestimates risk at clinically relevant decision points.

To construct a calibration plot, patients are typically divided into risk groups based on their predicted survival probabilities. Common approaches include grouping by deciles of predicted risk or using clinically meaningful cut-points. The observed event rate within each group is then estimated using the Kaplan-Meier estimator to account for censoring.

• Bias-Variance Trade-off: Fewer groups (e.g., 4-5) reduce variance but may mask miscalibration within groups.
• Adaptive Grouping: Some methods use locally weighted smoothing (loess) to create flexible calibration curves without arbitrary bin boundaries, providing a continuous assessment of calibration across the risk spectrum.

A robust statistical technique for constructing calibration plots in the presence of right-censoring. Pseudo-observations are computed for each individual at a specific time point, representing their contribution to the survival estimate. These pseudo-values can then be used as the outcome in a regression model to estimate observed event rates conditional on predicted risk.

• Advantage: Avoids the need for arbitrary risk grouping and provides a direct estimate of the expected event status for each individual.
• Implementation: The pseudo-observation for subject i is calculated as n × Ŝ(t) - (n-1) × Ŝ⁻ⁱ(t), where Ŝ⁻ⁱ(t) is the Kaplan-Meier estimator computed without subject i.

Beyond visual inspection, calibration can be quantified by regressing observed outcomes on predicted probabilities. The calibration slope measures the spread of predicted risks—a slope less than 1 indicates predictions are too extreme (overfitting), while a slope greater than 1 suggests they are too modest. The calibration intercept (calibration-in-the-large) assesses systematic over- or under-prediction.

• Ideal Values: Slope = 1, Intercept = 0.
• Clinical Relevance: A model with a slope of 0.8 systematically overestimates risk for high-risk patients and underestimates it for low-risk patients, potentially leading to inappropriate treatment decisions.

In the presence of competing risks (e.g., death from other causes preventing the event of interest), standard calibration plots using the Kaplan-Meier estimator will be biased. The observed event rate must instead be estimated using the Aalen-Johansen estimator or the Cumulative Incidence Function (CIF). Calibration plots for competing risks compare predicted CIF values against observed cumulative incidence.

• Critical Distinction: A model predicting cancer-specific mortality must account for non-cancer deaths, as treating them as simple censoring events violates the independent censoring assumption.
• Fine-Gray Context: Calibration of subdistribution hazard models should be assessed on the CIF scale, not the survival scale.

A complementary approach that evaluates calibration within clinically relevant risk thresholds. Instead of assessing calibration across the entire probability range, decision curves focus on the range of predicted risks where treatment decisions would change. A model may be well-calibrated globally but poorly calibrated in the critical zone where clinicians decide between intervention and observation.

• Net Benefit: Combines calibration and discrimination into a single decision-theoretic metric.
• Threshold-Specific: Evaluates whether a model with a predicted risk of 20% actually yields an observed event rate of 20% in the subgroup where that threshold matters most.

A rank-based discrimination metric that evaluates how well a model orders patients by risk. It measures the proportion of comparable patient pairs where the predicted risk aligns with actual event order.

• Value of 1.0: Perfect risk ordering
• Value of 0.5: No better than random chance
• Handles right-censored data through pairwise comparisons
• Does NOT assess absolute risk accuracy (calibration)

A model can achieve a high C-index while being poorly calibrated—hence calibration plots are essential to confirm that predicted probabilities match observed event rates.

Extends the standard receiver operating characteristic framework to survival data by evaluating how well a biomarker or model discriminates between subjects who will and will not experience an event by a specific time horizon.

• Cumulative/Dynamic: Cases defined by event before time t
• Incident/Dynamic: Cases defined by event exactly at time t
• Produces time-specific AUC values
• Identifies optimal risk thresholds for patient stratification

While ROC curves assess discrimination, calibration plots verify that the probabilities at those thresholds are accurate—both are required for clinical deployment.

Diagnostic residuals used to test the proportional hazards assumption in Cox models. They check whether a covariate's effect remains constant over the observation period.

• Plotted against time to detect time-varying effects
• The Grambsch-Therneau test provides a formal statistical test
• Non-zero slope indicates violation of proportional hazards
• Essential preprocessing step before interpreting calibration

If the proportional hazards assumption is violated, the underlying model is misspecified, and calibration plots will reflect systematic bias across time points.

A statistical technique that corrects for bias introduced by dependent or informative censoring by weighting uncensored observations inversely to their estimated probability of remaining in the study.

• Accounts for censoring that correlates with prognosis
• Requires modeling the censoring distribution
• Produces consistent estimates of survival probabilities
• Critical when evaluating calibration under non-random dropout

Calibration plots constructed without IPCW may appear optimistic if patients with worse prognoses are systematically lost to follow-up.

The area under the survival curve up to a specified time point τ, providing a clinically interpretable summary of treatment benefit without requiring the proportional hazards assumption.

• Expressed in absolute time units (months, years)
• Directly answers: "How much longer do patients live on average?"
• Robust when hazards are non-proportional
• Complements calibration by offering an alternative estimand

When calibration plots reveal poor fit at later time points, RMST provides a model-free summary that remains valid and interpretable for clinical decision-making.