Partial Dependence Plots

PDPs visualize the average prediction as a function of a target feature by marginalizing over the distribution of all other features. The core mechanism involves iterating through every instance in the dataset, holding the target feature constant at a specific value while keeping other features at their original values, and then averaging the model's predictions. This process is repeated across a grid of feature values to trace the curve. The resulting plot answers: 'How does the prediction change, on average, as feature X varies?'

The primary statistical limitation of PDPs is their reliance on the assumption of feature independence. When features are correlated, the averaging procedure forces the model to evaluate unrealistic data points—combinations of feature values that never occur in the real world. For example, if height and weight are correlated, a PDP for height will average predictions over all observed weights, including improbable pairings like a 7-foot person weighing 50kg. This can produce misleading effect estimates and is the key reason practitioners must check correlation matrices before trusting PDP results.

While one-way PDPs show the marginal effect of a single feature, two-way PDPs visualize the joint effect of two features simultaneously using a contour or heatmap. This is critical for detecting interaction effects that are invisible in one-way plots. For instance, a two-way PDP for 'age' and 'exercise frequency' predicting health risk might reveal that the effect of age is steep only for low-exercise individuals, while high-exercise individuals show a flat curve. This visual diagnosis helps modelers decide whether to engineer interaction terms.

PDPs show the average effect, which can mask heterogeneous responses across the population. Individual Conditional Expectation (ICE) plots are the disaggregated counterpart—they plot the functional relationship for every single instance as a separate line. Overlaying ICE curves on a PDP reveals whether the average effect is representative. If all ICE lines share the same shape, the PDP is trustworthy. If the lines diverge or cross, the average is a poor summary, indicating strong interaction effects that require further investigation.

To address the correlation bias inherent in PDPs, Accumulated Local Effects (ALE) plots were developed as an unbiased alternative. Instead of averaging over the full marginal distribution, ALE computes feature effects by accumulating local differences in predictions within small, conditional partitions of the feature space. This means ALE only evaluates the model on realistic data neighborhoods, making it robust to correlated features. For any dataset with known multicollinearity, ALE plots should be preferred over PDPs for reliable interpretation.

PDPs are model-agnostic and can be computed for any fitted model with a predict function. The algorithm is straightforward:

• Step 1: Define a grid of values for the feature of interest.
• Step 2: For each grid value, create a copy of the entire dataset with the feature set to that value.
• Step 3: Obtain predictions for all modified datasets.
• Step 4: Average predictions at each grid point. The computational cost scales linearly with the number of grid points multiplied by the dataset size, making PDPs efficient for moderate datasets but potentially slow for large-scale applications.