Why Use Partial Dependence?
A boosted tree model can combine many splits and interactions, so its predictions may be difficult to summarize with a single coefficient or rule. A partial-dependence plot provides a compact view of how the fitted model’s predictions change as one or more selected predictors vary, while averaging over the other predictors in a specified data set. It describes the model’s behavior; it does not, by itself, explain why outcomes differ or estimate a causal effect.
The plot complements variable importance. Importance summarizes how much a predictor contributed to fitting the model under a particular method; a partial-dependence plot shows the direction and shape of the model’s predictions across values of that predictor. Neither is a substitute for checking whether the patterns are supported by the data.
The fitted model is held fixed: partial dependence interprets an already fitted model rather than refitting it at each grid value. For a regression model, \(\hat f\) may return a predicted outcome such as dollars of claim cost. For binary classification, it may return a predicted probability of the positive class. State which prediction scale is being plotted because a plot of probabilities and a plot of log-odds have different vertical scales and can have different shapes.
The empirical formula is a finite average and can always be calculated for a specified model and data set. Interpreting it as a summary for a wider population requires the data used for averaging to be relevant to that population. In particular, the formula does not ensure that the modified predictor combinations are realistic.
Constructing a Partial-Dependence Plot
For a one-predictor plot, choose a grid of values for that predictor. At each grid value, make a copy of the data used for averaging, replace that predictor’s value in every row with the grid value, obtain fitted-model predictions for all rows, and take their arithmetic mean. Plot the grid values horizontally and the means vertically. For a plot of two predictors, choose a grid for each and calculate the average at each pair of grid values; the result can be shown as a surface or a set of contours.
In R, the central operation is to create modified copies of the data and pass them to the fitted model’s prediction function. Use the prediction type that returns the intended scale: for example, response-scale predictions for predicted probabilities. Preserve the other columns, their types, and factor levels expected by the model. The exact prediction argument can differ by boosting implementation, so check that it requests the desired scale rather than assuming every default does so.
Decide which predictor or predictors to vary and whether the plot should show fitted outcomes, probabilities, or another model output.
Use interpretable values within the predictor’s observed range. For categorical predictors, use valid levels; do not manufacture levels that the model was not fitted to handle.
At each grid point, set the selected predictor values in every averaging-data row to that point and request predictions from the fixed fitted model.
Take the mean of the row-level predictions at each grid point, then plot those means against the grid. Label the prediction scale and averaging population.
The averaging data matter: averaging over the training rows answers a different descriptive question from averaging over a portfolio or a validation sample. The model should remain fixed while comparing these choices. For a categorical predictor, the same procedure applies level by level, but the horizontal axis is a set of categories rather than a continuous scale.
Worked Example: A Regression Partial-Dependence Curve
A hypothetical boosted model predicts annual claim cost in dollars. An analyst wants a partial-dependence plot for annual mileage, using four policy profiles as the averaging data. The model is evaluated at two mileage values, 2,000 and 4,000 miles. The table gives the predictions after replacing mileage in every profile, while keeping each profile’s other predictors unchanged.
| Profile | Prediction at 2,000 miles | Prediction at 4,000 miles |
|---|---|---|
| 1 | $120 | $150 |
| 2 | $140 | $160 |
| 3 | $100 | $130 |
| 4 | $160 | $180 |
Method. Apply the empirical partial-dependence formula separately at each mileage value. Each profile has equal weight, so \(n=4\).
At 2,000 miles, the sum of predictions is \(120+140+100+160=520\). The partial-dependence value is \(520/4=\$130\).
At 4,000 miles, the sum is \(150+160+130+180=620\). The partial-dependence value is \(620/4=\$155\).
The plotted points are therefore \((2{,}000,\$130)\) and \((4{,}000,\$155)\). Across these two grid values, the fitted model’s average prediction rises by \(\$155-\$130=\$25\). This is an average change in the model’s predictions over these four profiles, not a statement that increasing an individual policyholder’s mileage causes a $25 increase. The calculation checks: the two plotted heights are the respective row averages, and the higher grid value has a $25 greater average prediction.
Common mistake avoided: The analyst does not select a single profile’s prediction as the plot height. The method averages all four predictions at each grid value.
Classification Plots and Prediction Scale
For binary classification, a partial-dependence plot often shows the average predicted probability of the event. Each row-level prediction must be on the probability scale before averaging. Averaging probabilities is not generally the same operation as averaging a model’s log-odds and then converting that average to a probability. Therefore, the intended scale must be chosen before calculating the plotted values.
Worked Example: Averaging Event Probabilities
A hypothetical boosted classifier estimates the probability that a submitted equipment claim will require specialist review. The analyst varies the equipment-age category and averages across three claim profiles. The fitted probabilities below are response-scale predictions.
| Profile | New equipment | Older equipment |
|---|---|---|
| 1 | 0.10 | 0.40 |
| 2 | 0.30 | 0.50 |
| 3 | 0.20 | 0.30 |
For new equipment, the probability sum is \(0.10+0.30+0.20=0.60\), so the average is \(0.60/3=0.20\). For older equipment, the sum is \(0.40+0.50+0.30=1.20\), so the average is \(1.20/3=0.40\).
The partial-dependence plot on the probability scale has values 0.20 and 0.40: the average fitted probability is 20% for the new-equipment grid point and 40% for the older-equipment grid point. The increase is 0.20, or 20 percentage points. The calculation checks because each result lies within the range of its three input probabilities.
Common mistake avoided: The output is explicitly the average of the predicted probabilities. If the model returns a link-scale prediction by default, those values must first be converted to probabilities if the requested plot is intended to show average probabilities.
Why the Average Has This Form
The formula follows directly from the quantity being summarized. Fix a value \(x_S\) for the selected predictor or predictors. For each row \(i\), the model prediction \(\hat f(x_S,x_{C,i})\) is the prediction obtained when the selected values are set to \(x_S\) and the row’s other predictor values are retained. The empirical average of those \(n\) predictions is, by definition, their sum divided by \(n\):
Equivalently, if the observed rows are treated as equally weighted points in an empirical distribution of the other predictors, this sum is the empirical expectation of the model prediction over that distribution. This explains both the averaging operation and its limitation: the values \(x_{C,i}\) come from the chosen data, regardless of whether pairing each one with the fixed \(x_S\) is plausible.
Worked Example: Correlated Predictors Create Unsupported Combinations
Suppose a hypothetical boosted model predicts a repair cost using two binary predictors: machine age \(A\), coded 0 for newer and 1 for older, and a maintenance indicator \(M\), coded 0 for routine maintenance and 1 for intensive maintenance. The averaging data contain two equally weighted profiles: \((A,M)=(0,0)\) and \((A,M)=(1,1)\). Thus, the predictors are perfectly correlated in these data; neither \((0,1)\) nor \((1,0)\) is observed.
Assume the fitted model predicts \(\$100\) when the two predictor values match and \(\$900\) when they differ. This is a possible pattern for a tree-based model with splits on both predictors. On the two observed profiles, the predictions are \(f(0,0)=\$100\) and \(f(1,1)=\$100\).
To calculate partial dependence for age at \(A=0\), replace age by 0 in both rows and retain each row’s maintenance value. The resulting combinations are \((0,0)\) and \((0,1)\), with predictions \(\$100\) and \(\$900\). Their mean is \((100+900)/2=\$500\).
At \(A=1\), the combinations are \((1,0)\) and \((1,1)\), with predictions \(\$900\) and \(\$100\). The mean is \((900+100)/2=\$500\). The age partial-dependence plot is therefore flat at \(\$500\), even though the model predicts \(\$100\) on both observed profiles.
The arithmetic is correct, but half of the modified combinations at each grid value are unsupported by these data. The resulting flat curve is a poor description of what the model predicts on the observed profiles. A plot cannot reveal that problem from its shape alone; the analyst must inspect predictor relationships and the combinations generated by the averaging procedure.
Common mistake avoided: The analyst does not interpret a mathematically valid average as reliable evidence about realistic profiles. Perfect correlation makes the required replacements create combinations absent from the data.
Interpretation, Limits, and Exam Traps
Partial dependence is a model-summary tool, not a causal analysis. Setting a predictor to a grid value for every row is a computational replacement, not an intervention that establishes what would happen if real individuals’ predictor values changed. In particular, a curve should not be described as the effect of changing a predictor unless a separate causal design and assumptions support that claim.
Correlation among predictors is a central diagnostic concern. When the selected predictor is associated with other predictors, replacing it while leaving the others untouched may create unusual combinations. The model must still return predictions for them, but those predictions may rely on splits in regions with little or no data support. A smooth-looking or flat curve does not solve this problem.
An individual conditional expectation curve, often abbreviated ICE, instead displays the sequence of predictions for one row as the selected predictor varies, rather than averaging across rows. Comparing such row-level curves can reveal variation hidden by an average, but it does not automatically fix unsupported combinations. A conditional average that restricts the averaging to profiles with similar values of correlated predictors answers a different question and also requires care about which profiles are included.
A practical interpretation should state the model output, the grid range, and the data used for averaging. It should also note whether the plotted range has adequate data support and whether important predictors are strongly associated with the selected predictor. For a categorical predictor, describe differences among levels rather than a continuous trend. For a numeric predictor, avoid extending the grid far beyond observed values merely to make the axis look complete.
Check Your Understanding
For each problem, calculate or interpret the requested partial-dependence quantity and check whether the data support the interpretation.
- A regression model gives predictions of $80, $100, and $120 when a selected predictor is set to 5 for three averaging rows. What is the partial-dependence value at 5? (Answer: $100.)
- A classifier’s response-scale probabilities at grid value \(x=2\) are 0.15, 0.25, 0.40, and 0.20. Calculate the partial-dependence value. (Answer: 0.2500.)
- A regression model has four equally weighted predictions at grid value \(x=10\): $40, $70, $90, and $120. At \(x=20\), they are $50, $80, $110, and $160. Calculate both plot heights and the change. (Answer: $80 at \(x=10\), $100 at \(x=20\), an increase of $20.)
- In a two-row averaging data set, the only observed predictor pairs are \((X,Z)=(0,0)\) and \((1,1)\). A fitted model predicts $50 for matching values and $450 for mismatches. What are the partial-dependence values for \(X=0\) and \(X=1\), and what is the support concern? (Answer: $250 at each value; each average includes an unobserved mismatched pair.)
- A binary classifier returns log-odds of \(-1\) and \(1\) for two profiles after setting a predictor to a selected value. The requested plot is average predicted probability. Convert each log-odds value using \(p=1/(1+e^{-\eta})\), then average the probabilities. (Answer: approximately 0.5000.)