Why a Performance Gap Needs Diagnosis
Suppose a tree ensemble predicts better than a generalized linear model (GLM) on the same validation observations. That is evidence about comparative predictive performance, not a diagnosis of why the models differ. The gap could reflect curvature in a predictor’s effect, an interaction, sparse regions where the GLM extrapolates poorly, or other differences in how the models represent the data. It could also be small enough to be resampling noise.
A useful next step is to compare how the fitted models’ predictions change across important predictor values. These partial prediction patterns can suggest a targeted GLM revision, such as adding a quadratic term or an interaction. The revised model must then be evaluated on the same target and matched validation partitions. A pattern is a diagnostic clue, not proof that a particular relationship is real, causal, or stable.
The comparison is meaningful only when the models address the same prediction target, use compatible data and preprocessing, and are assessed on matched validation observations. For a rate model, compare predicted rates rather than accidentally comparing rates in one model with counts in another. If an offset is used, keep its interpretation and treatment consistent. These controls help isolate a specification question from a target or pipeline mismatch.
Read Patterns on a Defined Scale
Consider a Poisson GLM for a count \(Y_i\), with exposure \(e_i\), log link, and a linear predictor \(\eta_i\). Its conditional mean is
This model assumes conditionally independent Poisson responses with \(E[Y_i\mid\mathbf{x}_i]=\mu_i\) and \(\operatorname{Var}(Y_i\mid\mathbf{x}_i)=\mu_i\). The offset \(\log(e_i)\) has coefficient fixed at one. If exposure is one unit, the expected count is also the predicted rate per unit exposure. A Poisson model may be a poor description if its conditional variance assumption is materially unsuitable; the formula should not be treated as universal for every count outcome.
For a predictor \(x\), a GLM with only a linear term for \(x\) has a straight-line effect on the log-mean scale, not on the mean scale. When other terms are held fixed, increasing \(x\) by one unit multiplies the mean by \(\exp(\beta_x)\). The mean-scale increase is therefore not a constant amount. A visually curved mean-scale pattern alone does not establish that a log-link GLM needs a nonlinear term.
One way to summarize a fitted model \(f\) is the empirical partial dependence at a chosen value \(x\):
where \(\mathbf{z}_i\) denotes the other predictors for reference observation \(i\), and \(f(x,\mathbf{z}_i)\) is the model prediction after setting \(x\) to the specified value. For an exposure-adjusted rate comparison, define \(f\) to return the rate, or use the same exposure convention for every prediction. This calculation averages predictions; it does not describe the outcome of a typical individual. If \(x\) and the other predictors are strongly associated, setting \(x\) to a value for every reference row can create combinations that are rare or absent in the data. The resulting pattern may then depend on unsupported model extrapolation.
From a Partial Pattern to a GLM Revision
A disciplined diagnosis separates three questions: what shape is visible, whether the data support the comparisons behind that shape, and whether a targeted GLM revision improves held-out predictions. A nonlinear pattern suggests considering a nonlinear term. A difference in slopes across levels of another predictor suggests considering an interaction. Neither pattern should prompt automatic inclusion of every term that could reproduce it.
Use the same outcome definition, prediction scale, validation observations, and appropriate exposure or weight treatment for the GLM and ensemble.
Plot or tabulate predictions over a sensible range of the predictor. State whether the scale is the link scale, response scale, or another defined scale, and identify the reference set.
Ask whether the data contain useful observations across the predictor range and, for a possible interaction, across combinations of the predictors involved.
Add a prespecified nonlinear term or interaction motivated by the pattern. Keep the rest of the modeling setup comparable where possible.
Assess predictions on the same matched partitions and compare the same metric. Treat a small or inconsistent gain cautiously.
Justification: Why a Quadratic Term Allows Curvature
A linear predictor in \(x\), \(\eta(x)=\beta_0+\beta_1x\), has a constant slope with respect to \(x\): \(d\eta(x)/dx=\beta_1\). Adding \(x^2\) gives \(\eta(x)=\beta_0+\beta_1x+\beta_2x^2\). Differentiating shows that the slope now depends on \(x\):
Thus, if \(\beta_2\ne 0\), the log-mean can bend rather than remain a straight line in \(x\). For a log-link model, \(\mu(x)=\exp(\eta(x))\), so the mean-scale curve is also generally nonlinear. This flexibility has a cost: the fitted shape can be unreliable near sparsely observed extremes, and a quadratic may not represent the true pattern. It is a candidate specification to evaluate, not a guarantee of improvement.
Worked Example: A Curved Rate Pattern Suggests a Quadratic Term
A hypothetical insurer models annual claim counts with a Poisson log-link GLM and unit exposure. For a representative reference profile, the fitted linear model has \(\eta=-2+0.01x\), where \(x\) is a centered driving measure. A tree ensemble’s partial predicted rates at \(x=0,10,20\) are \(0.135, 0.182, 0.333\). A targeted quadratic GLM has \(\eta=-2+0.01x+0.001x^2\). Compute its rates at those points and compare the patterns.
Method and assumptions. Because exposure is one, the Poisson GLM’s expected count equals its rate. Use \(\mu=\exp(\eta)\), and compare predictions at the same stated reference profile. The numbers are illustrative, not company data.
At \(x=0\), the revised linear predictor is \(-2+0+0=-2\), so \(\mu=\exp(-2)=0.1353\).
At \(x=10\), it is \(-2+0.01(10)+0.001(10^2)=-2+0.1+0.1=-1.8\), so \(\mu=\exp(-1.8)=0.1653\).
At \(x=20\), it is \(-2+0.01(20)+0.001(20^2)=-2+0.2+0.4=-1.4\), so \(\mu=\exp(-1.4)=0.2466\).
The revised GLM rates are therefore \(0.1353, 0.1653, 0.2466\), while the ensemble rates rise more sharply over the latter part of the grid. The quadratic term introduces curvature, but this particular fitted revision does not reproduce the ensemble’s last two values. The pattern motivates checking the fit and validation performance; it does not show that the quadratic explains the entire gap.
Check. Substituting \(x=0\) removes both added terms, recovering \(\exp(-2)\). The common mistake is to compare a coefficient or a log-mean directly with an ensemble rate. Exponentiating each linear predictor puts the GLM on the rate scale before comparison.
Interactions Have a Different Signature
A nonlinear main effect changes the shape of a predictor’s relationship across its range. An interaction means that the effect of one predictor varies with another predictor. In a GLM, the interaction can be represented explicitly. For numeric \(x\) and a binary indicator \(z\), a log-link model might use
At \(z=0\), the slope in \(x\) on the log-mean scale is \(\beta_1\). At \(z=1\), it is \(\beta_1+\beta_3\). If the slopes differ, the interaction term changes the relationship across the two groups. A difference between groups in overall level alone is represented by \(\beta_2\), not necessarily by an interaction.
Worked Example: Distinguishing a Group Difference from an Interaction
A hypothetical model predicts event rates by a centered risk score \(x\) and an indicator \(z\) for a second exposure. Its unit-exposure log-mean is \(-2+0.01x+0.3z+0.02xz\). Calculate rates at \(x=0\) and \(x=10\) for both groups, then identify the slope pattern.
Method and assumption. Treat the stated predictor as a Poisson log-link linear predictor with unit exposure. First calculate each log-mean, then exponentiate to obtain rates.
For \(z=0,x=0\), \(\eta=-2\), giving rate \(\exp(-2)=0.1353\). For \(z=0,x=10\), \(\eta=-2+0.1=-1.9\), giving \(\exp(-1.9)=0.1496\).
For \(z=1,x=0\), \(\eta=-2+0.3=-1.7\), giving \(\exp(-1.7)=0.1827\). For \(z=1,x=10\), \(\eta=-2+0.1+0.3+0.2=-1.4\), giving \(\exp(-1.4)=0.2466\).
On the log-mean scale, the slope is \(0.01\) for \(z=0\) and \(0.01+0.02=0.03\) for \(z=1\). The groups differ both in level and in slope. The \(0.3z\) term shifts the level; the \(0.02xz\) term makes the score effect different by group.
Check. Setting \(z=0\) removes both group terms, as it should. A common mistake is to call any separation between group prediction curves an interaction. Different levels alone do not establish different slopes; compare the changes over \(x\) on the model’s link scale.
Check Support Before Trusting the Pattern
Partial dependence can be especially hard to interpret when predictors are correlated. Averaging predictions after replacing \(x\) can pair a value of \(x\) with values of other predictors that rarely occur together. A tree ensemble may still return a prediction for such a combination, but that prediction can reflect how the fitted model behaves outside the data’s support.
Worked Example: A Partial Average Uses Unsupported Combinations
A hypothetical portfolio has four reference records: two urban records and two rural records. In the observed data, both urban records have low score \(x=0\), and both rural records have high score \(x=10\); no other combinations occur. A fitted model predicts rates of \(0.10\) for urban/low, \(0.30\) for urban/high, \(0.20\) for rural/low, and \(0.40\) for rural/high. Calculate the model’s empirical partial dependence at each score and assess its interpretation.
Method and assumption. Use the definition that averages predictions over the four observed records’ location values while setting \(x\) to the chosen value. Each location receives weight \(2/4\).
At \(x=0\), the averaged prediction is \([2(0.10)+2(0.20)]/4=(0.20+0.40)/4=0.15\). At \(x=10\), it is \([2(0.30)+2(0.40)]/4=(0.60+0.80)/4=0.35\). The partial dependence increases by \(0.20\).
However, the observed data contain only urban/low and rural/high combinations. The calculation also uses urban/high and rural/low predictions, which are unsupported by these records. Therefore, the \(0.20\) difference is a model-based partial average, not a directly observed within-location comparison. It suggests a pattern to investigate, but cannot by itself identify a stable score effect.
Check. The weights sum to one and both rates are averages of four predictions. The common mistake is to treat a partial dependence curve as if it compared observed individuals at each score. Check predictor support and, where appropriate, inspect patterns within supported groups or restrict the comparison to a meaningful range.
Validate the Revision, Not Just Its Appearance
After identifying a plausible pattern, fit a narrow revision and compare validation performance under the same partitions and metric. For example, a quadratic term may be compared with the original GLM; an interaction may be compared with the version containing only main effects. Keep tuning and preprocessing fair, and do not select a specification using the validation outcomes and then report those same outcomes as an unbiased final assessment.
A lower validation error supports the claim that the revised specification predicted better on those validation partitions. It does not prove that the hypothesized mechanism is correct. Conversely, a revision that resembles the ensemble’s partial pattern but does not improve validation performance has not shown that the pattern closes the predictive gap. Consider the size and consistency of the difference, not only which score is numerically smallest.
Worked Example: A Revised GLM Narrows but Does Not Remove the Gap
For four held-out observations in a hypothetical loss-cost study, actual costs in thousands of dollars are \(1,2,4,8\). The original GLM predicts \(1,3,5,7\); a revised GLM predicts \(1,2,4,7\); and an ensemble predicts \(1.2,1.8,4.5,7.5\). Calculate each model’s RMSE and state what the comparison supports.
Method and assumptions. Treat these as predictions for the same four held-out cases on the same scale. RMSE is the square root of the mean squared prediction error. These four values illustrate the calculation; they do not establish a generalization result by themselves.
For the original GLM, prediction errors (actual minus predicted) are \(0,-1,-1,1\). Their squares are \(0,1,1,1\), with mean \(3/4=0.75\). Its RMSE is \(\sqrt{0.75}=0.8660\) thousand dollars.
For the revised GLM, errors are \(0,0,0,1\); squared errors are \(0,0,0,1\), with mean \(1/4=0.25\). Its RMSE is \(\sqrt{0.25}=0.5000\) thousand dollars.
For the ensemble, errors are \(-0.2,0.2,-0.5,0.5\); squared errors are \(0.04,0.04,0.25,0.25\), with mean \(0.58/4=0.145\). Its RMSE is \(\sqrt{0.145}=0.3808\) thousand dollars.
The revised GLM improves on the original in this calculation, while the ensemble still has the lowest RMSE. The result is consistent with the revision narrowing, but not eliminating, the gap. It does not isolate nonlinearity as the cause unless the revision was specifically motivated by and adequately tests that possibility.
Check. The ensemble squared errors sum to \(0.58\), and all three RMSEs use the same four cases and response units. A common mistake is to compare models on different observations or to compare squared error for one model with RMSE for another.
Common Diagnostic Traps
- Calling an ensemble win proof of nonlinearity. It only establishes a performance difference under the evaluation design. Multiple modeling and data issues can produce that difference.
- Confusing response-scale curvature with nonlinear link-scale effects. A log-link GLM with a linear predictor already produces a curved mean-scale relationship.
- Reading a group-level gap as an interaction. An interaction requires the effect or slope to vary across groups, not merely different prediction levels.
- Ignoring unsupported combinations. A partial average can depend on predictions for predictor combinations absent from the data.
- Promoting a visual finding to a final result. Use a targeted revision and matched validation; avoid claiming causal explanation from predictive patterns alone.
Check Your Understanding
For each problem, identify the relevant pattern or calculation before checking the answer.
- A unit-exposure Poisson GLM has log-mean \(-1.5+0.02x\). Calculate its predicted rate at \(x=10\), then state whether the log-mean is linear or curved in \(x\). (Answer: rate \(=\exp(-1.3)=0.2725\); the log-mean is linear in \(x\).)
- A log-link model has predictor \(-2+0.04x+0.5z-0.01xz\), where \(z\) is binary. Find the log-mean slope in \(x\) for each value of \(z\). (Answer: slope \(0.04\) when \(z=0\), and \(0.03\) when \(z=1\); the interaction changes the slope by \(-0.01\).)
- For three reference records, the predictions after setting \(x=5\) are \(0.10,0.20,0.40\). Compute empirical partial dependence at \(x=5\). What does it summarize? (Answer: \((0.10+0.20+0.40)/3=0.2333\); it is the average model prediction over those reference records after setting \(x=5\).)
- Four held-out costs are \(2,4,6,8\), and a revised model predicts \(1,5,5,9\). Calculate RMSE and state why that calculation alone cannot establish that the revision generalizes better than another model. (Answer: squared errors are \(1,1,1,1\), so RMSE \(=\sqrt{4/4}=1\) in the cost units; comparison requires another model evaluated on the same cases, and broader evidence requires appropriate validation.)
- A partial pattern appears to rise sharply at high values of \(x\), but almost all high-\(x\) records belong to one region. Give one reason the pattern may be misleading and one suitable diagnostic step. (Answer: the pattern may rely on unsupported combinations of \(x\) and region; inspect support and compare predictions within regions or over a range with adequate overlap.)