Why Individual Prediction Gaps Deserve Investigation
A GLM, a penalized regression model, and a tree ensemble can perform similarly overall while giving markedly different predictions for a particular case. A disagreement is not, by itself, evidence that one prediction is wrong. It is a diagnostic signal: the models may be responding differently to a nonlinear relationship, a combination of features, or an unusual input value.
The task here is to explain the gap for a specified case. Begin by confirming that every model predicts the same target for the same case, using the same units, exposure definition, offsets, and preprocessing. Then examine how the case's inputs enter each model. A local explanation should identify a plausible source of disagreement and its limitations, rather than claim that the case establishes which model is best.
A gap is interpretable only after the scale is clear. For a model with a log link, the linear predictor is on the log-mean scale, while the prediction is on the response scale. For a logistic model, the linear predictor is a log-odds value and the usual reported prediction is a probability. Compare probabilities with probabilities, or expected costs with expected costs—not a log-odds value with a probability, for example.
If a model includes an offset or exposure adjustment, make sure the case's offset is treated consistently in the comparison. Similarly, apply any transformations, factor-level handling, and missing-value rules used when the models were fitted. The earlier material on offsets and weights and on cross-model comparisons establishes why these controls matter.
A Practical Case-Diagnosis Sequence
A disciplined sequence keeps the investigation tied to the case rather than to a preferred model. First quantify the gap. Then inspect the inputs and model forms that could produce it. Finally check whether the proposed explanation is supported by the data and remains plausible under small, defensible changes to the case.
Confirm the same response, prediction horizon, units, exposure, offset, and case-level input values. Identify whether outputs are probabilities, means, or another stated target.
Report each prediction and the signed and absolute gap. For positive predictions, a ratio can supplement the difference, but it should not replace a clearly stated primary scale.
For a GLM or penalized model, inspect the relevant transformed terms and coefficients. For a tree ensemble, inspect which splits or leaves apply to the case, to the extent the fitted model permits.
Look for a nonlinear region, an interaction, or an unusual input. Change one input at a time only when that change is meaningful, and do not treat the resulting prediction change as a causal effect.
Check whether comparable training cases support the explanation. Distinguish a model-structure clue from evidence that the model will predict well for this case.
Link-Scale Differences and Response-Scale Gaps
For a GLM with a log link, the conditional mean is \(\hat{\mu}_i=\exp(\hat{\eta}_i)\), where \(\hat{\eta}_i\) is the fitted linear predictor, including any applicable offset. The difference between two models' linear predictors is not their difference in predicted means. Instead, it determines a ratio of means.
This identity follows directly from the log-link definition and the exponent rule \(\exp(u)/\exp(v)=\exp(u-v)\). It explains why a modest link-scale difference can produce a consequential relative difference in the mean. It does not apply to every model: a tree ensemble's response prediction need not arise from a single log-link linear predictor. In a comparison, calculate response-scale predictions according to each model's actual prediction rule.
Worked Example: A Tree Threshold in Expected Cost
Method: compare response-scale predictions, then inspect the feature region used by the tree. Consider a hypothetical equipment-insurance portfolio. All three models predict expected annual cost in dollars for the same exposure. Let \(x\) be equipment age in years. A GLM has a log link and fitted predictor \(\hat{\eta}=\ln(1000)+0.02(x-40)\). A penalized log-link model has predictor \(\hat{\eta}_{EN}=\ln(1000)+0.012(x-40)\). A tree ensemble predicts $1,450 for ages below 65 and $2,050 for ages 65 or above. For a case with \(x=70\), calculate and investigate the predictions. These hypothetical model outputs are not industry data.
GLM prediction. Here \(x-40=70-40=30\), so the predictor is \(\ln(1000)+0.02(30)=\ln(1000)+0.60\). Therefore, \(\hat{\mu}_{GLM}=1000\exp(0.60)=1000(1.8221188)=\$1,822.12\), rounded to the nearest cent.
Penalized-model prediction. Its predictor is \(\ln(1000)+0.012(30)=\ln(1000)+0.36\). Thus, \(\hat{\mu}_{EN}=1000\exp(0.36)=1000(1.4333294)=\$1,433.33\), rounded to the nearest cent.
Tree-ensemble prediction. Since 70 is at least 65, the case follows the age-65-or-above rule, giving \(\hat{\mu}_{tree}=\$2,050.00\).
The tree exceeds the GLM by \(2050.00-1822.12=\$227.88\), and exceeds the penalized prediction by \(2050.00-1433.33=\$616.67\). The GLM exceeds the penalized prediction by \(1822.12-1433.33=\$388.79\). As a diagnostic, inspect the tree's boundary: it predicts $1,450 just below age 65 and $2,050 at age 65 or above, while the two regression predictions change smoothly with age. That discontinuity is a plausible source of the case-level gap. It is not proof that the true cost has a jump at age 65; check how much training data support the split and whether the pattern is stable.
Common mistake avoided: comparing \(0.60\), a GLM link-scale contribution, directly with $2,050. The GLM response prediction is obtained only after applying the exponential link and multiplying by the baseline mean.
Interactions Can Make a Case Exceptional
A main-effects regression model combines each feature's contribution without allowing one feature's effect to depend on another. An interaction term relaxes that restriction. A tree can also distinguish combinations through successive splits. Consequently, two models may agree for most combinations but disagree for a case with a particular combination of values.
For a logistic model, \(\hat{p}_i=1/(1+\exp(-\hat{\eta}_i))\), where \(\hat{p}_i\) is the predicted event probability and \(\hat{\eta}_i\) is the logit. A positive interaction coefficient changes the log-odds contribution when both corresponding indicators equal one. The probability change is not generally equal to that coefficient: the inverse-logit transformation is nonlinear.
Worked Example: A Combined Risk Feature
Method: calculate the case's probability under each fitted specification and inspect the interaction contribution. A hypothetical service provider models the probability that a scheduled inspection requires a follow-up visit. Let \(A=1\) indicate a high-load site and \(B=1\) indicate a recently installed sensor. For the case \(A=B=1\), a logistic GLM without an interaction has \(\hat{\eta}_{GLM}=-2+0.8A+0.6B\). A fitted penalized logistic model has \(\hat{\eta}_{EN}=-1.8+0.5A+0.4B+0.8AB\). The tree ensemble's prediction for the relevant terminal region is 0.5200. Calculate the predictions and identify what the structure suggests.
GLM. Substituting \(A=1\), \(B=1\), and \(AB=1\) into the GLM gives \(\hat{\eta}_{GLM}=-2+0.8+0.6=-0.6\). Thus \(\hat{p}_{GLM}=1/(1+\exp(0.6))=1/(1+1.8221)=0.3543\), to four decimal places.
Penalized model. Its predictor is \(-1.8+0.5+0.4+0.8=-0.1\). Therefore \(\hat{p}_{EN}=1/(1+\exp(0.1))=1/(1+1.1052)=0.4750\), to four decimal places.
Tree ensemble and gaps. The tree prediction is \(0.5200\). It exceeds the GLM prediction by \(0.5200-0.3543=0.1657\), or 16.57 percentage points, and exceeds the penalized prediction by \(0.5200-0.4750=0.0450\), or 4.50 percentage points. The penalized model's \(0.8AB\) term raises the logit for the joint \(A=B=1\) case; the main-effects GLM has no such term. A tree can also separate this combination into a distinct region. These structures offer a plausible explanation, but the case alone cannot establish that an interaction is reliable or that follow-up is caused by either feature.
Common mistake avoided: calling the interaction a 0.8 increase in probability. It is a 0.8 increase in the linear predictor when \(A=B=1\); the corresponding probability is calculated through the inverse-logit function.
Unusual Inputs and the Limits of Local Explanations
An input can be influential for a particular disagreement because it is extreme, unusual in combination with other inputs, or outside the range represented in training data. A linear predictor continues its specified trend beyond the observed range, whereas a tree prediction typically comes from terminal regions created using training observations. The resulting disagreement can therefore be a warning about extrapolation rather than evidence that the tree has discovered the correct behavior.
A useful check is to compare the case's feature values with the training ranges and to see how predictions change under a small, plausible perturbation. If changing one value greatly changes the regression prediction while barely changing the tree prediction, that identifies sensitivity to the input and model form. It does not show that the feature is erroneous, nor that changing it would cause a real-world outcome to change. For a derived or categorical input, also verify that its coding and level are valid.
Worked Example: An Age Beyond the Training Range
Method: check support and compare a controlled input change. A hypothetical repair-cost model predicts an expected annual amount in dollars from equipment age \(x\). Training equipment ages range from 0 through 8 years. For this example, a GLM with a log link predicts \(\hat{\mu}_{GLM}=500\exp(0.10(x-4))\), a penalized model predicts \(\hat{\mu}_{EN}=500\exp(0.05(x-4))\), and the relevant tree-ensemble terminal region predicts $700. Consider an \(x=12\) case whose observed annual amount is $690. All figures are hypothetical.
At age 12. The GLM exponent is \(0.10(12-4)=0.80\), so its prediction is \(500\exp(0.80)=500(2.2255409)=\$1,112.77\). The penalized exponent is \(0.05(12-4)=0.40\), so its prediction is \(500\exp(0.40)=500(1.4918247)=\$745.91\). The tree predicts $700.00. The GLM exceeds the penalized model by $366.86; the penalized model exceeds the tree by $45.91. Compared with the observed $690, the absolute errors are $422.77, $55.91, and $10.00, respectively.
Check sensitivity and support. At age 8, the upper end of the training range, the GLM gives \(500\exp(0.10(8-4))=500\exp(0.40)=\$745.91\), and the penalized model gives \(500\exp(0.05(8-4))=500\exp(0.20)=500(1.2214028)=\$610.70\). The tree remains at $700.00 under the stated terminal-region rule. The GLM-penalized gap thus grows from \(745.91-610.70=\$135.21\) at age 8 to $366.86 at age 12. Since age 12 is outside the observed range, the widening gap is consistent with differing extrapolation from the two log-linear formulas and the tree prediction rule. It does not establish which prediction is appropriate beyond the data.
Common mistake avoided: selecting the tree solely because its prediction is closest to $690 for this one case. The observed amount is useful context, but a single outcome cannot establish comparative predictive performance. First flag the unsupported age and explain that conclusions there are uncertain.
Turning a Gap into a Defensible Diagnosis
A strong case-level diagnosis states the comparison, the mechanism, and the evidence separately. For example: “For this case, the ensemble predicts more than the GLM. The case lies above a tree split that raises the ensemble prediction, while the GLM changes smoothly with the feature. Training support near the split should be checked before treating that difference as a stable threshold effect.” This wording avoids claiming that the case proves the true relationship is discontinuous.
The same discipline applies to influential inputs. Say that an extreme input makes the fitted predictions sensitive or involves extrapolation; do not label the record an error without checking its source. A useful diagnosis should survive basic verification of the input values, transformations, factor levels, and prediction scale. It should also acknowledge that a plausible local explanation is not a substitute for validation across cases.
Check Your Understanding
For each original problem, identify the requested gap or diagnosis. Monetary amounts are hypothetical.
- A log-link GLM predicts \(200\exp(0.10x)\), and a penalized log-link model predicts \(200\exp(0.06x)\), where \(x=5\). Calculate both expected-cost predictions and the GLM-minus-penalized gap, to the nearest cent. (Answer: GLM $329.74; penalized model $269.97; gap $59.77.)
- A logistic GLM has \(\hat{\eta}=-2+0.7A+0.5B\). For \(A=B=1\), calculate its predicted probability and the gap from a tree prediction of 0.4400. Give probabilities to four decimal places. (Answer: GLM probability 0.3100; tree-minus-GLM gap 0.1300, or 13.00 percentage points.)
- A case has predictions of $420, $390, and $610 from a GLM, penalized model, and tree ensemble. What is the maximum pairwise absolute gap, and which two models produce it? (Answer: $220, between the tree ensemble and penalized model.)
- Training values of a numeric feature range from 2 to 10. A case has value 14; the GLM prediction rises from $800 at value 10 to $1,200 at value 14, while the tree prediction is $900 at both values. Name the primary diagnostic concern and one appropriate check. (Answer: the case is outside the training range, so the GLM may be extrapolating; check input validity and support for comparable cases before interpreting the disagreement.)
- A penalized logistic model has predictor \(-2+0.4A+0.3B+0.9AB\), where \(A=B=1\). A main-effects-only model has predictor \(-2+0.4A+0.3B\). Calculate the penalized model's probability to four decimals and identify the interaction's contribution to the predictor for this case. (Answer: probability 0.4013; the interaction adds 0.9 to the linear predictor.)