Why Residual Structure Can Reveal Missing Predictor Effects
A GLM can use the appropriate response distribution and link function yet still describe the rating variables inadequately. For example, a model may force a straight-line effect of vehicle age on log expected claim frequency when the true pattern bends, or it may assume that the effect of annual mileage is the same in every territory when those effects differ. These are predictor-structure problems: the model is missing a nonlinear effect, an interaction, or both.
Residuals help locate such problems. A residual pattern that changes systematically across a continuous predictor suggests that the model’s fitted relationship may be too restrictive. A pattern that changes across combinations of rating factors can suggest an omitted interaction. Neither pattern proves a particular term is correct; it identifies a model feature worth investigating and testing.
Residuals for Diagnosing Predictor Structure
For a claim-count GLM with a Poisson response and log link, a common specification is
where \(Y_i\) is the claim count, \(e_i\) is exposure, and \(\eta_i\) is the linear predictor formed from rating variables. The term \(\log(e_i)\) is an offset when exposure is known and its coefficient is fixed at one. Under this Poisson model, \(\operatorname{Var}(Y_i\mid x_i)=\mu_i\). Given a fitted mean \(\hat{\mu}_i\), the Pearson residual is
A positive residual means the observed count exceeds its fitted mean; a negative residual means it falls below. The denominator scales the difference by the model’s expected standard deviation under the Poisson variance assumption. A residual plot can show these values against a continuous predictor, or summaries of them across combinations of rating factors.
For grouped data, it is often useful to compare observed and expected totals within a group. If group \(g\) contains observations with fitted means \(\hat{\mu}_i\), define \(O_g=\sum_{i\in g}Y_i\) and \(E_g=\sum_{i\in g}\hat{\mu}_i\). The grouped Pearson discrepancy is \((O_g-E_g)/\sqrt{E_g}\) under the Poisson model when the counts are independent. Grouping can make a broad pattern easier to see, but it can also conceal variation within a group or make the result depend on the chosen boundaries.
Reading Patterns Across a Continuous Predictor
When residuals are plotted against a continuous rating variable, a correctly specified mean structure should not leave a strong systematic trend in the conditional mean of the residuals. A smooth curve through residuals that bends from positive to negative and back, or the reverse, can indicate that a linear term is inadequate. A monotone trend can instead indicate a missing or incorrectly specified main effect, an incorrect link, or another modeling issue. Residual shape is a clue, not a unique diagnosis.
A practical approach is to inspect the plot and then summarize the pattern across sensible ranges of the predictor. The summaries should retain the exposure or expected-count scale: a raw count difference is not comparable across groups with very different expected counts. Use the model’s predicted means to calculate expected totals, and examine both the sign and relative size of the discrepancies.
Worked Example: A Curved Residual Pattern Across Vehicle Age
A hypothetical insurer uses a Poisson frequency model with a log link and exposure offset. The model’s straight-line vehicle-age effect is fitted on a training sample. For an independent validation sample, the model predicts the following expected claim counts in five age bands. The observed validation counts are shown alongside them. Treat the displayed expected values as fixed conditional predictions; claim counts are assumed independent Poisson variables under the model.
| Vehicle-age band | Expected \(E_g\) | Observed \(O_g\) |
|---|---|---|
| 1 | 25 | 33 |
| 2 | 36 | 34 |
| 3 | 49 | 43 |
| 4 | 64 | 65 |
| 5 | 81 | 97 |
Step 1: Calculate each grouped discrepancy. For each band, use \(O_g-E_g\) and divide by \(\sqrt{E_g}\):
Step 2: Read the pattern. The first and fifth bands have positive discrepancies, while the second and third are negative and the fourth is close to zero. The observed-to-expected ratios are \(33/25=1.320\), \(34/36=0.944\), \(43/49=0.878\), \(65/64=1.016\), and \(97/81=1.198\). The broad pattern is positive at the ends and negative in the middle, which is consistent with curvature that a straight-line age effect may not capture.
Step 3: State the diagnostic limit. This pattern motivates examining a nonlinear age specification, such as a spline or a prespecified curved term, and comparing its behavior on validation data. It does not establish that curvature is the cause: random variation, omitted predictors correlated with age, and other misspecification could also contribute. The method avoids treating one residual plot as proof of a particular model form.
Reading Patterns Across Combinations of Rating Factors
An interaction can be missed when residuals are summarized only by one rating factor at a time. For instance, positive residuals in one combination of territory and mileage may be balanced by negative residuals in another. Marginal totals can then look calibrated even though the model misses the pattern within combinations.
For two categorical factors, inspect a table of observed and expected values for their cross-classification. For one continuous predictor and one factor, compare residual patterns across ranges of the continuous predictor separately by factor level. If the curves differ in direction or shape, the predictor’s effect may depend on the factor. Use enough observations in each group for the summaries to be informative; sparse cells can produce volatile discrepancies.
Worked Example: A Factor Interaction Hidden by Marginal Totals
A hypothetical insurer evaluates a Poisson frequency model with a log link on independent validation data. Its predictions come from a model containing separate main effects for territory and vehicle use, but no interaction. Each of the four combinations has an expected count of 100. Assume independent Poisson counts conditional on the predictors.
| Territory | Personal use: observed | Personal use: expected | Business use: observed | Business use: expected |
|---|---|---|---|---|
| North | 120 | 100 | 80 | 100 |
| South | 80 | 100 | 120 | 100 |
Step 1: Calculate the cell Pearson discrepancies. Each expected count has square root 10, so the four residuals are \((120-100)/10=2\), \((80-100)/10=-2\), \((80-100)/10=-2\), and \((120-100)/10=2\).
Step 2: Check marginal calibration. The North observed total is \(120+80=200\), equal to its expected total \(100+100=200\). The South observed total is \(80+120=200\), also equal to expectation. Personal-use totals are \(120+80=200\), and business-use totals are \(80+120=200\); each equals its expected total. Thus, summaries by either factor alone show no discrepancy, while the cell residuals form a checkerboard pattern.
Step 3: Quantify the contrast. Under a log-link model with main effects only, the ratio of the North-to-South rate ratio for personal use to that for business use is one. The observed cell-rate contrast, since all exposures and expected counts are equal, is
Its log contrast is \(\log(2.25)\approx0.811\), rather than zero. This is evidence of a possible territory-by-use interaction in this validation sample. It is a diagnostic clue, not by itself a significance test or a guarantee that an interaction will generalize. The method avoids relying only on marginal residual summaries, which cancel in this example.
Why the Residual Expectation Needs Care
A useful justification explains why a fixed-mean residual should have no systematic conditional mean when the model is correct. Suppose \(m_i\) is a prediction fixed independently of the response being evaluated, and \(Y_i\mid x_i\) has mean \(\mu_i\). Then
The first equality follows because, conditional on \(x_i\) and the fixed prediction \(m_i\), the denominator and prediction are constants. If the prediction equals the true conditional mean, the expression is zero. If it does not, the sign reflects whether the true mean is above or below the prediction. This gives the fixed-prediction intuition behind residual diagnostics.
When \(\hat{\mu}_i\) is estimated using the same response \(Y_i\), however, it is generally random and dependent on \(Y_i\). It cannot automatically be taken outside the conditional expectation. For example, in a saturated Poisson fit with one fitted mean per observation, \(\hat{\mu}_i=Y_i\) (for positive counts in the ordinary unconstrained fit), so the fitted Pearson residual is zero. It would be incorrect to infer from this that the expectation equals \((\mu_i-\hat{\mu}_i)/\sqrt{\hat{\mu}_i}\). Same-data residuals remain useful, but their patterns can be affected by parameter fitting and leverage. Independent validation predictions provide a direct setting for the fixed-prediction argument; other approaches include checking whether the pattern persists after refitting a plausible alternative model.
A Disciplined Diagnostic Workflow
A residual plot should lead to a targeted modeling question rather than an automatic expansion of the rating plan. The following sequence keeps the interpretation tied to the observed structure.
Use residuals or observed-versus-expected summaries appropriate to the response distribution and the earlier material on residuals. For Poisson counts, Pearson residuals use the fitted mean in the denominator.
Look for systematic trends or curvature against each relevant continuous predictor, then compare those patterns across important rating-factor levels. Do not rely only on one-way summaries if an interaction is plausible.
Examples include a spline for a continuous predictor or an interaction between two rating factors. The proposed term should correspond to the pattern, rather than adding terms indiscriminately.
Check whether the residual structure is reduced and whether the fitted effects are stable and interpretable. Where possible, assess performance on independent data so that improvement is not merely a consequence of fitting the same responses that generated the diagnostic.
Common Mistakes and Practical Interpretation
- Calling every trend “nonlinearity.” A monotone residual trend may indicate a missing main effect, while a curved pattern may indicate an inadequate functional form. Correlated omitted predictors or link misspecification can also create structure.
- Checking only marginal summaries. Opposing cell discrepancies can cancel, as in the territory-by-use example. Inspect relevant combinations when an interaction is plausible.
- Comparing unscaled count differences. A difference of five claims has a different scale when the expected count is 10 than when it is 1,000. Consider expected counts, exposure, and the residual scale.
- Treating same-data fitted means as fixed. The identity for the expected residual applies when the prediction is fixed conditional on the information used for the expectation, such as an independent validation prediction. A fitted mean estimated from the same response is generally dependent on it.
- Overinterpreting sparse groups. A few observations can produce extreme cell residuals by chance. Check group size and exposure before concluding that a complex rating term is warranted.
The objective is not to make every plotted residual exactly zero. It is to find reproducible, meaningful structure that points to a missing predictor form, then determine whether a defensible model change captures it. This diagnostic reasoning is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.
Check Your Understanding
For each problem, use the stated predictions and assumptions to interpret the residual pattern. Answers are provided for checking.
- A Poisson model gives a fixed validation expected count of 16 in a predictor band, and the observed count is 12. Calculate the grouped Pearson discrepancy and the observed-to-expected ratio. (Answer: Pearson discrepancy \(=-1.000\); observed-to-expected ratio \(=0.750\).)
- Three ordered bands of a continuous rating variable have fixed expected counts \(25,49,81\) and observed counts \(30,42,90\). Calculate the three grouped Pearson discrepancies and describe the pattern. (Answer: \(1.000,-1.000,1.000\); positive at both ends and negative in the middle, consistent with possible curvature.)
- Two factors form four cells, each with expected count 50. Observed counts are 60 and 40 in the first factor level, and 40 and 60 in the second. Give the four Pearson discrepancies, state whether the one-factor margins are calibrated, and calculate the observed interaction contrast. (Answer: \(1.414,-1.414,-1.414,1.414\); both one-factor margins equal their expected totals; contrast \(=(60\cdot60)/(40\cdot40)=2.25\).)
- A model trained on one sample predicts 40 claims in each of three mileage bands for each of two territories. Validation counts are 48, 40, and 32 in Territory A, and 32, 40, and 48 in Territory B. Calculate the six cell Pearson discrepancies and explain why a pooled summary by mileage misses the pattern. (Answer: Territory A: \(1.265,0,-1.265\); Territory B: \(-1.265,0,1.265\); each pooled mileage total is 80 against an expected 80, so the opposing territory-specific patterns cancel.)
- In one sentence, state when the fixed-mean conditional expectation formula for a Pearson residual applies, and why it may fail for an in-sample fitted mean. (Answer: It applies when the prediction is fixed conditional on the information used for the expectation; an in-sample fitted mean can depend on the response, so it cannot generally be treated as a constant.)