Why Check Fit by Segment?
A Poisson model can produce a plausible portfolio-wide average while missing important differences among parts of the portfolio. For example, a model might predict the total number of claims accurately but consistently underpredict one group and overpredict another. Comparing observed and fitted counts by segment helps reveal such patterns before a fitted frequency is used for decisions.
A segment may be a policy class, territory, or another group defined for analysis. Its exposure measures how much risk was observed. The basic comparison is between the segment’s observed count and the count predicted by the model for that exposure. The purpose is not to treat every small difference as a model failure: random variation is expected. Instead, the question is whether the size and pattern of the differences are reasonably consistent with the model.
This model assumes that the counts being compared are independent, that each segment’s mean is correctly represented by its fitted count, and that the count variance equals the mean. The fitted mean must use the same exposure unit as the frequency. If a model includes rating variables or other predictors, its fitted frequency can differ by segment; the fit check then asks whether the resulting fitted counts agree adequately with observed counts.
Residuals and Goodness-of-Fit Measures
The raw difference \(N_j-\widehat{\mu}_j\) is not enough to compare segments fairly. A difference of five claims is much larger relative to expectation when the fitted count is ten than when it is one hundred. The Pearson residual scales the difference by the Poisson standard deviation.
A positive residual means the model underpredicts the count; a negative residual means it overpredicts. Squaring the residuals gives the Pearson goodness-of-fit statistic \(X^2\), which measures the total scaled discrepancy across the segments. A large value indicates that the observed counts are collectively far from their fitted means relative to the variation allowed by the model.
When the model is fitted to the same \(J\) segment counts using \(p\) estimated parameters, the approximate degrees of freedom are \(J-p\), provided the fitted model and cells meet the conditions for the approximation. The number \(p\) counts estimated model parameters that constrain the fitted means. A statistic should therefore be interpreted alongside its degrees of freedom, not in isolation. The comparison to a chi-square distribution is approximate, particularly when expected counts are small.
A second diagnostic is the Poisson deviance. It compares the fitted model with a saturated model, which assigns each cell its observed count as its fitted mean. With the convention that \(N_j\log(N_j/\widehat{\mu}_j)=0\) when \(N_j=0\), the deviance is:
Both \(X^2\) and \(D\) are nonnegative, and both are zero when every observed count equals its fitted count. They are often similar when the discrepancies are modest, but they need not be identical. They are diagnostics, not proof that a particular model is correct or incorrect.
Why Pearson Residuals Use the Fitted Mean
For a Poisson random variable with mean \(\mu_j\), the variance is also \(\mu_j\). Consequently, the standard deviation of the count is \(\sqrt{\mu_j}\). If the model is appropriate, an observed count typically fluctuates around its expected count on this standard-deviation scale. Replacing the unknown \(\mu_j\) with its fitted value gives the Pearson residual.
Thus, a residual of \(2\) represents a difference of about two fitted standard deviations, whether the segment has a small or large expected count. This scaling is the reason the Pearson statistic is more informative than adding unscaled count differences.
Worked Examples
Worked Example: Checking a Common-Frequency Model
A hypothetical portfolio has three independent segments. Their exposures are 100, 200, and 300 policy-years, and their observed counts are 8, 25, and 27. Assume a common Poisson frequency across segments and estimate it from the combined data. Calculate the fitted counts, Pearson residuals, goodness-of-fit statistic, and approximate degrees of freedom. State what the result suggests.
Method and estimate. The total observed count is \(8+25+27=60\), and total exposure is \(100+200+300=600\) policy-years. The estimated common frequency is \(60/600=0.10\) claims per policy-year. Under the assumed model, each segment count is Poisson with mean equal to its exposure times \(0.10\).
Fitted counts. The fitted counts are \(100(0.10)=10\), \(200(0.10)=20\), and \(300(0.10)=30\).
Residuals and statistic. The residuals are \((8-10)/\sqrt{10}=-0.6325\), \((25-20)/\sqrt{20}=1.1180\), and \((27-30)/\sqrt{30}=-0.5477\), rounded to four decimals. Their squared values are \(0.4000\), \(1.2500\), and \(0.3000\). Therefore \(X^2=0.4000+1.2500+0.3000=1.9500\).
Degrees of freedom and check. There are \(J=3\) segment counts and \(p=1\) estimated frequency parameter, so the approximate degrees of freedom are \(3-1=2\). The fitted counts sum to \(10+20+30=60\), matching the observed total. A statistic of 1.95 on 2 degrees of freedom does not indicate a large discrepancy; it does not establish that the model is true, but these counts do not provide a strong fit warning.
Method and mistake avoided: This is a Pearson goodness-of-fit check. A common mistake is to compare each observed count with the same expected count. The expected count must reflect each segment’s exposure, so the fitted counts here are 10, 20, and 30—not three copies of an overall average count.
Worked Example: Detecting Opposing Segment Errors
A hypothetical model assigns three segments relative fitted frequencies in the ratio \(1:2:3\). The scale is estimated from the data. Each segment has exposure 100 policy-years, and the observed counts are 18, 30, and 12. Calculate the fitted counts and Pearson statistic, then assess the model’s fit by segment.
Method and scale. Write the fitted frequencies as \(c\), \(2c\), and \(3c\). Since the exposures are equal, the fitted counts have the same ratio. The observed total is \(18+30+12=60\), so the fitted counts must sum to 60. The scale factor is \(c=60/(100+200+300)=0.10\), where the denominator is the exposure-weighted total for the three relative rates.
Fitted counts. The three means are \(100(0.10)=10\), \(100(0.20)=20\), and \(100(0.30)=30\). They sum to 60, as required.
Residuals and statistic. The Pearson residuals are \((18-10)/\sqrt{10}=2.5298\), \((30-20)/\sqrt{20}=2.2361\), and \((12-30)/\sqrt{30}=-3.2863\). The contributions to \(X^2\) are \(64/10=6.4000\), \(100/20=5.0000\), and \(324/30=10.8000\). Thus \(X^2=22.2000\).
Degrees of freedom and interpretation. The relative rates are fixed and only the common scale was estimated, so \(p=1\) and the approximate degrees of freedom are \(3-1=2\). The overall total is matched exactly, but the model overpredicts the third segment and underpredicts the first two. The large Pearson statistic flags a strong disagreement with the segment pattern. Matching the portfolio total is not enough to show that the segment-level model fits.
Method and mistake avoided: This uses the Pearson statistic with one estimated scale parameter. The trap is to conclude that fit is good because observed and fitted totals both equal 60. That total agreement was imposed by estimating the scale; the segment residuals still show substantial disagreement.
Worked Example: Separating Dispersion from a Residual Pattern
A hypothetical model predicts a Poisson mean of 20 claims in each of four equal-exposure segments, with the fitted means supplied by a model whose parameters were estimated elsewhere. The observed counts, in a meaningful low-to-high risk order, are 8, 16, 24, and 32. Calculate the residuals and Pearson statistic, and explain what the ordered pattern adds to the diagnosis.
Model and residuals. Each count is assumed to follow a Poisson distribution with mean and variance 20, independently of the other segment counts. The fitted standard deviation in each segment is \(\sqrt{20}=4.4721\). The residuals are \((8-20)/\sqrt{20}=-2.6833\), \((16-20)/\sqrt{20}=-0.8944\), \((24-20)/\sqrt{20}=0.8944\), and \((32-20)/\sqrt{20}=2.6833\).
Pearson statistic. The four contributions are \(144/20=7.2000\), \(16/20=0.8000\), \(16/20=0.8000\), and \(144/20=7.2000\). Thus \(X^2=7.2000+0.8000+0.8000+7.2000=16.0000\).
Interpretation and check. The residuals progress from negative to positive as risk order increases. This is a systematic pattern: the model appears to underpredict higher-risk segments and overpredict lower-risk segments. The statistic also shows that the total scaled discrepancy is large. Because the means were supplied from a model estimated elsewhere, no parameter was estimated from these four counts; the basic degrees-of-freedom count is \(4\), subject to the usual approximation conditions.
This pattern is evidence to investigate a missing or inadequate segment effect; it should not automatically be labelled random overdispersion. Underdispersion or overdispersion refers to variation relative to the Poisson mean-variance assumption, while a directional residual pattern can point to systematic misspecification. Both may be present, and neither diagnosis follows from one statistic alone.
Method and mistake avoided: This is a segment-level Pearson diagnostic. A common mistake is to call every large Pearson statistic “overdispersion” without inspecting the residuals. Here their ordered signs provide specific evidence of a systematic pattern.
Interpreting Patterns and Limits
A useful fit review combines the overall statistic with the segment residuals and knowledge of how segments were formed. Look for unusually large residuals, repeated positive or negative residuals among related groups, and sequences that change direction with a meaningful characteristic. A single large residual may reflect random variation, a data issue, or a real segment difference; a coherent pattern is a stronger reason to examine the model specification or data.
| Diagnostic finding | What it can suggest | What it does not establish by itself |
|---|---|---|
| Large \(X^2\) relative to degrees of freedom | More total discrepancy than the Poisson model typically allows | The cause of the discrepancy |
| Several residuals with the same sign in related segments | A systematic difference not captured by the fitted means | Which new variable or model form is correct |
| Many residuals more variable than expected, without an obvious pattern | Possible overdispersion, dependence, or unobserved heterogeneity | That a particular alternative distribution is appropriate |
| Small expected counts | Unreliable chi-square approximation and noisy individual residuals | That the model fails merely because a residual is large |
Overdispersion means the observed variation is greater than the Poisson model’s variance-equals-mean assumption allows. It can arise from omitted risk differences, dependence among counts, or other features not represented by the model. A large Pearson statistic can flag this issue, but it can also result from a systematic error in fitted segment means. Inspecting the residual pattern helps distinguish these possibilities.
When expected counts are small, the chi-square approximation can be poor and individual residuals can be unstable. A frequently used rule of thumb is to be cautious when expected counts are below about five; this is guidance, not a universal boundary. Combining cells may help only when the segments can reasonably be treated as comparable for the purpose of the check. Do not combine unlike segments merely to make a statistic look acceptable.
Exam trap: Do not confuse a segment’s observed frequency \(N_j/e_j\) with its observed count \(N_j\). The model’s expected count is \(e_j\widehat{\lambda}_j\), not simply \(\widehat{\lambda}_j\). Keep exposure units consistent, use the fitted mean in the residual denominator, and account for estimated parameters when stating approximate degrees of freedom.
Check Your Understanding
For each problem, use the stated Poisson model and report calculations to the precision shown in the answer.
- A segment has 80 earned exposure units, fitted frequency \(0.15\) claims per unit, and observed count 15. Find its fitted count, Pearson residual, and Pearson contribution. (Answer: fitted count \(12\); residual \(0.8660\); contribution \(0.7500\).)
- Three independent segments have exposures 100, 150, and 250 and observed counts 12, 18, and 30. Fit one common Poisson frequency from these data and calculate the fitted counts and Pearson statistic. (Answer: fitted frequency \(0.1200\); fitted counts \(12,18,30\); \(X^2=0\).)
- A model gives fitted means 8, 16, and 24 for three independent segments; observed counts are 12, 12, and 24. Calculate the three Pearson residuals and the total Pearson statistic. (Answer: residuals \(1.4142,-1.0000,0.0000\); \(X^2=3.0000\).)
- Three equal-exposure segments have observed counts 6, 14, and 30. Estimate one common Poisson frequency from the combined counts and exposure, giving fitted means \(16.6667\) in each segment. Calculate \(X^2\) and its approximate degrees of freedom. (Answer: \(X^2=17.9200\); degrees of freedom \(2\).)
- A model has fitted means 10, 20, and 30, and observed counts 5, 20, and 35. The fitted means use one estimated common scale parameter and otherwise fixed relative rates. Calculate the Pearson statistic and approximate degrees of freedom. (Answer: \(X^2=3.3333\); degrees of freedom \(2\).)