Grouped Residual Diagnostics for Rating Classes

Why Group Residuals by Rating Class?

A model can have an acceptable overall fit and still systematically miss for a particular rating class. For example, positive observed-minus-fitted residuals concentrated in one class indicate that observed outcomes there tend to exceed the model’s predictions. Negative residuals indicate the reverse. Grouping residuals by a rating class can reveal that structure more clearly than examining individual residuals one at a time.

A group summary is a diagnostic, not an automatic test that a rating factor should be added or changed. Its meaning depends on the response distribution, the fitted model, the class sizes, and whether the same class structure was used to fit the model. The calculations below make those distinctions explicit. This topic is aligned with the CAS Exam 8 (Advanced Ratemaking) syllabus.

Definition: For observation \(i\), let \(y_i\) be the observed response, \(\hat{\mu}_i\) its fitted mean, and \(V(\hat{\mu}_i)\) the variance function evaluated at that fitted mean. For a rating class \(g\), define its aggregate observed-minus-fitted residual as \(R_g=\sum_{i\in g}(y_i-\hat{\mu}_i)\). A positive \(R_g\) means the model underpredicts the class total; a negative \(R_g\) means it overpredicts the class total.

Three Related Group Summaries

The raw aggregate \(R_g\) reports the direction and size of the total difference in the response’s original units. Its size is affected by the number of observations in the class and, for many models, by the scale of the expected outcomes. It is therefore useful to report alongside standardized summaries rather than interpret in isolation.

For a generalized linear model, the Pearson residual for observation \(i\) is the observed-minus-fitted residual divided by its model-based standard deviation. The dispersion parameter is \(\phi\), so the variance of observation \(i\) is \(\phi V(\hat{\mu}_i)\). The sum of these record-level Pearson residuals is one possible class summary, but it is not the same calculation as standardizing the class total.

r_{Pi}=\frac{y_i-\hat{\mu}_i}{\sqrt{\phi V(\hat{\mu}_i)}}, \qquad P_g=\sum_{i\in g} r_{Pi}

A different summary standardizes the aggregate residual by the model-based standard deviation of the class total. Assuming the responses are conditionally independent and treating the fitted means and dispersion as fixed for this variance calculation, the variance of the total residual is the sum of the individual variances. This gives a standardized aggregate residual:

Z_g= \frac{R_g}{\sqrt{\phi\sum_{i\in g}V(\hat{\mu}_i)}}

The denominator is not generally the sum of the individual standard deviations. It is the square root of the sum of their variances. This distinction matters whenever the observations have different fitted means or variance contributions. For a Poisson model, \(\phi=1\) and \(V(\hat{\mu}_i)=\hat{\mu}_i\), so the denominator is \(\sqrt{\sum_{i\in g}\hat{\mu}_i}\).

Interpretation: \(R_g\) and \(Z_g\) always have the same sign because the denominator of \(Z_g\) is positive. The sum \(P_g\) has that same sign when all record-level observed-minus-fitted deviations in the class have the same sign. With mixed-sign deviations, \(P_g\) and \(R_g\) can have opposite signs because each Pearson residual has its own denominator.

Why the Standardized Aggregate Uses This Denominator

The standardization follows from the variance of a sum. Write the individual residual as \(e_i=y_i-\hat{\mu}_i\). If observations in class \(g\) are conditionally independent, the covariance between distinct residuals is zero, so the variance of their sum is the sum of their variances. Since the fitted mean is treated as fixed in this calculation, subtracting it does not change the variance of \(y_i\).

\begin{aligned} \operatorname{Var}(R_g) &=\operatorname{Var}\left(\sum_{i\in g}e_i\right)\\ &=\sum_{i\in g}\operatorname{Var}(e_i)\\ &=\phi\sum_{i\in g}V(\hat{\mu}_i). \end{aligned}

Dividing \(R_g\) by the square root of this variance expresses the class difference in approximate standard-deviation units. This is a model-based diagnostic scale, not a guarantee that \(Z_g\) has an exact standard normal distribution. In particular, fitted parameters are estimated from data, responses may not be independent, and a small or unusual class may not support a normal approximation. Interpret the standardized value in context rather than treating a particular cutoff as a universal pass-or-fail rule.

Worked Examples

Worked Example: When the Pearson Sum Reverses Direction

Scenario. In a hypothetical rating class, a Poisson frequency model gives fitted means of 100 and 1 for two policies. The observed claim counts are 102 and 0, respectively. Take \(\phi=1\). Calculate the class aggregate, the sum of Pearson residuals, and the standardized aggregate residual. Explain why the three results do not all have the same sign.

Method. For a Poisson model, each variance function is its fitted mean. First calculate the record-level observed-minus-fitted residuals:

For the first policy, \(102-100=2\). For the second policy, \(0-1=-1\). Therefore the class aggregate is \(R_g=2+(-1)=1\), so the model underpredicts the total by one claim.

Sum of Pearson residuals. The first Pearson residual is \(2/\sqrt{100}=0.2\). The second is \(-1/\sqrt{1}=-1\). Thus \(P_g=0.2-1=-0.8\), which is negative.

Standardized aggregate. The sum of the fitted means is \(100+1=101\), so \(Z_g=1/\sqrt{101}\approx 0.0995\). The standardized aggregate is positive, agreeing with \(R_g\), while the sum of Pearson residuals is negative.

As a further check, the class’s Pearson sum of squares is \(0.2^2+(-1)^2=0.04+1=1.04\). The sign reversal occurs because the negative deviation of one claim is divided by a much smaller standard deviation than the positive deviation of two claims. Common mistake avoided: assuming that a nonzero aggregate residual must have the same sign as the sum of Pearson residuals.

Worked Example: Comparing Two Poisson Rating Classes

Scenario. A hypothetical frequency model is reviewed for two rating classes. In the Urban class, observed counts are \(8,11,9\) and fitted means are \(10,10,10\). In the Rural class, observed counts are \(14,13,15\) and fitted means are \(12,12,12\). Assume independent Poisson responses, so \(\phi=1\) and \(V(\hat{\mu}_i)=\hat{\mu}_i\). For each class, calculate \(R_g\), \(P_g\), \(Z_g\), and the sum of squared Pearson residuals.

Urban class: aggregate. The three observed-minus-fitted values are \(8-10=-2\), \(11-10=1\), and \(9-10=-1\). Their sum is \(R_{\text{Urban}}=-2+1-1=-2\), indicating overprediction of the class total by two claims.

Urban class: Pearson summaries. The Pearson residuals are \(-2/\sqrt{10}\approx-0.6325\), \(1/\sqrt{10}\approx0.3162\), and \(-1/\sqrt{10}\approx-0.3162\). Their sum is \(P_{\text{Urban}}\approx-0.6325\). The sum of squares is \(0.4+0.1+0.1=0.6\). The standardized aggregate is \(-2/\sqrt{10+10+10}=-2/\sqrt{30}\approx-0.3651\).

Rural class: aggregate. The deviations are \(14-12=2\), \(13-12=1\), and \(15-12=3\), so \(R_{\text{Rural}}=2+1+3=6\). The model underpredicts the class total by six claims.

Rural class: Pearson summaries. The Pearson residuals are \(2/\sqrt{12}\approx0.5774\), \(1/\sqrt{12}\approx0.2887\), and \(3/\sqrt{12}\approx0.8660\). Their sum is \(P_{\text{Rural}}\approx1.7321\). The sum of squares is \(4/12+1/12+9/12=14/12\approx1.1667\). The standardized aggregate is \(6/\sqrt{12+12+12}=6/6=1.0000\).

The class patterns point in opposite directions: the model overpredicts the Urban total and underpredicts the Rural total. Across both classes, observed counts total \(8+11+9+14+13+15=70\), and fitted means total \(30+36=66\), giving an overall residual of \(70-66=4\). That positive overall difference does not remove the contrasting class patterns. Common mistake avoided: relying on the overall total alone to conclude that predictions are equally well calibrated across classes.

Worked Example: Ranking Class-Level Aggregate Deviations

Scenario. A hypothetical Poisson model is assessed in three rating classes. Class A has observed counts \(7,7\) and fitted means \(5,5\). Class B has observed counts \(21,19\) and fitted means \(20,20\). Class C has observed counts \(2,3\) and fitted means \(4,4\). Assume independent Poisson responses. Calculate each class’s aggregate residual and standardized aggregate residual, then identify the strongest directional signal on this scale.

Class A. The deviations are \(7-5=2\) and \(7-5=2\), so \(R_A=4\). The sum of fitted means is \(10\), giving \(Z_A=4/\sqrt{10}\approx1.2649\). Both deviations are positive, so the class total is underpredicted.

Class B. The deviations are \(21-20=1\) and \(19-20=-1\), so \(R_B=0\). The sum of fitted means is \(40\), giving \(Z_B=0/\sqrt{40}=0\). The two deviations cancel in the aggregate; this does not mean that both observations were predicted exactly.

Class C. The deviations are \(2-4=-2\) and \(3-4=-1\), so \(R_C=-3\). The sum of fitted means is \(8\), giving \(Z_C=-3/\sqrt{8}\approx-1.0607\). The class total is overpredicted.

The absolute standardized aggregate is largest for Class A: \(1.2649\), compared with \(0\) for Class B and \(1.0607\) for Class C. Thus Class A has the strongest directional signal among these three using this particular diagnostic. This ranking is descriptive; it does not by itself establish statistical significance or justify changing the rating structure. Common mistake avoided: ranking raw totals without accounting for their different model-based variances.

Interpreting Grouped Patterns Carefully

A useful class-level review reports the class definition, number of observations, observed total, fitted total, aggregate residual, and a standardized summary. Showing the fitted and observed totals alongside the residual preserves practical context: a difference of five claims has a different scale in a class expected to produce 20 claims than in one expected to produce 2,000. The standardized aggregate helps account for this variance scale under the model, but it does not replace inspection of the underlying data.

A positive \(R_g\) or \(Z_g\) signals underprediction of the class total; a negative value signals overprediction. A value near zero means the signed deviations approximately cancel at the class-total level. It does not prove that individual predictions are accurate: large positive and negative residuals can cancel. Likewise, the sum of Pearson residuals should be interpreted as its own summary, not substituted for \(Z_g\).

Also consider how the model was fitted. If the rating class itself was included as a model factor, in-sample residual totals by that factor can be constrained by the fitting equations, depending on the model, link, weights, and estimation method. A small in-sample class aggregate may therefore reflect how the model was fitted rather than independently confirming calibration. Grouping by a class not directly used to fit the model can provide a different diagnostic view, but it still does not establish that the pattern will persist in new data.

These summaries also rely on their variance calculation being reasonable. The formula for \(Z_g\) assumes conditionally independent responses and uses the model’s variance function and dispersion. If responses are dependent, or if the stated variance model is inappropriate, the denominator may not represent the actual variability of the class total. A small standardized aggregate is then not reliable evidence of good fit. Further interpretation should be based on the model and data structure, not just the arithmetic.

Common Exam Traps

Calling a positive residual overprediction. With \(R_g=\text{observed}-\text{fitted}\), a positive value means observed exceeds fitted: the model underpredicts.
Adding Pearson residuals and treating the result as standardized \(R_g\). Record-specific denominators make \(P_g\) a different quantity from \(Z_g\).
Assuming \(P_g\) and \(R_g\) always share a sign. They do if all deviations have one sign, but mixed-sign deviations can yield opposite signs, as the first example demonstrates.
Using the sum of standard deviations in the aggregate denominator. Under independence, the standard deviation of a sum is the square root of the sum of variances.
Treating a zero aggregate as proof of accurate individual predictions. Positive and negative deviations can cancel exactly.
Overstating an in-sample diagnostic. A class used directly in model fitting may have residual constraints; a small class total is not necessarily independent evidence of calibration.

Key takeaway: Start with the signed class total \(R_g\), then standardize it using the square root of the sum of model-based variances when the independence assumption is appropriate. Keep this standardized aggregate distinct from the sum of record-level Pearson residuals, and interpret both in light of class structure and model fitting.

Check Your Understanding

For each problem, use the stated model and assumptions. Treat displayed fitted means as fixed for the requested calculations.

A hypothetical Poisson rating class has observed counts \(6,9\) and fitted means \(7,8\). Calculate \(R_g\), \(Z_g\), and the Pearson sum of squares. (Answer: \(R_g=0\), \(Z_g=0\), Pearson sum of squares \(=1/7+1/8\approx0.2679\).)
For three independent Poisson observations in one hypothetical class, observed counts are \(5,7,11\) and fitted means are \(6,6,10\). Calculate \(R_g\), \(P_g\), and \(Z_g\). (Answer: \(R_g=1\), \(P_g=1/\sqrt{10}\approx0.3162\), \(Z_g=1/\sqrt{22}\approx0.2132\).)
A hypothetical Gaussian model has three independent responses with constant variance \(\phi=4\). Their observed values are \(18,13,24\), and their fitted means are \(16,15,20\). Calculate the aggregate residual, standardized aggregate, and sum of squared Pearson residuals. (Answer: \(R_g=4\), \(Z_g=4/\sqrt{12}\approx1.1547\), Pearson sum of squares \(=6\).)
Two hypothetical Poisson rating classes have the following values: Class A observed counts \(4,5\), fitted means \(5,5\); Class B observed counts \(9,12\), fitted means \(8,10\). Calculate each class’s aggregate and standardized aggregate, and state which class is underpredicted. (Answer: Class A \(R_A=-1,\ Z_A=-1/\sqrt{10}\approx-0.3162\); Class B \(R_B=3,\ Z_B=3/\sqrt{18}\approx0.7071\); Class B is underpredicted.)
A hypothetical Poisson class has observed counts \(102,0\) and fitted means \(100,1\). Calculate \(R_g\), the sum of Pearson residuals, and \(Z_g\). Does the Pearson sum have the same sign as the aggregate? (Answer: \(R_g=1\), Pearson sum \(=-0.8\), \(Z_g=1/\sqrt{101}\approx0.0995\); no, the Pearson sum has the opposite sign.)

What You'll Learn