Calibration Within Consequential Subgroups

Why Calibration Needs a Subgroup View

A model can match actual outcomes on average across a portfolio and still systematically overpredict for one group and underpredict for another. That matters when predictions inform decisions for individuals or segments: pooled agreement does not show whether the model is reliable for the people represented by each consequential subgroup.

Calibration assesses whether predicted values agree with observed outcomes. For subgroup analysis, the key question is whether that agreement also holds separately within groups relevant to the intended use. A subgroup might be defined by a categorical predictor, a service region, or a policy-relevant population characteristic. The definition should have a substantive reason; slicing data repeatedly until a striking pattern appears can produce unstable findings.

Definition: Let $y_i$ be the observed numeric outcome and $\hat{y}_i$ the model prediction for observation $i$. Define the residual as $e_i=y_i-\hat{y}_i$. For subgroup $g$, the mean residual $D_g$ is the subgroup's mean actual outcome minus its mean prediction. A positive $D_g$ indicates average underprediction; a negative $D_g$ indicates average overprediction. The subgroup is calibrated in the large in the evaluation sample when $D_g=0$.

For positive predictions and a positive sum of predictions, an observed-to-predicted ratio can also be reported. A ratio of 1 means the subgroup's total observed outcome equals its total predicted outcome; a ratio above 1 indicates underprediction in aggregate, and a ratio below 1 indicates overprediction. The difference $D_g$ is often easier to interpret in the outcome's units, while the ratio gives a relative comparison. A ratio is not useful when the prediction total is zero, and can be unstable when it is close to zero.

D_g=\frac{1}{n_g}\sum_{i\in g}(y_i-\hat{y}_i) =\bar{y}_g-\bar{\hat{y}}_g, \qquad C_g=\frac{\sum_{i\in g}y_i}{\sum_{i\in g}\hat{y}_i}

These are empirical summaries, not guarantees about future observations. They should be calculated on evaluation data that represent the population and period where the predictions will be used, with outcomes not used to fit or tune the predictions being assessed. Calibration is distinct from overall accuracy: a zero mean residual does not mean that individual predictions are close to their outcomes.

Pooled Agreement Can Hide Subgroup Errors

The pooled mean residual is a weighted average of subgroup mean residuals. The weights are the groups' shares of observations. Consequently, positive and negative subgroup residuals can cancel. The following identity explains exactly how this happens.

Derivation: Suppose the evaluation observations are divided into non-overlapping subgroups. The total residual is the sum of residuals in each subgroup. In subgroup $g$, that sum equals $n_gD_g$, by the definition of $D_g$. Dividing the total residual by the total number of observations $n$ gives the pooled mean residual:

D_{\mathrm{pooled}} =\frac{1}{n}\sum_i(y_i-\hat{y}_i) =\frac{1}{n}\sum_g\sum_{i\in g}(y_i-\hat{y}_i) =\sum_g\frac{n_g}{n}D_g

The weights $n_g/n$ are nonnegative and sum to 1. If two equally sized groups have residuals of $+10$ and $-10$, their pooled mean residual is 0, even though neither subgroup is calibrated. With unequal group sizes, the larger group has more influence on the pooled result. This is why subgroup results should be considered alongside, not instead of, the overall result.

Worked Example: Opposite Errors Cancel in the Pooled Result

A hypothetical model predicts one-year claim cost for eight policies. Four policies are in Group A and four in Group B. Every policy in both groups receives a prediction of $100. The observed costs are $110 for each Group A policy and $90 for each Group B policy. Calculate the subgroup and pooled calibration summaries. Name the method and the mistake it avoids.

Method. Calculate the mean actual outcome, mean prediction, and mean residual separately by group, then calculate them for the portfolio. This avoids relying on a pooled summary that could conceal offsetting subgroup errors.

Group A. The total prediction is $4\times \$100=\$400$. The total observed cost is $4\times \$110=\$440$. Thus the mean prediction is $\$400/4=\$100$, the mean actual cost is $\$440/4=\$110$, and $D_A=\$110-\$100=+\$10$. The group is underpredicted by $10 per policy on average. Its observed-to-predicted ratio is $\$440/\$400=1.10$.

Group B. The total prediction is also $4\times \$100=\$400$. The observed total is $4\times \$90=\$360$. The mean prediction is $\$400/4=\$100$, and the mean actual cost is $\$360/4=\$90$. Therefore $D_B=\$90-\$100=-\$10$, indicating overprediction by $10 per policy on average. Its ratio is $\$360/\$400=0.90$.

Pooled result. The predicted total is $\$400+\$400=\$800$, and the observed total is $\$440+\$360=\$800$. Both the pooled mean prediction and pooled mean actual cost are $\$800/8=\$100$. The pooled mean residual is $(\$800-\$800)/8=\$0$, and the pooled ratio is $\$800/\$800=1.00$.

Check. The groups have equal size, so the pooled mean residual must equal $(4/8)(+\$10)+(4/8)(-\$10)=\$0$. This confirms the calculation. The pooled ratio of 1 does not mean that both groups are calibrated: their subgroup residuals and ratios show opposing errors.

Account for Unequal Subgroup Sizes

When group sizes differ, the overall result may be dominated by the larger group. It is useful to show subgroup counts along with subgroup calibration summaries so that readers can see both the direction of the error and the amount of evaluation data behind it. Equal weighting of subgroup summaries answers a different question from the observation-weighted pooled result: it gives each subgroup equal influence, regardless of its size.

Worked Example: A Small Group's Overprediction in a Large Portfolio

In a hypothetical service-cost validation set, 90 observations in Group A each have a prediction of $50 and an observed outcome of $51. Ten observations in Group B each have a prediction of $50 and an observed outcome of $41. Calculate each group's mean residual and ratio, then the pooled result. Explain why reporting only the pooled result would be incomplete.

Method. Use subgroup means and totals, then weight the subgroup mean residuals by their observation shares. This avoids treating the two groups as equally large when calculating the pooled average.

Group A. Predicted total $=90\times \$50=\$4{,}500$. Observed total $=90\times \$51=\$4{,}590$. The mean residual is $(\$4{,}590-\$4{,}500)/90=\$90/90=+\$1$. The ratio is $\$4{,}590/\$4{,}500=1.02$.

Group B. Predicted total $=10\times \$50=\$500$. Observed total $=10\times \$41=\$410$. The mean residual is $(\$410-\$500)/10=-\$90/10=-\$9$. The ratio is $\$410/\$500=0.82$.

Pooled result. The predicted total is $\$4{,}500+\$500=\$5{,}000$, and the observed total is $\$4{,}590+\$410=\$5{,}000$. With 100 observations, both pooled means are $\$50$, so the pooled mean residual is $0 and the pooled ratio is 1.00. The weighted-residual check gives $(90/100)(+\$1)+(10/100)(-\$9)=\$0.90-\$0.90=\$0$.

Check. Group B is only 10% of the observations, but its predictions exceed its observed outcomes by $9 per observation on average. The pooled result is correct as an observation-weighted summary, yet it does not describe the experience of Group B. Reporting group sizes and subgroup results makes that limitation visible.

Check Calibration Across Prediction Levels

A subgroup's mean residual can itself conceal a pattern across predicted values. For example, a model could underpredict at both low and high predicted values by different amounts, or overpredict in one prediction range and underpredict in another. A calibration plot or table can address this by dividing predictions into bins within the subgroup and comparing each bin's mean prediction with its mean observed outcome.

For a bin $b$ in subgroup $g$, calculate the average prediction and average outcome using the observations in that bin. Plot or tabulate those paired averages. Perfect empirical agreement in a bin means the two averages match. Bins are summaries, however: agreement of averages within a bin does not prove that every individual prediction is accurate or that calibration holds at every prediction value. Use enough observations to make the summaries informative, and show bin counts so a small bin is not mistaken for strong evidence.

Worked Example: A Pooled Check Hides Group Differences at Two Prediction Levels

A hypothetical model predicts a numeric outcome for two relevant groups. Each group has five observations predicted at 20 and five predicted at 40. In Group A, the observed outcome is 22 for each prediction of 20 and 44 for each prediction of 40. In Group B, the observed outcome is 18 for each prediction of 20 and 36 for each prediction of 40. Calculate calibration summaries by group and prediction bin, and compare them with the pooled result.

Method. First compare mean outcomes and predictions within each group-by-bin cell. Then combine cells to calculate each group's overall means and the portfolio means. This avoids assuming that an overall subgroup average describes all prediction levels.

Group A, prediction 20. Predicted total $=5\times20=100$; observed total $=5\times22=110$. The bin means are 20 and 22, so the mean residual is $22-20=+2$, and the observed-to-predicted ratio is $110/100=1.10$.

Group A, prediction 40. Predicted total $=5\times40=200$; observed total $=5\times44=220$. The bin means are 40 and 44, giving a mean residual of $44-40=+4$ and a ratio of $220/200=1.10$. Across Group A, the prediction total is $100+200=300$, the observed total is $110+220=330$, and each total divided by 10 gives means of 30 and 33. Thus $D_A=+3$ and $C_A=330/300=1.10$.

Group B, prediction 20. Predicted total $=5\times20=100$; observed total $=5\times18=90$. The bin means are 20 and 18, giving a mean residual of $-2$ and a ratio of $90/100=0.90$.

Group B, prediction 40. Predicted total $=5\times40=200$; observed total $=5\times36=180$. The bin means are 40 and 36, giving a mean residual of $-4$ and a ratio of $180/200=0.90$. Across Group B, the prediction total is 300 and the observed total is $90+180=270$. Both totals divided by 10 give means of 30 and 27, so $D_B=-3$ and $C_B=270/300=0.90$.

Pooled check. Across both groups, the predicted total is $300+300=600$, the observed total is $330+270=600$, and there are 20 observations. The pooled means are both $600/20=30$; the pooled mean residual is zero and the pooled ratio is 1.00. At each prediction level, the two groups also cancel: for predictions of 20, the observed mean across the ten observations is $(110+90)/10=20$; for predictions of 40, it is $(220+180)/10=40$.

Check. The pooled calibration table matches at both prediction levels, but that agreement is produced by Group A being above predictions and Group B being below them. The subgroup-by-bin summaries expose this difference.

Practical Interpretation and Common Traps

Subgroup calibration is most informative when the groups are chosen because they matter to the prediction's intended use. Consider whether an error could affect a decision, whether the subgroup is adequately represented, and whether the group definition is reliable and appropriate. When several characteristics may matter together, a single-variable split may miss patterns in combinations of characteristics. At the same time, creating many small intersections can make results too noisy to interpret. Report counts and avoid presenting a small-sample difference as a stable model property.

Calibration assessment should also be separated from causal interpretation. A subgroup residual describes a difference between predictions and observed outcomes in the evaluation data; by itself, it does not establish why that difference occurred. Data quality, changes in the outcome process, prediction scope, and subgroup composition are possible issues to investigate. If predictions will be used in a later period or a different population, calibration should be checked in data relevant to that use.

Trap: treating pooled calibration as proof of group calibration. A pooled residual of zero can result from offsetting subgroup residuals. Calculate and report the relevant subgroup summaries.
Trap: reversing the residual sign. With $e_i=y_i-\hat{y}_i$, a positive mean residual means actual outcomes exceed predictions: the model underpredicts.
Trap: comparing totals without considering group size. A group with more observations can have a larger total residual even when its mean residual is smaller. Compare mean residuals as well as totals when the question is typical error per observation.
Trap: calling a group calibrated from one mean alone. A zero subgroup mean residual can hide errors that vary across prediction levels. Examine bins or a calibration plot when that variation matters.

Key takeaway: Evaluate numeric calibration at the overall level and within consequential subgroups. Report subgroup sizes, define the residual direction, and check prediction ranges when an overall subgroup average could conceal further structure. Pooled agreement is not a substitute for evidence about each relevant group.

Check Your Understanding

For each problem, use actual minus predicted as the residual. State whether a positive mean residual indicates underprediction or overprediction.

A model predicts 30 for each of six observations in Group C. Their observed outcomes are 32, 32, 32, 32, 32, and 32. Find the mean residual and observed-to-predicted ratio. (Answer: Mean residual $=+2$, underprediction; ratio $=1.0667$.)
Group D has 20 observations with mean prediction 12 and mean actual outcome 11. Group E has 5 observations with mean prediction 12 and mean actual outcome 15. Find both group mean residuals and the pooled mean residual. (Answer: $D_D=-1$, $D_E=+3$, and pooled $D=(-20+15)/25=-0.2$, so the pooled result indicates overprediction.)
A subgroup has 40 observations with total prediction 800 and total outcome 760. Calculate its mean residual and observed-to-predicted ratio. (Answer: Mean residual $=(760-800)/40=-1$; ratio $=760/800=0.95$.)
In Group F, four observations predicted at 10 have mean outcome 12, while six observations predicted at 30 have mean outcome 28. Find the group's overall mean prediction, overall mean outcome, and mean residual. (Answer: Mean prediction $=(4\times10+6\times30)/10=22$; mean outcome $=(4\times12+6\times28)/10=21.6$; mean residual $=-0.4$.)
Two equally sized subgroups have mean residuals $+4$ and $-4$. What is the pooled mean residual, and what additional information is needed before concluding that calibration is satisfactory? (Answer: Pooled mean residual $=0$; inspect subgroup summaries and, where relevant, calibration across prediction ranges, along with subgroup counts and evaluation-data suitability.)

What You'll Learn