Calibration Within Groups

What Does a Risk Score Mean Within a Group?

A model may assign each person a predicted probability of an event, such as a missed appointment within a specified period. A prediction of 30% has a clear probabilistic interpretation: among comparable cases assigned that risk, the event should occur about 30% of the time in repeated observations. A fairness audit can ask whether that interpretation holds within each group being examined.

This question concerns the meaning of the score, not merely whether groups receive similar scores or whether people above a decision threshold have similar outcomes. A groupwise calibration check compares predicted risks with observed event rates separately by group. The result is meaningful only when the outcome, observation period, group definition, and population being evaluated are stated consistently.

Definition: Let $Y$ be a binary outcome, where $Y=1$ denotes the event and $Y=0$ denotes no event. Let $\hat p$ be a model's predicted probability of the event, and let $G$ identify a group. The predictions are calibrated within group $g$ if, for each predicted probability $p$ that occurs in that group, $$ P(Y=1\mid \hat p=p,\ G=g)=p. $$ In words, among members of group $g$ assigned risk $p$, the event rate is $p$. In finite data, calibration is assessed approximately, because observed rates fluctuate.

Exact comparisons at every distinct score can be impractical, particularly when many people have different scores. A common diagnostic is therefore to divide predictions into risk buckets, such as scores from 20% up to (but not including) 30%. For each group and bucket, compare the average predicted probability with the observed proportion of events. A bucket check is a summary over a range of scores; it does not establish that every individual score within the range is calibrated.

Bucket-level diagnostic: For group $g$ and bucket $B$, let $n_{g,B}$ be the number of evaluated cases, $\hat p_j$ the predicted risk for case $j$, and $Y_j$ its observed binary outcome. The average predicted risk, observed event rate, and calibration difference are $$ \overline{\hat p}_{g,B} =\frac{1}{n_{g,B}}\sum_{j\in(g,B)}\hat p_j,\qquad \widehat r_{g,B} =\frac{1}{n_{g,B}}\sum_{j\in(g,B)}Y_j,\qquad \widehat r_{g,B}-\overline{\hat p}_{g,B}. $$ A difference near zero indicates agreement in that group and bucket. It is a descriptive estimate, not proof of exact calibration.

Why Compare the Two Averages?

The comparison follows directly from the meaning of a probability score. If predictions are calibrated at each score within a group, the expected event rate among cases whose scores fall in a bucket equals the average predicted risk among those cases. The bucket need not contain a single identical score for this result to hold.

To see why, condition first on the exact score and group, then average over the scores that fall in the bucket. Since $Y$ is binary, its conditional mean equals its conditional event probability. Under calibration, that probability is the score itself:

\begin{aligned} E[Y\mid \hat p\in B,\ G=g] &=E\!\left[E[Y\mid \hat p,\ G=g]\mid \hat p\in B,\ G=g\right]\\ &=E[\hat p\mid \hat p\in B,\ G=g]. \end{aligned}

The first equality is the law of iterated expectations. In the second, $E[Y\mid\hat p,G=g]=P(Y=1\mid\hat p,G=g)=\hat p$, which is the calibration condition. In observed data, the sample event rate estimates the left side and the average score estimates the right side. The observed difference will not generally be exactly zero, even when the model is calibrated, because the outcomes are random and the sample is finite.

Comparing Risk Buckets Across Groups

A practical audit places the same predicted-risk intervals alongside one another for each group. It is important to compare the averages as well as the event rates: two groups can have scores within the same broad bucket but different average scores. A small difference in observed event rates is not, by itself, evidence that a stated prediction has exactly the same meaning. The bucket widths, case counts, and underlying averages all matter.

Worked Example: Locating Group-Specific Calibration Differences

A hypothetical service provider evaluates a model that predicts whether a scheduled payment will be late within 30 days. The outcome is recorded for every evaluated case, and the two groups use the same outcome definition and evaluation period. In the 15% to 25% bucket, Group A has 100 cases, an average predicted risk of 20%, and 18 late payments. Group B has 80 cases, an average predicted risk of 20%, and 24 late payments. In the 55% to 65% bucket, Group A has 50 cases, an average prediction of 60%, and 32 late payments; Group B has 40 cases, an average prediction of 60%, and 20 late payments. Calculate and compare the bucket diagnostics.

Method. For each group and bucket, divide the event count by the case count to obtain the observed event rate. Compare that rate with the stated average predicted risk.

15% to 25% bucket, Group A. The observed rate is $18/100=0.18$, or 18%. The average predicted risk is 20%, so the calibration difference is $0.18-0.20=-0.02$, or −2 percentage points.

15% to 25% bucket, Group B. The observed rate is $24/80=0.30$, or 30%. The predicted average is 20%, so the difference is $0.30-0.20=0.10$, or +10 percentage points.

55% to 65% bucket, Group A. The observed rate is $32/50=0.64$, or 64%. The average prediction is 60%, so the difference is $0.64-0.60=0.04$, or +4 percentage points.

55% to 65% bucket, Group B. The observed rate is $20/40=0.50$, or 50%. The predicted average is 60%, so the difference is $0.50-0.60=-0.10$, or −10 percentage points.

The low-risk bucket has a higher observed event rate in Group B than in Group A, despite the same 20% average prediction. The high-risk bucket shows the opposite pattern: Group A's event rate exceeds its prediction, while Group B's falls below it. These are sample findings that call for investigation; they do not alone identify why the differences arose.

Common mistake avoided. Do not compare event counts directly: 24 events in Group B's low-risk bucket is not a 24% event rate. The correct denominator is that bucket's 80 cases, giving $24/80=30\%$.

Worked Example: Checking Whether a 30% Bucket Has Comparable Meaning

A hypothetical screening program groups predictions between 25% and 35% into one risk bucket. In Group C, 50 cases have predicted probabilities that sum to 15.00, and 15 cases experience the specified event. In Group D, 80 cases have predicted probabilities that sum to 24.00, and 24 experience the event. Calculate each group's average prediction and observed event rate, then interpret the comparison.

Method. The sum of predicted probabilities divided by the case count gives the average predicted risk. The number of events divided by the same case count gives the observed event rate.

For Group C, the average prediction is $15.00/50=0.30$, or 30%. The observed rate is $15/50=0.30$, also 30%. Its calibration difference is $0.30-0.30=0.00$, or 0 percentage points.

For Group D, the average prediction is $24.00/80=0.30$, or 30%. The observed rate is $24/80=0.30$, also 30%. Its calibration difference is $0.30-0.30=0.00$, or 0 percentage points.

At this bucket level, the evidence is consistent with a 30% prediction having the same meaning in these two groups: each group's observed event rate matches its average predicted risk. The calculation does not show that every score from 25% to 35% is calibrated, or that the model is equally useful in other respects. It is a comparison of these groups in this bucket and evaluation period.

Common mistake avoided. Using the group sizes as event counts, or comparing 15 with 24 directly, ignores the denominators. The relevant rates are $15/50$ and $24/80$, not the raw counts.

Why Pooled Calibration Can Mislead

A single calibration summary for an entire group can conceal offsetting errors in different risk ranges. A group may have an overall observed event rate equal to its overall average predicted risk while being systematically underpredicted in one bucket and overpredicted in another. Therefore, a conclusion about calibration should state whether it refers to the full group or to particular risk buckets.

Worked Example: An Overall Match That Hides Bucket Errors

A hypothetical model is assessed for a binary claim event. Within Group A, 50 cases in a low-risk bucket have an average prediction of 20% and 5 events; 50 cases in a high-risk bucket have an average prediction of 60% and 35 events. Within Group B, the low-risk bucket has 50 cases, an average prediction of 20%, and 15 events; the high-risk bucket has 50 cases, an average prediction of 60%, and 25 events. Compare the bucket results with the overall group results.

Group A, low bucket. The predicted total is $50(0.20)=10$ expected events. The observed rate is $5/50=0.10$, so the difference is $0.10-0.20=-0.10$, or −10 percentage points.

Group A, high bucket. The predicted total is $50(0.60)=30$. The observed rate is $35/50=0.70$, giving a difference of $0.70-0.60=0.10$, or +10 percentage points. Across both buckets, predicted total $=10+30=40$, and observed total $=5+35=40$. Thus the overall average prediction is $40/100=0.40$ and the overall event rate is $40/100=0.40$.

Group B, low bucket. The predicted total is $50(0.20)=10$. The observed rate is $15/50=0.30$, giving a difference of $0.30-0.20=0.10$, or +10 percentage points.

Group B, high bucket. The predicted total is $50(0.60)=30$. The observed rate is $25/50=0.50$, giving a difference of $0.50-0.60=-0.10$, or −10 percentage points. Across both buckets, predicted total $=10+30=40$, and observed total $=15+25=40$. The overall average prediction and event rate are again $40/100=0.40$.

Both groups match overall, but neither matches within either bucket. For example, saying “the model is calibrated for both groups because each has a 40% overall rate” would overlook the opposing bucket-level errors. The appropriate conclusion is narrower: the overall averages agree in this sample, while the bucket comparisons show substantial departures in opposite directions.

Common mistake avoided. Do not infer within-group calibration at all risk levels from one group-wide average. The overall calculation weights the buckets together, allowing the errors to cancel.

Interpreting and Reporting the Audit

Calibration is one property of predictions, not a complete fairness verdict. A close match between predicted and observed rates does not establish that the outcome label is appropriate, that outcomes were observed equally well, or that using the score produces acceptable consequences. Those questions require attention to how the outcome was defined and measured, who entered the evaluation data, and how decisions use predictions.

A responsible report specifies the event and its time window, the groups compared, the risk-bucket boundaries, and the number of cases and events in each bucket. It presents average predictions and observed rates together, rather than reporting only one. Where estimates differ, describe the size and direction of the differences and avoid treating a sample match as proof of exact or lasting calibration. If the data differ in outcome measurement or access to observed outcomes, those limitations can affect the comparison.

A further trap is to treat a bucket as if every person in it had the bucket's average risk. If a bucket contains scores from 25% to 35%, its mean may be 28% in one group and 32% in another. A comparison of event rates alone would then conflate differences in average predicted risk with differences between observed rates and predictions. Always report the bucket's average prediction as well as its event rate.

Key takeaway: Groupwise calibration asks whether observed event rates track predicted probabilities within each group. Risk buckets make the comparison practical, but conclusions must remain at the level actually measured: a pooled or bucket-level match does not guarantee calibration at every score or establish fairness in every relevant sense.

Check Your Understanding

For each problem, calculate the requested quantities and interpret the result at the stated level. Probabilities in the answers are rounded to four decimal places where needed.

In a hypothetical group, 60 cases have an average predicted risk of 25%, and 18 have the event. Calculate the observed rate and calibration difference. (Answer: observed rate $=0.3000$; difference $=+0.0500$, or +5 percentage points.)
Two groups are evaluated in the same risk bucket. Group M has 40 cases, predicted probabilities summing to 12, and 12 events. Group N has 70 cases, predicted probabilities summing to 21, and 21 events. Calculate both average predictions and event rates, and state the bucket-level comparison. (Answer: both groups have average prediction $=0.3000$, event rate $=0.3000$, and difference $=0.0000$; the results agree at this bucket level.)
A group has 80 cases in a low bucket with average prediction 10% and 16 events, and 20 cases in a high bucket with average prediction 50% and 4 events. Calculate the two bucket differences and the overall average prediction and event rate. (Answer: low-bucket difference $=0.2000-0.1000=+0.1000$; high-bucket difference $=0.2000-0.5000=-0.3000$; overall average prediction $=(80(0.10)+20(0.50))/100=0.1800$; overall event rate $=(16+4)/100=0.2000$.)
In a 20% to 30% bucket, Group R has 100 cases, an average prediction of 24%, and 22 events. Group S has 50 cases, an average prediction of 28%, and 14 events. Calculate each difference and explain why comparing event rates alone could be incomplete. (Answer: Group R rate $=0.2200$, difference $=-0.0200$; Group S rate $=0.2800$, difference $=0.0000$. Their predicted averages differ, so the event-rate comparison alone does not show how closely each rate matches its own average prediction.)
In each of two groups, a low bucket has 30 cases with average prediction 20%, while a high bucket has 70 cases with average prediction 60%. Group P has 6 low-bucket events and 42 high-bucket events; Group Q has 9 and 35, respectively. Calculate each group's overall average prediction and event rate, then explain why those totals do not establish bucket-level calibration. (Answer: each group's predicted total is $30(0.20)+70(0.60)=48$, so average prediction $=0.4800$. Group P has 48 events and rate $=0.4800$; Group Q has 44 events and rate $=0.4400$. The group totals do not replace separate low- and high-bucket comparisons.)

What You'll Learn