Why Examine Calibration Across the Prediction Range?
A single average can make predictions look reasonable even when they are systematically off in different parts of the prediction range. For example, a model may underpredict at lower values and overpredict at higher values, with the two errors cancelling in the overall average. A calibration plot helps reveal such patterns by comparing predictions with outcomes across groups of cases.
The plot is a diagnostic summary of a validation data set. It does not show whether a model is useful for every decision, establish why a discrepancy occurs, or guarantee that future predictions will behave the same way. Its value is more focused: it displays how average observed outcomes compare with average predictions at different prediction levels.
The outcome and prediction must use the same scale and units. For instance, if predictions are annual amounts in dollars, the observed outcomes must also be annual amounts in dollars—not totals over a different time period. Each validation case must have a paired prediction and observed outcome. Ideally, its prediction was produced without using that case’s outcome to fit the model; otherwise, the plot may reflect fit to the data rather than performance on new cases.
Constructing the Plot
A calibration plot uses one point per group, not one point per individual case. Groups can be formed by sorting cases by predicted value and splitting them into similarly sized groups, or by setting ranges of predicted values in advance. The chosen rule must be stated clearly and applied consistently. Empty groups produce no point.
For group \(g\), let \(n_g\) be the number of cases, let \(\hat{y}_j\) be case \(j\)’s prediction, and let \(y_j\) be its observed outcome. The plotted coordinates are:
Using the mean prediction as the horizontal coordinate is important. A group can contain predictions that vary substantially, so its midpoint, upper endpoint, or an arbitrary representative prediction is not generally the same as its mean prediction. The vertical coordinate is the mean of the observed outcomes for the exact same cases.
Use validation cases for which both the prediction and the corresponding observed outcome are available and defined on the same scale.
For example, sort by prediction and make groups of similar size, or define fixed prediction ranges. Check that each group has at least one case.
Use the same group membership for the mean prediction and mean outcome.
Plot \((\bar{\hat{y}}_g,\bar{y}_g)\), then assess the direction, size, and pattern of departures from \(y=x\).
Worked Example: Calculating Calibration Points for Three Prediction Ranges
A hypothetical validation sample contains 12 policy records. For ease of display, cases have already been sorted by their predictions and divided into three consecutive groups of four. All values are annual amounts in dollars. Calculate the three calibration points and interpret their departures from the reference line.
| Group | Predictions | Observed outcomes |
|---|---|---|
| 1 | 80, 100, 120, 140 | 90, 105, 125, 140 |
| 2 | 160, 180, 200, 220 | 150, 175, 205, 210 |
| 3 | 240, 260, 280, 300 | 260, 275, 285, 310 |
Method. For each group, calculate the mean prediction for the horizontal coordinate and the mean observed outcome for the vertical coordinate. The grouping is fixed, so the same four records contribute to both means in each group.
Group 1. The sum of predictions is \(80+100+120+140=440\), so the mean prediction is \(440/4=110\). The outcome sum is \(90+105+125+140=460\), so the mean outcome is \(460/4=115\). The point is \((110,115)\), five dollars above the reference line.
Group 2. The prediction sum is \(160+180+200+220=760\), giving mean prediction \(760/4=190\). The outcome sum is \(150+175+205+210=740\), giving mean outcome \(740/4=185\). The point is \((190,185)\), five dollars below the line.
Group 3. The prediction sum is \(240+260+280+300=1080\), giving mean prediction \(1080/4=270\). The outcome sum is \(260+275+285+310=1130\), giving mean outcome \(1130/4=282.5\). The point is \((270,282.5)\), twelve dollars and fifty cents above the line.
The plotted points are \((110,115)\), \((190,185)\), and \((270,282.5)\). The first group’s average prediction is below its average outcome, the second group’s is above, and the third group’s is below. The pattern is not a uniform upward or downward displacement. This example is checked by comparing each point’s vertical coordinate with its horizontal coordinate; the signed differences are \(115-110=5\), \(185-190=-5\), and \(282.5-270=12.5\).
Common mistake avoided: Do not plot a group’s largest prediction or a group boundary on the horizontal axis. The required coordinate is the group’s mean prediction.
What Departures from the Line Mean
At a point on the line, the group mean outcome equals the group mean prediction. A point above the line has \(\bar{y}_g>\bar{\hat{y}}_g\): the group’s outcomes average higher than predicted, so the model underpredicts on average in that group. A point below the line has \(\bar{y}_g<\bar{\hat{y}}_g\): the model overpredicts on average in that group.
A pattern across the horizontal axis can suggest that predictions are not varying enough or are varying too much relative to the observed group means. In particular, points above the line at low predictions and below it at high predictions mean that observed group means are less spread out than predicted group means. Predictions may be too spread out. The reverse pattern—points below the line at low predictions and above it at high predictions—may indicate predictions that are too compressed toward the middle.
These are interpretations of the plot’s shape, not proof of a particular cause. Grouping can affect the displayed pattern, and sampling variation can cause individual points to depart from the line. A sensible interpretation considers the size and consistency of the departures, the number of cases in each group, and the context of the validation data.
The final equality follows by combining the two sums, which have the same \(n_g\) cases and the same denominator. Thus, a point’s vertical departure from the line is exactly the group’s mean signed prediction error, using observed minus predicted as the sign convention. A positive value indicates underprediction on average; a negative value indicates overprediction on average.
Worked Example: Choosing Between Equal-Count and Fixed-Width Groups
A hypothetical service team validates eight numeric predictions, all measured in dollars. The paired predictions and outcomes, ordered by prediction, are:
| Case | Prediction | Observed outcome |
|---|---|---|
| 1 | 10 | 15 |
| 2 | 20 | 25 |
| 3 | 30 | 22 |
| 4 | 40 | 45 |
| 5 | 50 | 40 |
| 6 | 60 | 72 |
| 7 | 70 | 65 |
| 8 | 150 | 100 |
Equal-count method. Make two groups of four consecutive cases. For cases 1–4, the predictions sum to \(10+20+30+40=100\), so the mean prediction is \(100/4=25\). Outcomes sum to \(15+25+22+45=107\), so the mean outcome is \(107/4=26.75\). This gives the point \((25,26.75)\).
For cases 5–8, predictions sum to \(50+60+70+150=330\), giving mean prediction \(330/4=82.5\). Outcomes sum to \(40+72+65+100=277\), giving mean outcome \(277/4=69.25\). The second point is \((82.5,69.25)\).
Fixed-width method. Now use the explicitly defined ranges of predictions at most 50 and greater than 50. The first group contains five cases. Its prediction sum is \(10+20+30+40+50=150\), so the mean prediction is \(150/5=30\). Its outcome sum is \(15+25+22+45+40=147\), giving mean outcome \(147/5=29.4\). Its point is \((30,29.4)\).
The second group contains predictions 60, 70, and 150. Their sum is \(280\), so the mean prediction is \(280/3=93.333\ldots\). The outcomes sum to \(72+65+100=237\), giving mean outcome \(237/3=79\). Its point is approximately \((93.33,79)\), with the horizontal coordinate rounded to two decimal places.
The two grouping rules produce different points because they combine cases differently. The high prediction of 150 pulls the equal-count group’s mean prediction upward, while the fixed-width group places that case with two smaller predictions. Neither plot changes the underlying observations; each is a different summary of them. State the rule and inspect group sizes before interpreting. The fixed-width upper group has only three cases, so its point may be less stable than a point based on a larger group.
Common mistake avoided: Do not assume that two reasonable grouping rules must produce the same calibration plot. The calculation is correct only when the stated group membership is used consistently for both coordinates.
Overall Agreement Can Conceal a Pattern
The overall mean prediction and overall mean outcome are each averages across all validation cases. If they agree, the model has aggregate mean agreement on that sample. This does not establish agreement at different prediction levels: positive and negative group errors can cancel.
Worked Example: Detecting Cancellation in the Overall Mean
A hypothetical model produces four predictions and outcomes, in dollars: predictions \(100, 200, 300, 400\), and outcomes \(180, 220, 280, 320\). Form two groups, with the first two cases in the low-prediction group and the last two in the high-prediction group. Compare the overall means with the group calibration points.
Overall means. The predictions sum to \(100+200+300+400=1000\), giving overall mean \(1000/4=250\). The outcomes sum to \(180+220+280+320=1000\), giving overall mean \(1000/4=250\). Overall, the means agree.
Low-prediction group. Its mean prediction is \((100+200)/2=150\). Its mean outcome is \((180+220)/2=200\). The point \((150,200)\) is 50 dollars above the line.
High-prediction group. Its mean prediction is \((300+400)/2=350\). Its mean outcome is \((280+320)/2=300\). The point \((350,300)\) is 50 dollars below the line.
The group departures cancel: \(+50\) and \(-50\). This pattern means observed group means are less spread out than predicted group means; the predictions may be too spread out. The calculation also verifies the overall agreement: the two groups have equal size, and their group means average to \((150+350)/2=250\) for predictions and \((200+300)/2=250\) for outcomes.
Common exam trap. A wrong interpretation would be: “The low-prediction point is above the line and the high-prediction point is below it, so predictions are too compressed.” That reverses the pattern. The correction is that observed group means lie closer together than predicted group means, suggesting predictions may be too spread out. This shape alone does not prove the model’s cause of error or determine a corrective adjustment.
Practical Checks Before Interpreting
A calibration plot is most useful when the construction and limitations are made explicit. Before drawing a conclusion, check the following:
- Same cases and scale: Each group’s prediction and outcome means must use the same cases, units, and outcome definition.
- Transparent groups: Report whether groups were formed by equal counts, fixed prediction ranges, or another stated rule. Avoid changing group membership after inspecting outcomes just to produce a preferred pattern.
- Group sizes: Very small groups can produce unstable means. Equal-count grouping helps make group sizes similar, while fixed-width grouping can leave some ranges sparse or empty.
- Use of precision: A point based on many cases and one based on few cases do not necessarily have the same reliability. The plot alone does not display that difference unless group sizes are also reported or encoded.
- Scope of the conclusion: The plot summarizes average agreement within the chosen groups. It does not show every individual error, and a point on the line does not mean that all cases in the group were predicted accurately.
Calibration plots complement, rather than replace, error summaries. A plot can reveal systematic patterns in the conditional averages that a single overall average hides, while other metrics are needed to describe the magnitude or distribution of individual errors. When reporting the plot, state the validation population, grouping rule, axes and units, group sizes, and the main pattern relative to the line. These details make the interpretation reproducible and appropriately limited, aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each problem, use the specified grouping rule and compare the plotted coordinates with the reference line.
- A hypothetical model has predictions \(20,30,40,50,60,70\) and outcomes \(18,36,44,46,69,77\). Form two groups from the first three and last three cases. Calculate both points and state whether each is above, below, or on the line. (Answer: \((30,32.67)\), above; \((60,64)\), above.)
- A validation sample has predictions \(5,10,15,20\) and outcomes \(8,7,19,16\). Make two groups from the first two and last two cases. Calculate the points and describe the grouped calibration. (Answer: \((7.5,7.5)\) and \((17.5,17.5)\); both are on the line.)
- A plot has points \((40,50)\) and \((100,90)\), corresponding to low and high prediction groups. State the average prediction direction in each group and the pattern suggested about prediction spread. (Answer: Underprediction in the low group and overprediction in the high group; observed group means are less spread out than predictions, so predictions may be too spread out.)
- Four validation predictions are \(100,200,300,400\), and their outcomes are \(180,220,280,320\). Calculate the overall means and the two calibration points using the first two and last two cases as groups. What does the plot reveal that the overall means alone do not? (Answer: Both overall means are \(250\); group points are \((150,200)\) and \((350,300)\). Overall agreement conceals opposite group departures, with predictions potentially too spread out.)
- Eight ordered predictions are \(10,20,30,40,50,60,70,150\), with outcomes \(15,25,22,45,40,72,65,100\). Using fixed ranges of at most 50 and greater than 50, calculate each point to two decimals where needed, then identify which point is based on fewer cases. (Answer: \((30,29.4)\) and approximately \((93.33,79)\); the second point is based on three cases, compared with five in the first group.)