Reading a Calibration Plot

What a Calibration Plot Shows

A prediction can be close on average yet behave differently across the range of predicted values. For example, predictions may tend to be too low for one range of cases and too high for another. A calibration plot makes such differences easier to see by comparing predictions with outcomes across groups of cases.

For each case, there is a prediction and a corresponding observed outcome, measured on the same scale. Cases are organized into groups, and the means of the predictions and outcomes are calculated separately within each group. The plot places each group’s mean prediction on the horizontal axis and its mean outcome on the vertical axis. Thus, one plotted point summarizes a group; it is not an individual prediction.

Definition: A grouped calibration plot displays, for each group $g$, the point $(\bar{\hat{y}}_g,\bar{y}_g)$, where $\bar{\hat{y}}_g$ is the group’s mean prediction and $\bar{y}_g$ is the group’s mean observed outcome. The identity line is the line $y=x$. Points on this line have equal group mean predictions and group mean outcomes.

The identity line is a reference for agreement of the two means. It is not a fitted regression line. To interpret a point, compare its vertical coordinate (mean outcome) with its horizontal coordinate (mean prediction). A point above the line has a higher mean outcome than mean prediction: predictions are too low on average in that group, or underpredict. A point below the line has a lower mean outcome than mean prediction: predictions are too high on average, or overpredict.

From Paired Values to a Plotted Point

Let $\hat{y}_i$ be the prediction and $y_i$ the observed outcome for case $i$. In a group $g$ with $m_g$ cases, calculate the two means using the same cases:

\bar{\hat{y}}_g=\frac{1}{m_g}\sum_{i\in g}\hat{y}_i, \qquad \bar{y}_g=\frac{1}{m_g}\sum_{i\in g}y_i.

The plotted point is $(\bar{\hat{y}}_g,\bar{y}_g)$, in that order. Both coordinates have the units of the outcome. The point’s vertical difference from the identity line is $\bar{y}_g-\bar{\hat{y}}_g$. A positive difference means underprediction on average in that group; a negative difference means overprediction on average.

Key result: If prediction error is defined as observed outcome minus prediction, $e_i=y_i-\hat{y}_i$, then the group’s mean prediction error is exactly the vertical difference of its calibration point from the identity line: $$ \bar{y}_g-\bar{\hat{y}}_g =\frac{1}{m_g}\sum_{i\in g}(y_i-\hat{y}_i) =\bar{e}_g. $$

This equality follows because subtracting the two means uses the same cases and the same denominator. It explains why the plot’s vertical position relative to the line gives a direct description of the group’s average error, rather than merely a visual association.

Worked Example: Calculating Three Grouped Points

Suppose a hypothetical validation set records predicted and observed systolic blood pressure, in mmHg. The cases have already been arranged into three groups. Calculate and interpret each plotted point.

Group	Predictions (mmHg)	Observed outcomes (mmHg)
A	100, 110	105, 115
B	120, 130	118, 127
C	140, 150	150, 155

Group A. The mean prediction is $(100+110)/2=105$ mmHg, and the mean outcome is $(105+115)/2=110$ mmHg. Plot $(105,110)$. Since $110>105$, the point is 5 mmHg above the identity line: the predictions underpredict the group mean by 5 mmHg.

Group B. The mean prediction is $(120+130)/2=125$ mmHg, and the mean outcome is $(118+127)/2=122.5$ mmHg. Plot $(125,122.5)$. Since $122.5<125$, the point is 2.5 mmHg below the line: the predictions overpredict the group mean by 2.5 mmHg.

Group C. The mean prediction is $(140+150)/2=145$ mmHg, and the mean outcome is $(150+155)/2=152.5$ mmHg. Plot $(145,152.5)$. This point is 7.5 mmHg above the line, indicating underprediction of the group mean by 7.5 mmHg.

The pattern is mixed: group B is overpredicted, while groups A and C are underpredicted. These conclusions concern each group’s average, not every case within the group.

Why the Vertical Difference Has This Meaning

For a group with $m_g$ cases, subtracting the mean prediction from the mean outcome gives:

\begin{aligned} \bar{y}_g-\bar{\hat{y}}_g &=\frac{1}{m_g}\sum_{i\in g}y_i -\frac{1}{m_g}\sum_{i\in g}\hat{y}_i\\ &=\frac{1}{m_g}\sum_{i\in g}(y_i-\hat{y}_i)\\ &=\bar{e}_g. \end{aligned}

The identity line has vertical coordinate equal to horizontal coordinate at every point. Therefore, for the plotted point $(\bar{\hat{y}}_g,\bar{y}_g)$, its signed vertical difference from that line is $\bar{y}_g-\bar{\hat{y}}_g$, which the calculation shows is the mean of the observed-minus-predicted errors. A positive signed difference places the point above the line; a negative one places it below. This is a descriptive equality, not a claim that every individual error has the same sign.

Worked Example: Reading Direction and Size of Disagreement

A hypothetical plot summarizes three groups’ predictions of daily water use, measured in litres. The plotted coordinates are $(40,46)$, $(60,60)$, and $(80,73)$. Interpret them in context.

For the first group, the mean prediction is 40 litres and the mean outcome is 46 litres. The signed difference is $46-40=6$ litres, so the point is above the line and the group is underpredicted by an average of 6 litres.

For the second group, $60-60=0$ litres. Its point lies on the identity line: the group mean prediction equals the group mean outcome.

For the third group, $73-80=-7$ litres. Its point is below the line, so the group is overpredicted by an average of 7 litres.

The distances here are vertical differences in litres, not perpendicular distances to the line. They describe the difference between the two group means. For instance, the first point does not imply that every household in that group used exactly 6 litres more than predicted.

Grouped Agreement Can Hide Differences

A plotted point compresses all cases in a group into two numbers. The group mean prediction and group mean outcome can agree even when individual predictions are not close to their corresponding outcomes. Nor does one group’s point reveal whether the errors within it are small or large, or whether positive and negative errors cancel. The plot is therefore useful for examining group-level agreement, but it is not a display of every individual error.

An overall mean prediction equal to the overall mean outcome also does not guarantee that predictions agree across the range. A positive mean error in one group can be offset by a negative mean error in another. Looking at several grouped points can reveal this pattern even when the overall mean prediction error is zero.

Worked Example: Zero Overall Error with Different Group Patterns

Consider three equally sized groups with the following hypothetical mean daily expenses, in dollars:

Group	Mean prediction	Mean outcome	Outcome minus prediction
Low	$10	$14	$4
Middle	$20	$20	$0
High	$30	$26	−$4

The calibration points are $(10,14)$, $(20,20)$, and $(30,26)$. The low group lies $4 above the identity line, so its mean is underpredicted by $4. The middle group lies on the line. The high group lies $4 below the line, so its mean is overpredicted by $4.

Because the groups are equally sized, the overall mean prediction is $(10+20+30)/3=20$ dollars and the overall mean outcome is $(14+20+26)/3=20$ dollars. The overall mean prediction error is therefore $20-20=0$ dollars. Yet the plotted points show that the errors differ by group. The zero overall average results from the $+$4$ and $−$4$ group differences cancelling, not from every group being calibrated.

Assumptions, Scope, and Common Mistakes

A calibration plot is a descriptive summary. Constructing one does not require the linear-model assumptions used for regression inference: the identity line is a fixed reference, not a line estimated from the plotted points. Still, the comparison is meaningful only if each prediction is paired with its corresponding outcome, both are expressed in comparable units and on the same scale, and the group means are calculated over the same cases. If the observations are from a sample intended to represent a broader population, the plot alone does not establish how precisely the displayed group differences estimate population behavior.

Interpret the points as evidence about the groups shown, not as proof that a prediction method will behave the same way in every future setting. The plot also does not by itself tell you the size of individual errors, whether a difference is statistically distinguishable from random variation, or why any systematic discrepancy occurs.

Reversing the axes. With mean prediction on the horizontal axis and mean outcome on the vertical axis, above the identity line means underprediction; below means overprediction.
Calling the identity line a fitted trend. It is the reference $y=x$, not a regression line estimated from the points.
Interpreting a group mean as an individual result. A point above the line says the group mean outcome exceeds the group mean prediction; it does not say every case was underpredicted.
Concluding that zero overall error means calibration everywhere. Opposing group differences can cancel. Inspect the separate grouped points as well as any overall summary.
Treating closeness to the line as a complete performance assessment. Group means do not show the spread of individual errors or how large individual misses are.

Key takeaway: Plot each group’s mean prediction on the horizontal axis and its mean outcome on the vertical axis. Relative to $y=x$, a point above indicates average underprediction in that group, and a point below indicates average overprediction. The plot describes grouped averages, not every individual case.

Check Your Understanding

Answer each question using the horizontal-axis prediction and vertical-axis outcome convention.

A group’s mean prediction is 25 minutes and its mean outcome is 22 minutes. Give the plotted coordinates and interpret the point’s position relative to the identity line.
Why must the same cases be used to calculate a group’s mean prediction and mean outcome for the vertical difference to equal its mean prediction error?
A point lies on the identity line. What does this establish about the group means, and what does it not establish about individual cases?
Two equally sized groups have mean errors of $+$5$ and $−$5$. Explain how the overall mean error can be zero while the calibration plot still shows group-level disagreement.
Does a grouped calibration plot by itself show that a group-level difference is statistically significant? Explain briefly.

What You'll Learn