What Does It Mean for a Prediction to Be Calibrated?
A prediction can be close to the observed outcome on average yet systematically too high for some predicted values and too low for others. The overall average can hide this pattern because positive and negative errors cancel. Calibration asks a more specific question: among cases given the same predicted value, does the average observed outcome agree with that prediction?
Consider a model that predicts a continuous outcome such as a laboratory measurement, recovery time, or annual cost. Let \(Y\) denote the observed outcome for a case and \(\hat{Y}\) the prediction produced for that case. Across the target population, both are treated as quantities that vary from case to case. We use the prediction-error convention \(Y-\hat{Y}\): a positive error means the observed outcome exceeded the prediction.
The definition concerns average agreement at each predicted value, not closeness for each case. For example, a prediction of 20 units is calibrated if cases receiving that prediction have an average observed outcome of 20 units, even if some outcomes are substantially below 20 and others substantially above it. Calibration is therefore a property of predictions across cases in a population, not a guarantee about an individual case.
Calibration is meaningful only when predictions and outcomes refer to the same quantity and use the same units. It is also an evaluation question: predictions must be paired with the outcomes they were intended to predict. No assumption of normally distributed errors or a linear relationship is part of the definition.
Calibration and Overall Prediction Error Ask Different Questions
A common summary of prediction errors is the mean prediction error. For \(n\) equally weighted cases, define \(e_i=y_i-\hat{y}_i\). The mean prediction error is
A mean prediction error near zero indicates little overall signed bias: positive and negative errors balance in the aggregate. It does not show that errors balance within every range of predicted values. In contrast, calibration requires the average error to be zero conditional on the prediction.
MAE and RMSE summarize the size of prediction errors rather than their conditional pattern. MAE averages the absolute errors, while RMSE takes the square root of the average squared errors. Both are useful measures of typical error magnitude, but neither alone identifies whether outcomes tend to exceed predictions in one part of the prediction range and fall below them elsewhere.
Worked Example: Opposing Conditional Errors Cancel
Suppose six hypothetical cases receive predictions of either 10 or 20 units. The observed outcomes are as follows. All cases have equal weight.
| Predicted value (units) | Observed outcomes (units) |
|---|---|
| 10 | 12, 12, 12 |
| 20 | 18, 18, 18 |
For the three cases predicted to be 10 units, the mean observed outcome is \((12+12+12)/3=12\) units. Their mean prediction error is \(12-10=2\) units. For the three cases predicted to be 20 units, the mean observed outcome is \((18+18+18)/3=18\) units. Their mean prediction error is \(18-20=-2\) units. The average outcome therefore differs from the prediction in both groups: predictions are too low at 10 and too high at 20.
Across all six cases, however, the mean prediction error is $$ \frac{3(12-10)+3(18-20)}{6} =\frac{6-6}{6} =0\text{ units}. $$ The overall MAE is $$ \frac{3|12-10|+3|18-20|}{6} =\frac{6+6}{6} =2\text{ units}, $$ and the RMSE is $$ \sqrt{\frac{3(2^2)+3((-2)^2)}{6}} =\sqrt{\frac{24}{6}} =2\text{ units}. $$ A zero mean prediction error does not make these predictions calibrated: within each prediction value, the average outcome does not equal the prediction. The overall average hides two opposing conditional errors.
Why Calibration Implies Zero Overall Mean Error
Calibration is a stronger condition than a zero mean prediction error. Its implication follows from the law of iterated expectation. If \(E[Y-\hat{Y}\mid\hat{Y}]=0\), then averaging those conditional mean errors over the distribution of predictions gives a zero overall mean error:
The reverse implication does not follow. A zero overall mean error only says that the conditional errors average to zero when combined across all prediction values. One conditional error can be positive and another negative, as in the preceding example. Thus, a mean prediction error of zero is a useful summary of overall bias, but it is not evidence by itself that predictions are calibrated.
Worked Example: Small Errors Can Still Show Miscalibration
Now suppose four hypothetical cases have predictions of 10 or 20 units, with outcomes as shown. This example has smaller errors than the previous one.
| Predicted value (units) | Observed outcomes (units) |
|---|---|
| 10 | 10.5, 10.5 |
| 20 | 19.5, 19.5 |
For predictions of 10 units, the average outcome is 10.5 units, so the conditional mean error is \(10.5-10=0.5\) units. For predictions of 20 units, the average outcome is 19.5 units, so the conditional mean error is \(19.5-20=-0.5\) units. Both groups depart from calibration, despite the modest size of the errors.
The mean prediction error is $$ \frac{2(0.5)+2(-0.5)}{4}=0\text{ units}. $$ The MAE is $$ \frac{4(0.5)}{4}=0.5\text{ units}, $$ and the RMSE is $$ \sqrt{\frac{4(0.5^2)}{4}} =\sqrt{0.25} =0.5\text{ units}. $$ The errors are small in outcome units, but the predicted values still do not match the group-average outcomes. A small MAE or RMSE describes error magnitude; it does not establish calibration.
Calibration Does Not Guarantee Individual Accuracy
The definition refers to a conditional average. It places no requirement that individual outcomes be close to their predictions. Consequently, predictions can be calibrated and still have large errors for particular cases, or even a large typical error overall. This distinction matters whenever predictions are used to make decisions about individuals: group-level agreement should not be mistaken for certainty about one person's outcome.
Worked Example: Calibrated Predictions With Large Individual Errors
Suppose a model gives the same prediction, 15 units, to four hypothetical cases. Their observed outcomes are 0, 10, 20, and 30 units. In this finite set, the average outcome among cases receiving the prediction of 15 is $$ \frac{0+10+20+30}{4} =\frac{60}{4} =15\text{ units}. $$ Thus, the group’s average outcome agrees exactly with its prediction.
The individual prediction errors, using observed minus predicted, are $$ 0-15=-15,\quad 10-15=-5,\quad 20-15=5,\quad 30-15=15\text{ units}. $$ The errors have substantial magnitudes, even though they average to zero. Their MAE is $$ \frac{15+5+5+15}{4} =10\text{ units}, $$ and their RMSE is $$ \sqrt{\frac{15^2+5^2+5^2+15^2}{4}} =\sqrt{\frac{500}{4}} =\sqrt{125} \approx 11.18\text{ units}. $$ In this example, calibration at the single prediction value describes the average outcome for the group; it does not make the prediction precise for each case. In a larger population, observing this agreement in a finite sample would not prove that the population conditional mean is exactly 15. It would be evidence about calibration that must be judged with the amount and quality of available data in mind.
Assessing Calibration in Practice
For a finite evaluation data set, exact repeated prediction values may be uncommon, especially when predictions are continuous. Calibration is therefore assessed by comparing average outcomes with predictions among cases that have similar predicted values, with appropriate care about how those cases are grouped or summarized. The central question remains the same: does the average observed outcome agree with the prediction in the region being examined? A single overall average cannot answer that question across the full range of predictions.
Calibration should also be interpreted in the context of the outcome and the cases being evaluated. A prediction in hours should be compared with outcomes measured in hours, not with values on a different scale. And a calibration statement applies to the population represented by the evaluated cases; it does not automatically guarantee the same agreement in a different population.
Calibration and error metrics are complementary. Calibration describes conditional average agreement, while MAE and RMSE describe the magnitude of individual errors. Mean prediction error summarizes overall signed bias. A careful account of prediction performance keeps these questions separate rather than treating any one number as a complete description.
Common Mistake
Wrong reasoning: “The mean prediction error is zero, so the predictions are calibrated.”
Correction: A zero overall mean prediction error means positive and negative errors cancel across all cases. Calibration requires zero mean error among cases with each predicted value (in the population, expressed as \(E[Y-\hat{Y}\mid\hat{Y}]=0\)). Check agreement across prediction values rather than inferring it from the overall average.
A second mistake is to treat calibration as proof that a prediction is accurate for an individual. Calibration only concerns the average outcome among cases sharing a prediction. Individual errors can still be large, so calibration should be considered alongside measures of error size.
Check Your Understanding
Answer each question using the distinction between conditional agreement and overall error.
- In your own words, state the conditional-mean definition of calibration for a continuous outcome.
- A model has a mean prediction error of zero. What can you conclude about its overall signed errors, and what can you not conclude about calibration?
- For cases receiving a prediction of 30 units, the mean observed outcome is 32 units. What is the conditional mean prediction error for that prediction value, using observed minus predicted?
- Can predictions be calibrated while having a large MAE? Explain why or why not.
- Why does a small RMSE not, by itself, show that predictions are calibrated?