Binned Residual Means as a Calibration Check

Why Check Residual Means Across Fitted Values?

A regression model can have residuals that average to zero overall and still systematically miss in parts of its fitted-value range. For example, it might tend to underpredict for cases with low fitted values and overpredict for cases with high fitted values. Looking at residuals one by one can make such a pattern hard to see. Grouping fitted values into ranges and averaging the residuals within each range gives a compact check of whether the model’s errors balance across the scale of its predictions.

This is a descriptive calibration check for a continuous response. Here, calibration across fitted values means that, within ranges of fitted values, observed responses do not show a consistent average departure from the model’s fitted values. It does not mean that every individual prediction is accurate. Nor does a pattern in the check, by itself, identify the cause of a model’s shortcomings.

Definition and Interpretation

Suppose a regression model is fitted to \(n\) observations. For observation \(i\), let \(y_i\) be the observed response, \(\hat{y}_i\) its fitted value, and \(e_i=y_i-\hat{y}_i\) its residual. The residual is in the response’s units: it is positive when the observed response exceeds the fitted value, and negative when the observed response is below it.

The binning variable is the fitted value, not necessarily an individual predictor. A bin can contain observations with different predictor values and different fitted values, so its mean residual summarizes the average discrepancy for that portion of the model’s fitted-value scale. It is useful to report the range, the number of observations, and the mean residual for every bin.

The standard regression model for a continuous response can be written \(Y_i=m(X_i)+\varepsilon_i\), where \(m(X_i)\) is the conditional mean response at the predictors \(X_i\), and \(\varepsilon_i\) is the error. For a linear regression model, the proposed mean structure is linear in the included predictors. The calibration check is most informative when the recorded responses and predictors are valid, the fitted model is the one intended for the analysis, and its fitted values meaningfully summarize the proposed mean structure. A residual pattern can then provide evidence that the fitted mean structure does not capture the response equally well across its fitted-value range. The check is exploratory: it does not establish that the model assumptions hold or fail.

Choosing Bins and Reading the Results

Bins should make the pattern across fitted values visible without leaving too few observations in each range. One option is to use ranges of equal width on the fitted-value scale. Another is to sort observations by fitted value and divide them into groups with roughly equal counts. Equal-width bins make the scale of the ranges easy to compare; roughly equal-count bins avoid very small groups when fitted values are concentrated in some parts of the scale. Neither choice is automatically best.

Before interpreting the pattern, check how many observations are in each bin and whether the bin boundaries have created an empty or especially small group. A mean from a small group can be strongly affected by one residual. Also check the units: the mean residual has the same units as the response, so the size of the departure can be judged in context. The direction alone does not say whether the departure is practically important.

A Useful Identity and Its Limitation

The relation between binned and overall residual means follows directly from the definition. If the bins partition all observations, each residual appears in exactly one bin. Therefore, adding the bin residual sums recovers the total residual sum:

Dividing by \(n=\sum_j n_j\) gives the overall mean residual as a weighted average of the bin means:

For ordinary least squares with an intercept, the residuals sum to zero, so \(\bar{e}=0\). The identity then says only that the bin means balance to zero after weighting by their group sizes. It does not require every bin mean to equal zero. This is why a zero overall residual mean is not evidence that the model is calibrated in every part of the fitted-value scale.

What the Check Can and Cannot Establish

A sustained positive mean over a fitted-value region is evidence of average underprediction there; a sustained negative mean is evidence of average overprediction. If the means remain near zero across a sensible set of bins, there is no obvious mean discrepancy at that level of grouping. But near-zero binned means do not prove that every individual prediction is accurate, that the model is correctly specified, or that other assumptions are satisfied. A pattern might also be hidden by wide bins that combine regions with opposite-signed departures.

Binned means are descriptive summaries, not a formal test with an automatic significance threshold. Their values alone do not provide a p-value or quantify the uncertainty in each mean. A small difference might be unimportant in context, while a larger difference may merit investigation. The residual-versus-fitted plot can show the individual points alongside bin summaries, helping distinguish a broad mean pattern from a result driven by only a few observations.

This check is useful because it condenses many residuals into a scale-ordered view of average prediction error. It can flag where the model’s fitted values tend to sit above or below observed responses, guiding a careful review of the model and data. It does not, by itself, tell the analyst which change should be made.

Common Mistakes

• Claiming that a zero overall mean proves calibration everywhere. Opposite-signed bin means can cancel. Check the weighted mean within each fitted-value range.
• Reversing the interpretation of the sign. Since \(e_i=y_i-\hat{y}_i\), a positive mean means observed responses exceed fitted values on average: the model underpredicts. A negative mean means overprediction.
• Averaging bin means without regard to bin size. When sizes differ, the overall mean weights each bin by its number of observations, not equally across bins.
• Treating one binning choice as conclusive. Boundaries affect the summaries. Check whether the pattern is reasonably stable under sensible alternative groupings.
• Calling the check a proof or a formal test. Binned means are a descriptive diagnostic. They can reveal a pattern but cannot, alone, establish model adequacy or explain the pattern’s cause.