Why Prediction Errors May Have Different Spread
A prediction error of a few dollars may be substantial for a small claim but relatively unimportant for a large one. Likewise, predictions for some ranges of a predictor may be much less certain than predictions for others. A model can have errors centered near zero throughout its range and still have error spread that changes substantially. That pattern is called heteroscedasticity.
Heteroscedasticity concerns the variance of errors, not whether errors systematically overpredict or underpredict. Separating those questions matters: an average residual near zero does not establish that prediction uncertainty is uniform, and a changing residual spread does not by itself establish mean bias.
In a correctly specified conditional-mean model, \(E[Y\mid X]=m(X)\), the prediction error relative to the true conditional mean has conditional expectation zero: \(E[Y-m(X)\mid X]=0\). Its variance can nevertheless vary with \(X\). In practice, \(\hat{y}_i\) estimates \(m(X_i)\), so residuals can also reflect estimation error and mean-model misspecification. A residual plot is therefore diagnostic evidence, not a direct observation of the true conditional variance.
How to Look for Changing Residual Spread
Start with residuals plotted against fitted values. A roughly horizontal cloud whose vertical spread increases or decreases across the fitted-value range suggests changing error variance. A similar plot against an important continuous predictor can reveal variation that is obscured when only fitted values are considered. For categorical predictors, compare residual spreads across groups.
A useful check is to divide the fitted values or predictor into a small number of meaningful ranges, then calculate the mean residual and a measure of residual spread in each range. The mean residual helps assess systematic overprediction or underprediction. The variance or mean squared deviation describes spread around that range's mean residual. For descriptive summaries, define the within-group mean squared deviation as \(s_g^{2*}=\frac{1}{n_g}\sum_{i\in g}(e_i-\bar e_g)^2\). The denominator is \(n_g\), not \(n_g-1\); this quantity describes the observed group spread and is not the usual unbiased sample-variance estimator.
Plot residuals against fitted values, a relevant predictor, or a categorical group. Use validation residuals when the question concerns performance on new observations.
Look for a systematic shift above or below zero, or calculate the mean residual by range. A changing center is evidence to investigate mean bias or mean-model misspecification.
Use plots and group summaries such as mean squared deviations. A widening or narrowing spread with little change in the center is evidence consistent with heteroscedasticity.
Small groups produce unstable variance estimates, and outlying observations can strongly affect squared-residual summaries. Confirm that a pattern is not driven by sparse ranges, data errors, or a changing residual mean.
Worked Example: A Fan-Shaped Residual Pattern
A hypothetical validation set is divided into three fitted-value ranges. The residuals, in dollars, are as follows: low fitted values, \(-1,1,-2,2\); middle fitted values, \(-2,2,-3,3\); high fitted values, \(-4,4,-6,6\). Calculate each range's mean residual and mean squared deviation, then assess whether the pattern indicates changing spread or changing mean.
Method. Calculate the residual mean in each range, then calculate the average squared deviation from that range's mean. This separates the center from the spread and avoids calling a larger absolute error a bias without checking its sign.
For the low range, the residual sum is \(-1+1-2+2=0\), so \(\bar e_{\text{low}}=0/4=0\). Its squared deviations sum to \(1+1+4+4=10\), giving \(s_{\text{low}}^{2*}=10/4=2.5\) dollars squared.
For the middle range, the sum is \(-2+2-3+3=0\), so \(\bar e_{\text{mid}}=0\). The squared deviations sum to \(4+4+9+9=26\), giving \(s_{\text{mid}}^{2*}=26/4=6.5\).
For the high range, the sum is \(-4+4-6+6=0\), so \(\bar e_{\text{high}}=0\). The squared deviations sum to \(16+16+36+36=104\), giving \(s_{\text{high}}^{2*}=104/4=26\).
All three range means are zero, while the mean squared deviation rises from \(2.5\) to \(6.5\) to \(26\). The pattern is consistent with increasing residual variance across fitted values, not with a directional mean error in these ranges. As a check, the total squared residual sum is \(10+26+104=140\), and the overall mean squared residual is \(140/12=11.6667\). Its square root is an overall RMSE of approximately \(3.416\) dollars, which alone does not show how different the group spreads are.
Common mistake avoided. The high-range residuals are larger in magnitude, but they are balanced between positive and negative values. Calling them evidence of overprediction would confuse spread with mean bias.
Why the Overall Error Summary Can Hide the Pattern
A pooled mean squared error combines the errors from every range. It is a useful summary for the particular mix of observations evaluated, but it does not show whether all ranges have similar error spread. The reason can be stated exactly by separating each residual into its difference from its group mean and the group mean itself.
For groups \(g\) containing \(n_g\) observations, let \(N=\sum_g n_g\), let \(\bar e_g\) be the mean residual in group \(g\), and let \(s_g^{2*}=\frac{1}{n_g}\sum_{i\in g}(e_i-\bar e_g)^2\). The pooled mean squared residual has the decomposition below.
To justify it, write \(e_i=(e_i-\bar e_g)+\bar e_g\) for each observation in group \(g\), square, and sum. The cross-product vanishes because \(\sum_{i\in g}(e_i-\bar e_g)=0\). The remaining within-group squared deviations give the first term; the squared group means give the second. Thus pooled mean squared error includes both within-group spread and variation in group-level mean error. Its square root is the pooled RMSE.
This identity also explains why an overall mean residual near zero does not rule out errors in particular ranges. Positive and negative group means can cancel overall. Conversely, a rising within-group spread can occur even when every group mean is zero. These are different features of the errors and should be reported separately when material.
Worked Example: A Zero Overall Mean Can Hide Group Bias
A hypothetical model is evaluated in three equally sized fitted-value ranges, with two residuals per range: low, \(-3,-1\); middle, \(0,4\); high, \(-6,6\). Calculate the range means, within-range mean squared deviations, pooled mean residual, and pooled mean squared error. Use the decomposition to explain what the overall summaries conceal.
Method. First summarize each range, using deviations from its own mean. Then apply the decomposition with equal group weights \(2/6=1/3\).
For the low range, the mean is \((-3-1)/2=-2\). Deviations from \(-2\) are \(-1,1\), so the mean squared deviation is \((1+1)/2=1\).
For the middle range, the mean is \((0+4)/2=2\). Deviations are \(-2,2\), so the mean squared deviation is \((4+4)/2=4\).
For the high range, the mean is \((-6+6)/2=0\). Deviations are \(-6,6\), so the mean squared deviation is \((36+36)/2=36\).
The overall residual sum is \((-3-1)+(0+4)+(-6+6)=-4+4+0=0\), so the pooled mean residual is \(0/6=0\). Yet the low and middle group means are \(-2\) and \(2\), respectively: those group-level errors cancel in the overall mean.
The pooled mean squared error is the total squared residual sum divided by six. The squared residual sums are \(9+1=10\), \(0+16=16\), and \(36+36=72\). Therefore, the pooled mean squared error is \((10+16+72)/6=98/6=16.3333\), and the pooled RMSE is \(\sqrt{16.3333}\approx4.0415\).
The decomposition gives the same result: the weighted within-range spread is \((1+4+36)/3=41/3=13.6667\), and the weighted squared group means are \(((-2)^2+2^2+0^2)/3=8/3=2.6667\). Their sum is \(49/3=16.3333\). Rounded to four decimals, the RMSE is \(4.0415\). The overall mean residual of zero masks both range-specific mean errors and a much larger spread in the high range.
Common mistake avoided. A zero pooled mean residual does not prove the residuals are centered within every range. Check subgroup means as well as subgroup spread.
Consequences for Uncertainty and Error Interpretation
When residual spread changes, a single residual standard deviation may be a poor description of uncertainty for an individual prediction. A scale estimated over the whole validation set reflects that set's composition: if one range is overrepresented, it contributes more to the pooled scale. Applying that one scale uniformly can overstate uncertainty in some ranges and understate it in others.
The following calculation illustrates the implication under a specific assumption. Suppose, hypothetically, that errors conditional on the fitted-value range are normally distributed with mean zero and known standard deviations. For a normal error with standard deviation \(\sigma_g\), a central 95% error interval is approximately \([-1.96\sigma_g,1.96\sigma_g]\). This is an interval for outcome error under the stated distribution, not a guarantee of coverage for an estimated model.
Worked Example: Range-Specific Prediction Uncertainty
A hypothetical validation analysis supports the working assumptions \(e\mid\text{low range}\sim N(0,2^2)\) and \(e\mid\text{high range}\sim N(0,5^2)\), where the second parameter is the variance. Calculate the approximate central 95% error interval in each range, and compare them with an interval formed from one pooled scale when three quarters of observations are in the low range and one quarter are in the high range.
Method. Use the normal quantile multiplier \(1.96\) for each conditional standard deviation. For the illustrative pooled scale, take the square root of the weighted conditional variances; both conditional means are zero.
In the low range, \(\sigma_{\text{low}}=2\), so the interval is \([-1.96(2),1.96(2)]=[-3.92,3.92]\). In the high range, \(\sigma_{\text{high}}=5\), giving \([-1.96(5),1.96(5)]=[-9.80,9.80]\).
The weighted variance is \(0.75(2^2)+0.25(5^2)=0.75(4)+0.25(25)=3+6.25=9.25\). The corresponding pooled standard deviation is \(\sqrt{9.25}\approx3.0414\), giving a common interval of approximately \([-1.96(3.0414),1.96(3.0414)]=[-5.9611,5.9611]\).
That common interval is wider than the low-range interval and narrower than the high-range interval. It therefore does not represent the same conditional uncertainty in both ranges. These calculations assume the conditional normal distributions and known parameters; practical intervals may also need to account for estimation uncertainty, and normality should not be assumed merely because a residual plot has a fan shape.
Common mistake avoided. A pooled scale is not automatically a suitable range-specific uncertainty estimate. The calculation is conditional on the stated distribution and variance assumptions, which must be checked rather than inferred from an overall RMSE.
Interpretation and Common Traps
Heteroscedasticity is not synonymous with a model that predicts too high or too low. A trend in the residual center calls for investigation of mean bias or misspecification; a trend in spread calls for investigation of changing variance. Both may occur together. For example, residuals could become more dispersed at higher fitted values while also being mostly positive there.
For ordinary least squares, changing conditional variance does not, by itself, bias coefficient estimates when the conditional mean is correctly specified and the predictors are exogenous. It does mean that the usual constant-variance formula for coefficient uncertainty is not appropriate; under the relevant conditions, heteroscedasticity-robust standard errors can address inference about coefficients. Robust standard errors do not remove the changing outcome uncertainty or fix a misspecified conditional mean. For prediction assessment, retain the distinction between average error and error spread.
- Do not infer bias from magnitude alone. Large positive and negative residuals can indicate high spread with a mean near zero.
- Do not infer constant uncertainty from a small pooled RMSE. The pooled metric depends on the evaluated population's range composition and can conceal a high-variance segment.
- Do not overinterpret a few extreme observations. Squared deviations are sensitive to large residuals, and small groups yield unstable spread estimates.
- Do not treat a residual plot as proof of a variance model. A visible pattern is a diagnostic clue; investigate data quality, mean structure, and relevant predictors before selecting a remedy.
Check Your Understanding
For each question, distinguish residual center from residual spread and show the calculation indicated.
- A low fitted-value range has residuals \(-2,2\), and a high range has residuals \(-5,5\). Find each range's mean residual and mean squared deviation. What pattern is present? (Answer: Both means are \(0\); the mean squared deviations are \(4\) and \(25\), respectively, so the spread is larger in the high range with no range-level mean bias in these data.)
- Two fitted-value ranges each contain four residuals. The low-range residuals are \(-1,1,-1,1\); the high-range residuals are \(-2,2,-4,4\). Calculate each range's mean squared residual, the pooled mean squared residual, and the pooled RMSE. (Answer: The range mean squared residuals are \(1\) and \(10\); the pooled mean squared residual is \((4+40)/8=5.5\), and the pooled RMSE is \(\sqrt{5.5}\approx2.3452\).)
- Two equally sized ranges have residuals \(-2,2\) and \(1,5\). Find each range's mean and mean squared deviation, then use the decomposition to calculate the pooled mean squared residual. (Answer: The means are \(0\) and \(3\), and the within-range mean squared deviations are \(4\) and \(4\); pooled MSE is \((4+13)/2=8.5\), equal to the average within-range spread \(4\) plus the average squared group mean \(4.5\).)
- Suppose errors are conditionally normal with mean zero and standard deviation \(3\) in one predictor range and \(7\) in another. Using \(1.96\) as the central 95% normal multiplier, calculate both error intervals. (Answer: The intervals are \([-5.88,5.88]\) and \([-13.72,13.72]\), respectively.)
- A residual plot has a widening vertical cloud as fitted values increase, but residual means in three broad ranges are \(-4,0,4\). What are the two distinct features to report, and why is the overall mean residual insufficient? (Answer: Report both the increasing spread and the changing range means; the range means indicate potential mean error, and the negative and positive values can cancel in an overall mean.)