Prediction Error Across Outcome Ranges

Why Compare Errors Across Outcome Ranges?

A model can have an acceptable overall error while performing quite differently for small and large outcomes. For example, predictions may be close for values near the middle of the response scale but consistently too low at its upper end. An overall metric compresses those cases into one number, so comparing errors across ranges of the observed outcome can reveal where prediction errors are concentrated.

This comparison groups validation observations according to their observed response values, then summarizes the errors within each group. It is distinct from examining errors across ranges of predicted values: here, the grouping variable is what was actually observed. The aim is descriptive—identify differences in error patterns across the outcome scale, not establish that outcome level caused those differences.

Definition: For observation \(i\), let \(Y_i\) be the observed outcome and \(\hat{Y}_i\) its prediction. Define the prediction error as \(e_i=Y_i-\hat{Y}_i\). A positive error means the prediction was too low (underprediction); a negative error means it was too high (overprediction). An outcome range is a specified interval of observed \(Y_i\) values used to group observations for comparison.

A useful comparison begins with predictions made for data not used to fit the model. The observed outcomes and their corresponding predictions must be paired correctly, and all groups should use the same prediction procedure and outcome units. For a descriptive comparison, no normality assumption is needed. However, the evaluation data should be relevant to the setting where predictions will be used; otherwise, the comparison may not describe future performance well.

Metrics Within Each Outcome Range

For a group \(G\) containing \(n_G\) observations, calculate metrics from the errors in that group. The mean signed error describes the average direction of the errors, while the mean absolute error (MAE) and root mean squared error (RMSE) describe their magnitude in different ways.

\overline{e}_G=\frac{1}{n_G}\sum_{i\in G}e_i,\qquad \mathrm{MAE}_G=\frac{1}{n_G}\sum_{i\in G}|e_i|,\qquad \mathrm{RMSE}_G=\sqrt{\frac{1}{n_G}\sum_{i\in G}e_i^2}

Each metric is in the same units as the outcome. A positive \(\overline{e}_G\) indicates average underprediction in that range; a negative value indicates average overprediction. MAE measures typical absolute error without allowing positive and negative errors to cancel. RMSE also measures error magnitude, but squaring gives relatively greater influence to large errors. A near-zero mean signed error alone does not imply accurate predictions: sizable errors in opposite directions can cancel.

Worked Example: Comparing Low, Middle, and High Outcomes

Suppose a hypothetical validation set contains nine observations of a continuous concentration, measured in mg/L. The ranges are defined in advance as low (below 15 mg/L), middle (15 to below 25 mg/L), and high (25 mg/L or more). The table lists observed values and predictions; each error is observed minus predicted.

Range	Observed \(Y_i\) (mg/L)	Predicted \(\hat{Y}_i\) (mg/L)	Error \(e_i\) (mg/L)
Low	10, 12, 14	12, 11, 16	-2, 1, -2
Middle	20, 22, 24	18, 24, 22	2, -2, 2
High	30, 32, 34	25, 35, 31	5, -3, 3

For the low range, the mean signed error is \((-2+1-2)/3=-1.00\) mg/L. Its MAE is \((2+1+2)/3=1.67\) mg/L, and its RMSE is \(\sqrt{(4+1+4)/3}=\sqrt{3}=1.73\) mg/L.

For the middle range, the mean signed error is \((2-2+2)/3=0.667\) mg/L. The MAE is \((2+2+2)/3=2.00\) mg/L, and the RMSE is \(\sqrt{(4+4+4)/3}=2.00\) mg/L.

For the high range, the mean signed error is \((5-3+3)/3=1.67\) mg/L. The MAE is \((5+3+3)/3=3.67\) mg/L, and the RMSE is \(\sqrt{(25+9+9)/3}=\sqrt{14.33}=3.79\) mg/L.

In this small hypothetical set, both magnitude metrics are largest in the high range, and its positive mean signed error indicates average underprediction. That pattern is worth investigating, but three observations per range are not enough to support a broad claim about future performance.

How Range Metrics Relate to Overall Metrics

When the ranges contain different numbers of observations, the overall MAE is not generally the simple average of the range-specific MAEs. Each group contributes in proportion to its sample size. The same principle applies to mean squared error; take the square root only after combining the group mean squared errors.

Result: If the groups form a partition of all \(n\) observations, then \(\mathrm{MAE}=\sum_j(n_j/n)\mathrm{MAE}_{G_j}\) and \(\mathrm{RMSE}=\sqrt{\sum_j(n_j/n)\mathrm{RMSE}_{G_j}^2}\), where \(n_j\) is the number of observations in group \(j\).

These identities follow by splitting the sums in the overall metrics into group-specific sums. For MAE, the sum of absolute errors in group \(j\) is \(n_j\mathrm{MAE}_{G_j}\). Adding those sums and dividing by \(n\) gives the weighted average. For RMSE, the group’s sum of squared errors is \(n_j\mathrm{RMSE}_{G_j}^2\). Adding across groups gives the overall mean squared error; its square root gives overall RMSE.

\mathrm{MAE} =\frac{1}{n}\sum_j n_j\mathrm{MAE}_{G_j} =\sum_j\frac{n_j}{n}\mathrm{MAE}_{G_j}, \qquad \mathrm{RMSE} =\sqrt{\frac{1}{n}\sum_j n_j\mathrm{RMSE}_{G_j}^2}

Worked Example: Combining Unequal-Sized Ranges

In another hypothetical validation set, the low range has two absolute errors, 1 and 3 mg/L; the middle range has three absolute errors, 1, 2, and 3 mg/L; and the high range has one absolute error, 5 mg/L. The group MAEs are \((1+3)/2=2.00\), \((1+2+3)/3=2.00\), and \(5/1=5.00\) mg/L.

There are six observations altogether. The overall MAE is \((4+6+5)/6=15/6=2.50\) mg/L. Equivalently, weighting the group MAEs by their sample sizes gives \((2/6)(2)+(3/6)(2)+(1/6)(5)=2.50\) mg/L.

The unweighted average of the three group MAEs is \((2+2+5)/3=3.00\) mg/L, which is not the overall MAE. It gives the single-observation high range the same weight as the three-observation middle range. For RMSE, one must weight group mean squared errors, not average group RMSE values directly.

Reading the Pattern Carefully

A range-by-range summary is most informative when it reports both error direction and error magnitude. If the mean signed error changes from negative at low outcomes to positive at high outcomes, predictions may be too high in the low range and too low in the high range on average. If MAE or RMSE increases across ranges, errors are larger in magnitude at one part of the observed outcome scale. These patterns can occur together, but they answer different questions.

Worked Example: Why Signed Error Is Not Enough

Consider a hypothetical comparison of two outcome ranges, with errors measured in mg/L. The low range has errors \(-4\) and \(4\); the high range has errors \(2\) and \(2\).

In the low range, the mean signed error is \((-4+4)/2=0\) mg/L, but the MAE is \((4+4)/2=4.00\) mg/L and the RMSE is \(\sqrt{(16+16)/2}=4.00\) mg/L. The zero average reflects cancellation, not small errors.

In the high range, the mean signed error is \((2+2)/2=2.00\) mg/L, while both MAE and RMSE are \(2.00\) mg/L. Thus the high range shows consistent underprediction, but smaller error magnitudes than the low range. Reporting only signed error would miss the larger low-range errors; reporting only MAE or RMSE would miss the direction of the high-range errors.

Observed-outcome grouping also requires a particular caution: the error \(e_i=Y_i-\hat{Y}_i\) contains the observed outcome itself. Sorting observations by \(Y_i\) can therefore produce a relationship between outcome range and signed error even when the model’s predictions do not change across those observations. For instance, if every prediction equals the same constant \(c\), then \(e_i=Y_i-c\): outcomes below \(c\) have negative errors and outcomes above \(c\) have positive errors. A pattern across outcome ranges is useful descriptive evidence about where errors occur, but it is not, by itself, proof of a specific model defect or cause.

Choose ranges for a clear reason. Meaningful thresholds in the response scale can make the results easier to interpret. Quantile-based ranges can produce more similarly sized groups, but their boundaries depend on the evaluation sample and may be less meaningful in context. In either case, state the cutpoints, report the number of observations in each range, and avoid making strong comparisons when a range contains very few cases. If many possible cutpoints are tried and only the most striking pattern is reported, the apparent pattern can be misleading.

A Practical Reporting Sequence

Set the evaluation conditions.
Use paired observed outcomes and predictions from the same held-out evaluation data, with predictions produced without using those outcomes to fit the model.

Define outcome ranges.
State the cutpoints and units, and give a substantive or practical reason for the grouping choice.

Calculate group errors and metrics.
For each observation calculate \(e_i=Y_i-\hat{Y}_i\), then summarize signed error and at least one magnitude metric within each range.

Report counts and interpret cautiously.
Include each range’s sample size, metric values, units, and the limits of a descriptive comparison.

Common Mistake

Wrong reasoning: “The mean signed error is zero in the low-outcome range, so predictions there are accurate.” This conclusion ignores the possibility that positive and negative errors cancel. Correction: inspect a magnitude measure such as MAE or RMSE as well as signed error. Also avoid treating a difference between ranges as causal or definitive: the comparison describes the evaluated observations, and the outcome-based grouping itself can shape the pattern.

Key takeaway: Compare prediction errors across observed-outcome ranges by reporting direction and magnitude, with group sizes and units. Treat the result as a description of performance across the outcome scale—not, by itself, as an explanation of why errors differ.

Check Your Understanding

Use these questions to check your interpretation and calculations.

If an outcome-range mean signed error is \(-1.2\) mg/L, what does its sign say about the predictions in that range?
A range has errors \(-3\) and \(3\) mg/L. Calculate its mean signed error and MAE. Why do the two summaries tell different things?
Two ranges have MAEs of 2 mg/L, but one contains 20 observations and the other contains 5. Is the overall MAE their simple average? Explain.
Why can sorting observations into ranges of \(Y_i\) create a pattern in errors even when every prediction is the same constant?
Name two details, besides metric values, that should be included when reporting error comparisons across outcome ranges.