Diagnostic Metrics for Prediction Error

Why Summarize Prediction Errors?

A prediction plot can show whether predictions track observations, but a numerical summary is useful when a report needs to describe the overall size and direction of prediction errors. Three common summaries are bias, mean absolute error, and root mean squared error. They answer related but different questions: is there an average signed offset, how large are errors without regard to direction, and how strongly do large errors affect the summary?

A metric is only interpretable when its construction is explicit. In particular, the sign convention, prediction type, measurement scale, observations included, and aggregation unit matter. A value called “RMSE” without these details may not be reproducible or comparable with another reported RMSE.

Define the Paired Error and the Metrics

Suppose \(n\) matched observations and predictions are available. For observation \(i\), let \(y_i\) be the observed value and \(\hat{y}_i\) the corresponding predicted value. The pair must refer to the same record and use compatible units. Throughout this tutorial, define the prediction error as observed minus predicted:

With this convention, a positive error means the prediction is below the observation; a negative error means the prediction is above it. The definitions below describe the selected set of \(n\) errors. They do not require a probability distribution or independence assumption: they are descriptive calculations on the values included.

A positive bias indicates that observations exceed predictions on average, so predictions tend to be low in the signed average. A negative bias indicates that predictions tend to be high in the signed average. Bias can be close to zero even when individual errors are large, because positive and negative errors cancel. MAE measures average absolute error and is not reduced by that cancellation. RMSE also has the error’s units, but squaring before averaging gives relatively more influence to large errors.

Here, “bias” means the average signed error in the reported data. It should not automatically be read as the formal statistical bias of an estimator, which is defined relative to repeated-sampling expectation and a target parameter.

Calculate the Metrics and Check Their Relationship

For a fixed set of errors, the relationship between RMSE and bias can be derived by writing each error as its mean plus a deviation. Let \(\bar e=\frac{1}{n}\sum_i e_i\), which is the bias. Since \(\sum_i(e_i-\bar e)=0\), the cross-product term vanishes:

The first term is the mean squared deviation of errors around their mean, using divisor \(n\). It is nonnegative, so RMSE is at least the absolute value of bias. This identity also explains why a low bias alone does not establish small errors: the errors may be widely dispersed around a mean near zero.

The Scale Is Part of the Result

The same paired observations can produce substantially different summaries depending on the scale on which errors are calculated. For example, one may calculate errors in concentration units or calculate differences after taking logarithms. The latter measures discrepancies in log units and reflects ratios rather than additive differences. State the transformation and units (or dimensionless log scale) wherever the metrics are reported.

A transformation also changes how large discrepancies are emphasized. If multiplicative differences are scientifically relevant, log-scale error metrics may be useful, provided the values being logged are positive and the chosen transformation is stated. They are not interchangeable with metrics on the original scale, and converting a log-scale summary back to the original scale does not generally produce the corresponding original-scale MAE or RMSE.

Choose the Aggregation Unit Explicitly

When records are clustered within subjects, a pooled calculation gives each included record equal weight. A subject with more records therefore contributes more to the pooled metric. A different report may first calculate a metric for each subject and then average those subject-level metrics, giving each subject equal weight. Neither aggregation is universally correct; each answers a different descriptive question. State which one was used and how many records and subjects contributed.

Reporting, Diagnostics, and Limitations

A useful report makes the calculation reproducible. State the prediction type, the error sign convention, the scale, the aggregation, and the number of contributing observations. Include units, and identify any weighting if observations do not contribute equally. For example: “Across 120 paired observations, using observed minus individual prediction on the original concentration scale, pooled bias was 0.12 mg/L, MAE was 1.8 mg/L, and RMSE was 2.6 mg/L.” The numbers in this sentence are illustrative; the reporting structure is the important point.

These summaries complement rather than replace residual plots. One overall metric cannot show whether error changes with time, prediction magnitude, or another covariate; residual diagnostics address patterns that an average conceals. Nor does a metric by itself establish whether a model is adequate for a particular decision. Repeated measurements from the same subject also mean that the pooled records are not necessarily independent observations. That dependence does not invalidate the descriptive arithmetic, but it matters if the summaries are used for inferential claims or uncertainty calculations.

Common alternatives answer different questions. MAE is less sensitive than RMSE to a very large error, while RMSE emphasizes such errors. The median absolute error is even less affected by a few extremes but describes a typical error rather than the average absolute magnitude. Percentage-based measures can make errors comparable across scales, but may become undefined or unstable when observations are zero or near zero. A normalized metric requires an explicitly stated denominator and can be difficult to compare across data sets if the normalization differs. These alternatives should not be substituted without naming and justifying the target quantity.

• Cancellation mistaken for accuracy: A bias near zero does not imply small errors. Check MAE or RMSE and inspect the error pattern.
• Unstated sign convention: A positive bias has no clear meaning unless the error direction is defined. State whether errors are observed minus predicted or the reverse.
• Scale omitted: An RMSE in mg/L is not comparable in meaning to an RMSE in log units. Name the scale and units.
• Unequal record counts ignored: Pooled metrics weight records, not subjects. If equal subject contribution is intended, aggregate by subject and label the resulting summary.
• Large errors hidden by one number: RMSE is sensitive to extremes, but neither RMSE nor MAE identifies which observations contribute most. Review the underlying diagnostic plots and data.

A concise, well-labeled set of metrics gives readers complementary information: bias describes signed average error, MAE describes average absolute discrepancy, and RMSE describes squared-error-weighted discrepancy. Interpreting them together requires attention to the scale, the aggregation unit, and the diagnostic patterns they cannot display.