Quantifying Prediction Error With Fold Error And Bias

Why Use More Than One Prediction-Error Measure?

An independent PBPK prediction can be close to observed data in one sense and misleading in another. A set of predictions may have a small average signed error because overpredictions and underpredictions cancel, even while individual predictions differ substantially. Conversely, a consistent small concentration difference may matter differently at a low-concentration endpoint than at a high-concentration endpoint. No single summary describes all these features.

This tutorial defines three complementary summaries for matched observed and predicted values: geometric fold error, mean prediction error, and endpoint-specific bias. They summarize prediction performance; they do not establish why errors occurred. Interpret them alongside the observed-predicted plots and time-structured residual diagnostics used in validation.

The fold error is at least 1. A value of 1 means exact agreement for that observation; a value of 2 means that the prediction is twice the observation or half the observation. The geometric fold error is the geometric mean of those individual fold errors. It summarizes the typical multiplicative discrepancy without retaining its direction. The mean prediction error retains direction: a positive value indicates average overprediction in the endpoint’s units, and a negative value indicates average underprediction.

Calculating and Interpreting the Metrics

The geometric fold error requires strictly positive observed and predicted values because it uses ratios and logarithms. It is symmetric to exchanging observation and prediction: a ratio of 2 and a ratio of 0.5 both have fold error 2. This is useful when multiplicative discrepancies are meaningful, but it means the metric cannot say whether predictions tend to be high or low. The MPE supplies a signed summary but is expressed in the original endpoint’s units.

A related directional summary is the mean log ratio, \(\frac{1}{n}\sum_i \ln(P_i/O_i)\). Exponentiating it gives the geometric mean prediction-to-observation ratio. A value above 1 indicates an average multiplicative tendency to overpredict; a value below 1 indicates an average tendency to underpredict. Unlike the GFE, this directional ratio can be close to 1 when large errors in opposite directions cancel. It is not the same as arithmetic MPE or mean relative error.

Bias Must Be Reported by Endpoint

For an endpoint \(j\), define its endpoint-specific MPE as the average of \(P_{ij}-O_{ij}\) across the matched validation observations for that endpoint. This is an arithmetic bias in the endpoint’s units. A dimensionless companion is mean relative prediction error, calculated as the average of \((P_{ij}-O_{ij})/O_{ij}\), often reported as a percentage. These summaries answer different questions: the first gives the average signed difference on the measurement scale, while the second gives the average signed difference relative to each observed value. Neither should be confused with the signed log-ratio summary.

Endpoints such as maximum concentration and area under the concentration-time curve generally have different units and scales. Their raw errors should not be pooled into one MPE: the numerical result would mix unlike quantities and could be dominated by the endpoint with larger numerical values. Report each endpoint’s number of pairs, units, MPE, and—when observations are positive and a relative summary is relevant—its relative bias and GFE separately.

Why the Geometric Fold Error Uses Logarithms

The log transformation makes multiplicative errors additive. Let \(r_i=P_i/O_i>0\). The fold error is \(\max(r_i,1/r_i)\), which equals \(\exp(|\ln r_i|)\). Taking the geometric mean across \(n\) observations therefore gives the following derivation:

This establishes the stated GFE formula and its reciprocal symmetry: replacing every ratio \(r_i\) with \(1/r_i\) leaves each \(|\ln r_i|\), and hence the GFE, unchanged. The absolute value also removes direction, so the signed MPE or a directional relative or log-ratio summary is needed to describe average bias.

Diagnostics, Alternatives, and Limitations

These summaries should be calculated from correctly matched observed-predicted pairs, using the same endpoint definition and units for each pair. The predictions should be the independent predictions intended for evaluation, not values refitted to the validation observations. Otherwise the summaries describe an in-sample or recalibrated fit rather than independent predictive performance.

Check the underlying pairs and plots when a summary appears reassuring. A near-zero MPE can result from cancellation; a GFE near 1 can conceal a small number of very large discrepancies if most errors are small; and a relative bias can be strongly affected by small positive observations because each error is divided by its own observed value. Also check whether a single unusual pair dominates an arithmetic mean. A plot by time or endpoint may reveal structure that a single aggregate metric cannot.

Zero or negative values cannot be used in the ratio-and-log GFE as defined. Do not silently add a constant or replace zeros with an arbitrary small value: that changes the metric and can materially change the result. Report how such pairs were handled and use a clearly justified alternative for those data. The arithmetic MPE remains defined when values are zero, although its interpretation still depends on endpoint, scale, and data quality.

The MPE and mean relative prediction error are signed averages, so they can hide opposing errors. The GFE is unsigned and multiplicative, so it does not identify direction. Mean absolute error, the average of \(|P_i-O_i|\), avoids cancellation but remains in endpoint units; root mean squared error gives greater influence to large absolute errors. These alternatives can supplement the three metrics, but none makes endpoint-specific reporting unnecessary. Correlation is not a substitute: predictions can track variation while being systematically shifted or scaled away from the observed values.