Setting Predictive Error Criteria Before Seeing Results

Why Set Prediction Limits in Advance?

A validation result is useful only in relation to a stated standard for acceptable predictive performance. A twofold error might be tolerable for an exploratory estimate of a concentration profile but too large for a decision that depends on a narrow exposure margin. Likewise, a one-hour error in time to peak may be negligible for one sampling schedule and consequential for another. There is no single error limit that is automatically appropriate for every PBPK application.

Setting criteria before inspecting validation outcomes makes the assessment interpretable: the rules represent the intended use rather than a response to the results. It also prevents a model from being labelled acceptable by choosing a forgiving metric or threshold after its errors are known. Criteria should be fixed before unblinding the predictions and observations, and should state the endpoints, error scales, summaries, thresholds, and units used in the decision.

The earlier material on fold error and bias provides useful measures of prediction error. Here the focus is how to choose and apply acceptance criteria for those measures. A criterion is a prespecified decision rule, not a claim that any particular numerical threshold is universally correct.

Define the Endpoint, Scale, and Decision Rule

For a positive observed value \(Y\) and its corresponding prediction \(\hat{Y}\), the prediction ratio is \(R=\hat{Y}/Y\). A fold-error bound \(F\), with \(F>1\), classifies a pair as within the bound when \(1/F \leq R \leq F\). This multiplicative scale treats an overprediction and its reciprocal underprediction comparably: a prediction twice the observation has ratio 2, while one half the observation has ratio 0.5.

The last form follows by taking logarithms of the positive ratio and expressing the two-sided bounds around zero. It shows why fold errors are symmetric on a log scale, even though the corresponding absolute differences in the original units are not equal. For example, an \(F=2\) bound allows ratios from 0.5 through 2. It does not allow the same number of concentration units above and below the observation.

The criterion must also specify how multiple comparisons are combined. One option is to require a stated proportion of prespecified concentration-time pairs to fall within a fold bound, separately for each validation condition. Another is to require each condition's AUC ratio to fall inside a narrower interval. A pooled pass rate across all studies can hide a poor result in one important condition, so the aggregation rule should match the intended domain of use.

Choose Limits From the Intended Use

An endpoint-specific criterion begins with the decision the model is meant to support. The analyst should ask how large an error could be without changing that decision, which direction of error is most consequential, and how the endpoint is measured. These questions help set a defensible scale and tolerance. They do not justify tuning a rule to make a particular validation data set pass.

Concentrations, AUC, and Cmax are positive quantities and are commonly assessed with ratios or fold-error limits. A concentration-time profile has many prespecified sampling times, so its rule must state whether performance is summarized over all pairs, at each time, or within each condition. AUC and Cmax are single summary endpoints for a dosing scenario and can be assessed with their own ratio intervals. Time to peak is a time-valued endpoint: an absolute difference in hours or minutes is often easier to interpret than a ratio, particularly when the observed time is near zero or limited to scheduled sampling times.

The limits should be justified by the model's intended application, relevant prior knowledge, and the consequences of errors. They should also be specific enough to reproduce the decision. “Predictions should be reasonably close” is not a criterion; “at least 80% of prespecified concentration pairs must be within twofold in each condition” is a rule that can be applied consistently. Neither example threshold is a universal standard.

Apply Concentration Criteria Without Changing Them

For concentration data, the paired values must refer to the same prespecified time, condition, and comparison. When the criterion uses fold error, the observations must be positive for the ratio to be defined. The analysis plan should identify the time points and conditions that enter the denominator before outcomes are known. Details such as how to handle values outside assay quantification limits require their own prespecified procedures; they should not be resolved by selectively removing inconvenient pairs.

A pass proportion is descriptive: it reports the fraction of the defined pairs meeting the threshold. It is not, by itself, an estimate that future samples will meet the threshold with that probability. In particular, several time points from one profile can be related rather than independent. The criterion should therefore be understood as the agreed performance rule for the specified validation data, not as an independence-based probability statement.

Assess Summary Endpoints and Time to Peak

For AUC and Cmax, compute the prediction-to-observation ratio using the same endpoint definition for both values. Apply each endpoint's own prespecified interval. Do not substitute a concentration-profile criterion merely because it is more familiar: integrated exposure and peak concentration answer different questions and can show different errors. If the rule requires each condition to pass, report each condition rather than relying only on an overall average.

For time to peak, let \(\hat{t}_{\max}\) be the predicted time and \(t_{\max}\) the observed time. An absolute-error rule uses \(|\hat{t}_{\max}-t_{\max}|\), with units. The tolerance should reflect the resolution of the sampling schedule and the intended decision. Because observed peak time can depend on when samples are taken, a small difference should not automatically be interpreted as a precise estimate of a biological timing difference.

Justification, Diagnostics, and Alternatives

The log-ratio form provides a short justification for using fold limits when proportional error is the concern. Let \(R=\hat{Y}/Y>0\). The two-sided fold rule \(1/F \leq R \leq F\) becomes \(-\log(F)\leq\log(R)\leq\log(F)\), since the logarithm is increasing and \(\log(1/F)=-\log(F)\). Equivalently, the absolute log ratio cannot exceed \(\log(F)\). This is why reciprocal errors have equal magnitude on the chosen scale: \(|\log(2)|=|\log(1/2)|\). The result justifies the scale's symmetry; it does not establish which value of \(F\) is acceptable.

Diagnostics should check whether the prespecified rule has been applied to the intended data and whether the chosen summary is concealing a failure. Confirm that predictions and observations are paired correctly, units and endpoint definitions agree, and the denominator includes exactly the records defined in advance. Then inspect errors by time, condition, and endpoint. A pooled pass rate can appear satisfactory even when one condition systematically performs poorly; reporting condition-specific results helps expose this. If a diagnostic suggests a problem, report it and investigate under the analysis plan rather than changing the threshold to restore a pass.

Alternative error measures can be useful for different purposes, but they answer different questions. Absolute error in concentration units gives larger weight to differences at high concentrations and is sensitive to the units used. Percentage error and fold error emphasize relative discrepancy, but ratios are unsuitable when the observed value is zero and can be unstable near zero. A single mean error can also conceal large errors in opposite directions that cancel. Choosing a different measure after seeing which one makes the model look best is not a defensible alternative; any additional measures should be specified in advance and reported alongside the primary rule.

A pass/fail criterion also compresses information. A result just inside a limit and one near perfect agreement both count as passes, while a slight miss and a large miss both count as failures. Report the underlying endpoint values and error summaries with the decision so readers can judge the magnitude and pattern of discrepancies. A criterion is a practical standard for a stated use, not proof that the model is universally accurate or a statistical test that the model is correct.

Common Pitfall

Another error is to apply the same numerical tolerance to all endpoints simply for convenience. A limit of 1 hour has no meaning for AUC, while a 1.25 ratio may not be appropriate for time to peak. Endpoint-specific criteria preserve interpretable scales and make explicit which predictive performance matters for the intended decision.