External Validation and Predictive Error

Why Test an IVIVC Outside Its Development Data?

An IVIVC is useful only if it predicts in vivo performance for data that did not determine its relationship. A close match between predicted and observed pharmacokinetics for formulations used to construct a model is evidence about fit to those data; it does not, by itself, show that the model will predict a new formulation or an independent dataset. External validation asks that more demanding question.

The central rule is simple: make the prediction using the model as specified, then compare it with an observed in vivo result that was not used to construct or adjust the model. This tutorial focuses on that comparison for AUC and Cmax. The earlier material on Internal Predictability of an IVIVC addresses assessment using development formulations; external validation adds an independent test, not another opportunity to tune the model.

Definition: External validation evaluates predictions from a previously specified IVIVC against observed in vivo data from an independent formulation or dataset that was not used to select, fit, or modify the model. The model, its time alignment, and its prediction procedure are held fixed during this evaluation.

Independence is about how the model was developed, not merely whether a dataset has a different label. A formulation is not an external test if its in vivo outcomes were used to choose the model form, estimate parameters, select a lag time, or decide which data to exclude. Likewise, a new study of a familiar formulation can provide external evidence if its outcomes were genuinely withheld, the prediction was specified in advance, and the study is sufficiently comparable to the intended prediction setting.

Prediction Error for AUC and Cmax

For either endpoint, use the predicted and observed values on the same scale, for the same dose or an explicitly justified dose-normalized comparison. Let \(Y_{\text{obs}}\) be the observed endpoint and \(Y_{\text{pred}}\) the IVIVC-predicted endpoint. The signed percentage prediction error is defined with the observed value in the denominator.

\%PE = \frac{Y_{\text{obs}}-Y_{\text{pred}}}{Y_{\text{obs}}}\times 100

For AUC, apply the formula to AUC values; for Cmax, apply it to Cmax values. A positive error means the prediction is lower than the observation (underprediction). A negative error means it is higher (overprediction). The percentage is dimensionless, while the endpoint values retain their units: for example, AUC in mg·h/L and Cmax in mg/L.

Interpretation: A prediction error of \(+8\%\) for AUC means that predicted AUC is 8% below observed AUC, relative to the observed value. An error of \(-8\%\) means predicted AUC is 8% above observed AUC. The sign describes direction; the absolute value describes the size of the error.

When multiple independent formulations are evaluated, report each formulation’s errors as well as summaries. The mean signed error describes average direction, but positive and negative errors can cancel. The mean absolute percentage error (MAPE) summarizes the average size of errors without that cancellation:

\overline{\%PE} = \frac{1}{m}\sum_{j=1}^{m}\%PE_j, \qquad \mathrm{MAPE} = \frac{1}{m}\sum_{j=1}^{m}|\%PE_j|

Here, \(m\) is the number of independently evaluated formulations or datasets. Report these summaries separately for AUC and Cmax. A small mean signed error alone is not evidence that predictions are accurate: large overpredictions and underpredictions can average to nearly zero.

Worked Predictions

Worked Example: External Prediction for One Formulation

In a hypothetical evaluation, an IVIVC was locked before testing a new formulation. The model predicts AUC of 494 mg·h/L and Cmax of 13.2 mg/L. The independent study observes AUC of 520 mg·h/L and Cmax of 12.0 mg/L. Calculate the signed percentage prediction errors.

For AUC:

\%PE_{\mathrm{AUC}} = \frac{520-494}{520}\times100 = \frac{26}{520}\times100 = 5.00\%

The positive value indicates underprediction: predicted AUC is 5.00% below the observed AUC of 520 mg·h/L.

For Cmax:

\%PE_{\mathrm{Cmax}} = \frac{12.0-13.2}{12.0}\times100 = \frac{-1.2}{12.0}\times100 = -10.0\%

The negative value indicates overprediction: predicted Cmax is 10.0% above the observed Cmax of 12.0 mg/L. These two results should be assessed separately; a modest AUC error does not erase the larger Cmax error.

Worked Example: Summarizing Errors Across Independent Formulations

A hypothetical locked IVIVC is tested on three independent formulations. The endpoint values below are predicted and observed values, respectively. AUC is in mg·h/L and Cmax is in mg/L.

Formulation	Predicted AUC	Observed AUC	Predicted Cmax	Observed Cmax
A	570	600	14.1	15.0
B	528	480	9.9	11.0
C	665	700	20.7	18.0

For AUC, the formulation-specific errors are:

\%PE_{\mathrm{A,AUC}}=\frac{600-570}{600}\times100=5.00\%, \quad \%PE_{\mathrm{B,AUC}}=\frac{480-528}{480}\times100=-10.0\%, \quad \%PE_{\mathrm{C,AUC}}=\frac{700-665}{700}\times100=5.00\%.

The signed mean is \((5.00-10.0+5.00)/3=0.00\%\). That zero hides nonzero errors. The MAPE is \((5.00+10.0+5.00)/3=6.67\%\).

For Cmax:

\%PE_{\mathrm{A,Cmax}}=\frac{15.0-14.1}{15.0}\times100=6.00\%, \quad \%PE_{\mathrm{B,Cmax}}=\frac{11.0-9.9}{11.0}\times100=10.0\%, \quad \%PE_{\mathrm{C,Cmax}}=\frac{18.0-20.7}{18.0}\times100=-15.0\%.

The signed mean is \((6.00+10.0-15.0)/3=0.333\%\), whereas the MAPE is \((6.00+10.0+15.0)/3=10.3\%\). Formulation C has a 15.0% Cmax error despite the near-zero signed mean. A useful report therefore includes endpoint-specific summaries and the individual errors, not just one average.

Why the Validation Data Must Not Tune the Model

External validation is a test of a prediction made by a fixed model. If an observed external endpoint is used to change the model and the revised prediction is then compared with that same endpoint, the comparison no longer measures independent predictive performance. The revised model has been informed by the answer.

Worked Example: Refitting Can Make Error Look Smaller

Suppose a locked model predicts an external formulation’s AUC as 450 mg·h/L, while its independent observed AUC is 500 mg·h/L. Its external prediction error is:

\%PE = \frac{500-450}{500}\times100 = 10.0\%.

An analyst then uses this observed result to revise the model so that it predicts 490 mg·h/L for the same formulation. The error recalculated after this revision is \((500-490)/500\times100=2.00\%\). That 2.00% is not an external validation result: the observed 500 mg·h/L helped produce the revised prediction. The valid external result for the original locked model remains 10.0%.

For the revised model to receive an external test, it must be evaluated on another independent formulation or dataset whose outcomes have not informed that revision. The same principle applies if the analyst adjusts a time shift, changes the model form, or selects a subset of observations after seeing the external errors.

Justification for the Error Formula

The observed value provides a direct reference for expressing prediction error as a fraction of what occurred. Rearranging the definition shows precisely what a reported percentage means:

\%PE = 100\left(1-\frac{Y_{\text{pred}}}{Y_{\text{obs}}}\right) \quad\Longrightarrow\quad Y_{\text{pred}} = Y_{\text{obs}}\left(1-\frac{\%PE}{100}\right).

Thus, if \(\%PE=5\%\), predicted exposure is \(0.95Y_{\text{obs}}\); if \(\%PE=-10\%\), it is \(1.10Y_{\text{obs}}\). This also explains why the sign convention must be stated: reversing the numerator reverses the interpretation. The calculation requires a meaningful, nonzero observed value and comparable units. It is not appropriate to divide by zero, and a very small observed value can make a percentage error unstable or misleading.

Design, Diagnostics, and Interpretation

A persuasive external validation begins before the external outcomes are examined. Specify the model, the formulation or dataset to be tested, the prediction procedure, the endpoints, and the rules for comparing values. Predictions should be generated from inputs available to the IVIVC, such as the independent formulation’s dissolution data, without using its observed in vivo AUC or Cmax to adjust the prediction.

Check that the test is scientifically interpretable. The predicted and observed endpoints must use compatible definitions, units, dose conditions, and analysis rules. Differences in study design, population, sampling, or measurement can affect the comparison; an apparent model failure may reflect a mismatch in the prediction target, while a seemingly good result from a poorly matched test may not establish useful prediction. These factors should be described rather than silently corrected after seeing the results.

Several commonly used regulatory benchmarks for IVIVC predictability use percentage prediction error near 10% as a key reference, with stricter attention to individual errors in addition to averages. The specific acceptance decision depends on the applicable regulatory guidance, product context, and validation design; a benchmark is not a universal guarantee of adequacy. In particular, do not convert a threshold into a claim that the model will predict all future formulations within that percentage.

Useful checks include:

Confirm the external formulation or dataset was withheld from model construction, selection, and tuning.
Check that predicted and observed AUC and Cmax refer to compatible doses, units, and endpoint definitions.
Inspect signed errors and absolute errors by formulation and endpoint; do not rely on a signed average alone.
Look for a pattern of consistent underprediction or overprediction, which may indicate systematic bias.
Investigate whether the test formulation lies far outside the release-rate or dissolution conditions represented during model development.

A large error does not identify its own cause. It may indicate limitations in the IVIVC, an inappropriate prediction target, incompatible study conditions, or an input outside the model’s useful range. Report the discrepancy and investigate it; do not remove an inconvenient observation or revise the model and still call the revised comparison external validation.

How External Validation Differs From Other Assessments

Internal assessment examines performance on development formulations, which can be informative but may be optimistic because those data helped shape the model. External validation uses independent outcomes and therefore gives stronger evidence about prediction beyond the development data. A held-out subset or cross-validation can help assess performance when data are limited, but it is not equivalent to testing a genuinely independent formulation or study. Refitting after evaluation may be appropriate for developing a final model, but performance on the data used for that refit is not an independent test of the revised model.

External validation also has limits. One successful formulation provides evidence for that test, not proof of performance across all products, release rates, doses, or study settings. A small number of external tests can produce unstable summaries, and percentage errors can be sensitive to small observed values. This tutorial reports point prediction errors; it does not quantify uncertainty around those predictions.

Common Mistakes

Using predicted value as the denominator. The defined error uses \(Y_{\text{obs}}\) in the denominator. Changing the denominator changes the quantity and can change its interpretation.
Reporting only signed average error. Opposing errors can cancel. Report the individual errors and an absolute-error summary as well.
Calling a refitted comparison external validation. If the test outcomes influenced the model or prediction procedure, the resulting comparison is no longer an independent test.
Interpreting a small error as universal predictive success. An external result supports performance for the tested conditions; it does not guarantee performance for untested formulations.

Key takeaway: External validation compares prespecified, unadjusted IVIVC predictions with independent observed outcomes. Calculate AUC and Cmax prediction errors separately, state their direction, inspect individual errors as well as summaries, and preserve the independence of the test data.

Check Your Understanding

Answer each question, then check the solutions below.

A model predicts AUC of 760 mg·h/L and the independent study observes 800 mg·h/L. Calculate signed percentage prediction error and interpret its direction.
Two formulations have signed Cmax errors of \(+12\%\) and \(-12\%\). What are the mean signed error and MAPE, and why are both useful?
An analyst changes the IVIVC time shift after inspecting errors on an external dataset. Can the revised model’s errors on that same dataset still be called external validation errors?
Name two checks that help determine whether predicted and observed endpoints are meaningfully comparable.

Solutions. (1) \((800-760)/800\times100=5.00\%\), a 5.00% underprediction relative to observed AUC. (2) The mean signed error is \((12-12)/2=0\%\); the MAPE is \((12+12)/2=12.0\%\). The signed mean shows no net direction, while MAPE reveals the average error size. (3) No. The external outcomes informed the time-shift adjustment, so the revised model must be tested on new independent data. (4) For example, verify compatible units and dose conditions, and check that endpoint definitions and study analysis rules match.