Why Internal Predictability Requires More Than a Good Fit
An IVIVC can describe the formulations used to build it well and still predict a new formulation poorly. A small residual or a close fitted concentration-time profile is evidence about fit to the development data; it is not, by itself, evidence that the relationship predicts pharmacokinetic performance for a formulation that was not used to estimate it. Internal predictability asks a more demanding question: when a formulation is withheld from model development, how closely does the resulting model predict its in vivo performance?
For this assessment, a prediction must be made without using the withheld formulation’s in vivo results to select, tune, or fit the model. The predicted and observed pharmacokinetic metrics must also be comparable: they should refer to the same dose, analyte, population or summary statistic, and study conditions. Common metrics are AUC and \(C_{\max}\). Predictions of these metrics are usually obtained from the concentration-time profile generated by the IVIVC; the comparison here concerns the resulting metric-level errors, not how to construct the underlying IVIVC.
Prediction Errors for AUC and \(C_{\max}\)
For a given formulation and metric, let \(Y_{\text{obs}}\) be the observed value and \(Y_{\text{pred}}\) the IVIVC-predicted value. The signed percentage prediction error (PE) is
A positive PE means the prediction is higher than the observation (overprediction); a negative PE means it is lower (underprediction). The absolute percentage prediction error is \(|\mathrm{PE}|\), which measures the size of the discrepancy without its direction. For \(m\) validated formulations, the mean absolute PE for a particular metric is
Calculate these errors separately for AUC and \(C_{\max}\). A low average for one metric does not establish good prediction of the other. The percentage calculation also requires a meaningful, nonzero observed value; in ordinary IVIVC validation, the observed exposure and peak concentration are positive.
A commonly used benchmark for internal predictability is a mean absolute PE no greater than 10% across the formulations, with no individual formulation’s absolute PE greater than 15%, assessed separately for AUC and \(C_{\max}\). These are useful conventional benchmarks, not a guarantee of performance or a universal rule that replaces scientific and regulatory judgment. Always state which metric, formulations, and validation strategy the summary represents.
Worked Example: Calculating Errors for Three Formulations
Suppose a hypothetical IVIVC is evaluated using three formulations, each withheld in turn. The table gives observed and predicted AUC in mg·h/L and observed and predicted \(C_{\max}\) in mg/L. Assume the observations and predictions are matched summaries on the same dose and study basis.
| Formulation | AUC observed | AUC predicted | \(C_{\max}\) observed | \(C_{\max}\) predicted |
|---|---|---|---|---|
| A | 100 | 108 | 20.0 | 22.0 |
| B | 120 | 111 | 25.0 | 22.5 |
| C | 150 | 165 | 30.0 | 33.0 |
For formulation A, the AUC PE is \(100(108-100)/100=8.00\%\). For B it is \(100(111-120)/120=-7.50\%\); for C it is \(100(165-150)/150=10.0\%\). Therefore, the mean absolute AUC PE is \((8.00+7.50+10.0)/3=8.50\%\). Each absolute AUC PE is at most 15%, and the mean is at most 10%.
For \(C_{\max}\), the errors are \(100(22-20)/20=10.0\%\), \(100(22.5-25)/25=-10.0\%\), and \(100(33-30)/30=10.0\%\). The mean absolute \(C_{\max}\) PE is \((10.0+10.0+10.0)/3=10.0\%\), and no individual absolute error exceeds 15%. The \(C_{\max}\) predictions do not all overpredict: the signed errors are positive, negative, and positive, so there is no single direction shared by every formulation.
By these conventional benchmarks, both metrics meet the stated average and individual-error limits in this hypothetical example. That conclusion describes these three internal validation predictions; it does not show that predictions for formulations outside the tested range will be equally accurate.
Prespecifying a Formulation-Level Validation
The validation strategy should be set before examining validation results. With enough formulations to support model fitting, a direct approach is to withhold one formulation, estimate the IVIVC using the others, and predict the withheld formulation. In leave-one-formulation-out validation, this is repeated until every formulation has served as the holdout once. Each prediction then comes from a model that did not use that formulation’s in vivo data.
A crucial detail is the unit of withholding. Replicate measurements from one formulation are not independent formulations. If some replicates from the formulation appear in model development while others are treated as validation data, the supposedly held-out prediction may benefit from information about the same formulation. Withholding should therefore keep all relevant observations for a formulation together.
Model choices are part of model development. If the analyst tests several dissolution inputs, time transformations, or fitting options and chooses the one with the best holdout errors, those holdout errors have influenced selection. A defensible procedure prespecifies the model-building steps or performs selection within each training set, without consulting the held-out in vivo result. In all cases, the data used to predict the holdout must be clearly identified.
Worked Example: One Leave-One-Formulation-Out Prediction
For illustration, suppose a simplified IVIVC maps a scalar in vitro release measure \(x\), in percentage points, to AUC in mg·h/L using a straight line. Three hypothetical training formulations have \((x,Y_{\text{obs}})\) values \((50,80)\), \((60,90)\), and \((80,110)\). A fourth formulation, with \(x=70\) and observed AUC \(105\) mg·h/L, is withheld.
Fit the line using only the three training formulations. The training means are \(\bar{x}=190/3=63.333\) and \(\bar{Y}=280/3=93.333\). The least-squares slope is
The intercept is \(a=\bar{Y}-b\bar{x}=93.333-1.000(63.333)=30.00\) mg·h/L. Thus the fitted prediction for the withheld formulation is \(Y_{\text{pred}}=30.00+1.000(70)=100.0\) mg·h/L. Its signed PE is \(100(100.0-105)/105=-4.762\%\), so its AUC is underpredicted by 4.762% relative to the observed value. This is one fold’s result, not enough by itself to establish acceptable internal predictability. In a real Level A assessment, the model may predict a profile before AUC is calculated; this simple line only illustrates the holdout logic and error calculation.
Why the Absolute Errors and Validation Design Matter
The mean signed PE can conceal poor predictions because positive and negative errors cancel. For example, two formulations with signed errors of \(+10\%\) and \(-10\%\) have mean signed PE \((10-10)/2=0\%\), even though each prediction is 10% away from its observation. Their mean absolute PE is \((10+10)/2=10\%\), which reflects the typical magnitude. This is why prediction accuracy is summarized using absolute errors, while signed errors remain useful for detecting systematic overprediction or underprediction.
The calculation also explains what an individual PE means: it is the difference between the predicted and observed value divided by the observed value. It is a relative error, not an uncertainty interval for the true pharmacokinetic metric. An error of 10% for AUC is expressed relative to the observed AUC; the underlying AUC values retain units such as mg·h/L. Report the observed and predicted values with units alongside the percentage errors so the scale remains clear.
Worked Example: An Average Can Hide an Individual Failure
Suppose a hypothetical internal assessment produces AUC PEs of \(+4\%\), \(-4\%\), and \(+16\%\). The signed mean is \((4-4+16)/3=5.333\%\), while the mean absolute PE is \((4+4+16)/3=8.000\%\). The average absolute error is below 10%, but the third formulation has an absolute PE of 16%, exceeding the conventional 15% individual benchmark. Therefore, the result does not meet both stated benchmarks.
For the same formulations, suppose \(C_{\max}\) PEs are \(-2\%\), \(+3\%\), and \(+5\%\). Their mean absolute PE is \((2+3+5)/3=3.333\%\), and each individual absolute error is below 15%. This supports good internal prediction for \(C_{\max}\) by these benchmarks, but it does not erase the individual AUC failure. The two metrics are judged separately.
Diagnostics, Alternatives, and Limitations
A numerical summary should be accompanied by checks that the validation was informative and correctly executed. Confirm that every holdout was excluded from the relevant fitting and selection steps, that predictions and observations use compatible units and summaries, and that each formulation’s calculations use its own observed value as the denominator. Tabulate signed and absolute errors by formulation. A pattern of errors with one sign may indicate directional bias, while a large error for one formulation can be hidden by a modest average.
Also inspect the range of the held-out formulation’s in vitro input relative to the training formulations. A holdout beyond that range tests extrapolation, which may be much more demanding than interpolation. If predictions vary sharply across folds, or a fold cannot support the prespecified model, the internal assessment is unstable or infeasible; it should not be rescued by quietly changing the validation rule after seeing the results. Residual diagnostics for the fitted IVIVC and prediction errors answer related but different questions: acceptable residual behavior does not guarantee acceptable held-out prediction, and a small set of prediction errors cannot replace checking model fit.
Leave-one-formulation-out validation uses the available formulations efficiently and gives every formulation a prediction, but each fitted model can be sensitive to which formulation is omitted, especially when there are few formulations. A prespecified train/validation split is simpler, but its result depends on the particular split and may leave too little data for model fitting. Repeated random splits can show sensitivity to split choice, but they do not create independent evidence when the same formulations recur in many splits. Whichever strategy is chosen, report it and the formulation-level results, rather than only a pooled number.
Internal validation remains limited to the formulations, release range, and study context represented in the development data. It cannot establish performance for a new formulation or setting merely because its internal errors are small. That broader question requires a separate external validation assessment. Nor do prediction errors alone establish that the dissolution method is discriminatory or that a model is suitable for every regulatory or clinical use.
Check Your Understanding
Use the definitions and benchmarks stated above. Solutions follow the questions.
- A formulation has observed AUC \(200\) mg·h/L and predicted AUC \(180\) mg·h/L. Calculate signed PE and state its direction.
- Two formulations have \(C_{\max}\) PEs of \(+12\%\) and \(-12\%\). What are their mean signed PE and mean absolute PE, and why do they differ?
- Three formulations have absolute AUC PEs of 6%, 9%, and 16%. Does this set meet both conventional benchmarks?
- Why is it not a valid formulation-level holdout to put some replicate data from a formulation in training and other replicates from the same formulation in validation?
Solutions. (1) \(\mathrm{PE}=100(180-200)/200=-10.0\%\); the prediction underestimates observed AUC by 10.0%. (2) The mean signed PE is \((12-12)/2=0\%\), and the mean absolute PE is \((12+12)/2=12.0\%\). The signed values cancel, while absolute errors retain their magnitudes. (3) The mean absolute PE is \((6+9+16)/3=10.333\%\), above 10%, and one individual error, 16%, exceeds 15%; it meets neither benchmark. (4) Training replicates reveal information about the same formulation’s characteristics, so the validation observations are not a genuinely unseen formulation and can make prediction appear more successful than it is.