Why Specify the Prediction Before Seeing Outcomes?
An independent validation asks how well a PBPK model predicts data it did not use for calibration. That question is only interpretable if the prediction target is defined before the validation outcomes are examined. Otherwise, an analyst can unintentionally change which concentrations, time points, summary endpoints, or comparison rules count as the evaluation after learning which choices make the model look better.
Prespecification is not just writing down a list of model outputs. It defines the quantity the validation is meant to assess and the rule by which a model prediction corresponds to an observed result. The earlier material on comparing concentration-time profiles and summary endpoints distinguishes those forms of evaluation; here the focus is deciding in advance which form will be used and how it will be operationalized.
A target should say what is predicted, for whom, and under which conditions. For example, “concentration at 4 hours” is incomplete unless it identifies the analyte, concentration matrix, dose and route, population, and whether the prediction is for a typical individual or a population summary. “AUC” is also incomplete unless the integration interval and calculation convention are specified. The target may be a profile at selected times, one or more summary endpoints, or both, but each should have a stated role.
The model, parameter set, population assumptions, and intended validation domain should already be defined for an independent evaluation. Prespecification makes the output request and the comparison procedure explicit; it does not authorize recalibration or changing model inputs in response to validation observations. If a target cannot be computed under the stated model or the data do not support the stated comparison, that limitation should be reported rather than silently resolved by redefining the target after unblinding.
What a Prediction Specification Needs to Fix
The essential principle is that the specification must remove meaningful discretion from the analysis after outcomes are available. This does not require anticipating every unforeseen circumstance. It does require identifying foreseeable choices that could change the quantity being evaluated. A practical specification commonly records the following:
- Prediction context: the population, dose, route, formulation, regimen, covariates, and other conditions defining the simulation. State which conditions are fixed and which vary across validation participants or groups.
- Output identity: the analyte, concentration matrix, and units. If parent drug and metabolite, or total and unbound concentrations, are possible outputs, identify which are included and which are primary.
- Prediction level: whether the target is an individual prediction, a typical-subject prediction, or a population summary such as a median. These are different quantities; one should not be substituted for another because it agrees better with the observations.
- Times and endpoints: list the target time points and define each summary endpoint, including its time interval and calculation method. State which are primary and which, if any, are secondary or exploratory.
- Matching and summarization: say how a model value will be obtained for an observed sampling time, how observations will be paired with predictions, and how the specified results will be summarized across participants or studies.
- Data contingencies: identify foreseeable cases that affect whether a planned comparison can be made, and state how they will be handled or reported. Do not select a replacement endpoint after seeing which planned comparison is inconvenient.
These choices should be scientifically motivated by the prediction question and the intended use, not by which outputs appear favorable. Prespecification does not require that every possible model output be hidden from view forever. It requires distinguishing planned validation results from exploratory analyses and not presenting a result selected after examining outcomes as though it had been the original target.
A Short Justification: Changing the Target Changes the Question
Let \(Y_j\) denote the observed quantity for candidate endpoint \(j\), and let \(\widehat{Y}_j\) be the corresponding prediction. Before outcomes are examined, an analyst selects one endpoint \(j_0\). The validation then concerns the relationship between \(Y_{j_0}\) and \(\widehat{Y}_{j_0}\). If the analyst instead chooses an endpoint after seeing all observed values, the selected endpoint is a function of the outcomes. The resulting evaluation concerns a data-selected target, not the originally intended fixed target.
The selection effect is visible even in a simple illustrative comparison based on absolute difference. If two candidate endpoints are available and the analyst reports whichever has the smaller observed-predicted difference, then
Thus, selecting the endpoint with the closer agreement cannot produce a larger difference than either candidate endpoint considered alone. The apparent agreement is partly a consequence of the selection rule. Prespecifying the endpoint prevents this particular outcome-driven choice. This argument does not imply that every post hoc analysis is useless; it means that such analyses must be identified as exploratory and interpreted as answering a different, selected question.
Worked Examples
Worked Example: Defining a Concentration-Time Profile Target
A hypothetical validation is intended to assess predictions of plasma parent-drug concentrations after a single oral dose. Before any validation concentrations are examined, the team specifies the target as the population median concentration at 1, 4, and 8 hours, in mg/L, for the stated dose and population. It also specifies that each observed concentration will be paired with the prediction at the same nominal time, and that results will be reported separately at each time rather than reduced to a single profile summary.
The model produces prespecified predictions of 12.0 mg/L at 1 hour, 8.00 mg/L at 4 hours, and 3.00 mg/L at 8 hours. Suppose the validation results are later revealed to be 10.8, 8.60, and 3.60 mg/L at those times. The signed observed-minus-predicted differences are calculated separately:
These calculations apply the stated pairing rule and retain the three specified time-specific comparisons. They do not turn the profile into a single endpoint, and they do not establish whether the predictions meet an acceptance criterion. Any such criterion would need to be defined separately. The example assumes the observed samples correspond to the specified nominal times and that the intended population summary is meaningful for the validation data.
Worked Example: Prespecifying an AUC Endpoint
In another hypothetical evaluation, the primary endpoint is the area under the plasma concentration-time curve from 0 to 12 hours, in mg·h/L. The specification states that AUC will be calculated by the linear trapezoidal rule using concentrations at 0, 2, 6, and 12 hours. The predicted concentrations at these times are 0, 9.00, 6.00, and 2.00 mg/L.
For adjacent times \(t_i\) and \(t_{i+1}\), the linear trapezoidal area is the time interval multiplied by the average of its endpoint concentrations:
Substituting the predicted values gives:
Suppose the observed concentrations, revealed later at the same times, are 0, 8.00, 5.00, and 2.50 mg/L. Applying the same prespecified rule yields:
The endpoint comparison is therefore between 63.0 mg·h/L predicted and 56.5 mg·h/L observed under the specified calculation. The calculation assumes concentrations are available at the stated interval endpoints and uses the same time interval and trapezoidal convention for both. If observations are not available at a required time, the handling rule must be prespecified; choosing a different integration interval only after seeing the data would change the endpoint.
Worked Example: Fixing the Time-Matching Rule
A hypothetical protocol defines a comparison at the actual sample time, using log-linear interpolation of the model prediction between adjacent positive concentrations. At 5 hours the predicted concentration is 4.00 mg/L, and at 7 hours it is 2.00 mg/L. A validation sample is collected at 5.8 hours. The prespecified interpolation assumes the predicted concentration changes exponentially between those model output times. For positive concentrations, the interpolated value is:
Substitute \(t_1=5\) h, \(t_2=7\) h, \(C_1=4.00\) mg/L, \(C_2=2.00\) mg/L, and \(t=5.8\) h:
If the subsequently observed concentration is 3.30 mg/L, the comparison uses 3.031 mg/L as the prediction, because the rule was to evaluate at the actual sample time. A different rule—comparison at the nominal 6-hour time—would give \(4.00(0.500)^{(6-5)/(7-5)}\approx2.828\) mg/L. These are different matched predictions, so the choice must be made in advance. The log-linear rule requires positive endpoint concentrations; the specification should state what to do if that condition fails rather than applying the rule selectively.
Choosing Among Common Approaches
No single prediction target is universally appropriate. The scientifically relevant question determines whether profiles, summary endpoints, or both should be prespecified. The important requirement is consistency: use the same defined target and matching procedure for the predictions and observations, and label additional analyses according to their status.
| Approach | What it evaluates | Choice to prespecify |
|---|---|---|
| Concentrations at selected times | Agreement at particular parts of the concentration-time course | Analyte, population summary, time points, and time-matching rule |
| Summary endpoint such as AUC over a fixed interval | A defined integrated or otherwise summarized feature of the profile | Endpoint definition, interval, calculation convention, and input times |
| Profile plus summary endpoint | Both time-specific behavior and a specified overall summary | Which output is primary, which are supplementary, and how each is calculated |
Selected-time comparisons retain timing information but can miss behavior between the selected times. AUC summarizes exposure over an interval but can conceal compensating differences at different parts of the profile. Reporting both can be useful when both questions matter, but adding many unplanned endpoints increases opportunities to emphasize whichever result appears most favorable. A population summary and an individual-level comparison also answer different questions; the specification should not switch between them after seeing the validation results.
Common Pitfalls and Practical Checks
- Choosing the “best-looking” endpoint after unblinding. This selects the evaluation target based on the result. Keep the planned target as the primary analysis; label any outcome-informed alternatives as exploratory.
- Leaving “concentration at a time” underspecified. A concentration value depends on the analyte, matrix, population summary, conditions, units, and time-matching rule. State these explicitly.
- Changing the time grid or integration interval. A new grid or interval can change the quantity being compared. Apply the predeclared rule, and report when it cannot be applied as written.
- Substituting a different prediction level. A population median, a typical-subject prediction, and individual predictions are not interchangeable. Specify the intended level and use it consistently.
- Confusing a comparison rule with a pass/fail threshold. A rule defines which prediction is compared with which observation and how results are summarized. An acceptance criterion decides whether performance is adequate. They are distinct decisions and should not be conflated.
- Using a data-dependent fallback without disclosure. If a planned output is unavailable or a specified calculation condition fails, document the issue and the resulting limitation. A fallback chosen because it improves agreement is not a neutral implementation detail.
Before outcomes are examined, check that another analyst could identify the target, produce the requested model outputs, apply the matching rule, and calculate each planned endpoint without having to make a result-dependent choice. If not, the specification still leaves room for the target to drift. The goal is not to eliminate scientific judgment, but to exercise that judgment before the validation result can influence it.
Check Your Understanding
Answer each question using the definitions and rules in this tutorial. Suggested solutions follow.
- A validation plan states “compare predicted and observed AUC.” Name two details that must be added to define the endpoint.
- Predicted concentrations at two candidate times have observed-predicted absolute differences of 1.0 and 0.2 mg/L. Why is choosing the second time only after seeing these differences a problem?
- A model predicts a median concentration, but the analyst compares it with each participant’s observed concentration. What prediction-level issue should be resolved in the specification?
- For a specified profile comparison, observations are collected at 5.8 hours while model outputs are available at 5 and 7 hours. What must be fixed before outcomes are examined?
Solutions. (1) Specify the integration interval and the calculation convention, such as linear trapezoidal integration; also identify the analyte, matrix, and units. (2) The endpoint was selected using the outcomes, so the reported comparison is for a data-selected target and will favor the time with closer agreement. (3) A population median and participant-level predictions are different quantities; define whether the target is a population summary or individual predictions and make the observed comparison consistent with that target. (4) Prespecify whether to compare at nominal time or actual sample time and, if interpolation is used, define the interpolation method and its conditions.