Why Missingness and Deviations Matter in Validation
An independent validation study tests predictions against new observations without refitting the model to those observations. Its interpretation depends not only on the model and assay, but also on what was actually administered and measured. A concentration sample that was not collected cannot be treated as a measured concentration, and a dose given at the wrong time may make an observed concentration an imperfect test of the prediction for the planned regimen.
The goal is not to make every participant or sample appear protocol-perfect. It is to document what happened, preserve the predictions made before results were examined, and show how data limitations affect the evidence. Excluding inconvenient observations after seeing their prediction errors can make performance appear better than it is. Conversely, counting a protocol deviation as a model failure without acknowledging that the model was given the wrong regimen can misstate what was tested.
Missingness is distinct from a result reported below the assay’s lower limit of quantification. A below-quantification result is an assay outcome with information about the concentration; a missing sample has no concentration result. Record the reason for each gap where known—for example, a missed visit, a failed sample, a lost specimen, or an assay problem—and do not replace the gap with zero or relabel it as below quantification.
Record What Happened Before Summarizing Performance
A useful validation record links each planned sample to its participant, scheduled time, actual collection time, assay status, and disposition. It also records the planned and actual dose amount, formulation, route, and administration time, alongside other relevant deviations such as prohibited concomitant medication, food conditions, or sample handling problems. Record the source and timing of any correction to the study record; do not silently overwrite the original entry.
For each discrepancy, distinguish the fact from its possible consequence. “Sample collected 42 minutes after the nominal 30-minute time” is a factual record. “This timing difference may affect comparison during the absorption phase” is an interpretation. Whether it does affect the prediction depends on the model’s time course and the prespecified tolerance for collection times.
Keep the prediction file locked as generated before unblinding. If a model prediction was made for a 100 mg dose at 08:00, do not replace it after seeing the concentration with a prediction chosen to fit the observed value. A separate prediction using a documented actual dose or time can be informative, but it must be labelled as a sensitivity analysis unless that use of actual dosing records was prespecified and performed without using concentration results.
List scheduled samples and doses, then identify each missing, mistimed, failed, or altered record. Preserve the original record and document corrections.
Record the reason, time, and potential relevance to the intended comparison. Do not use the size or direction of a prediction error to decide whether a deviation “counts.”
State which comparisons are available, which are not, and how deviations are represented. Report the number of observations contributing to every summary.
Compare the planned primary analysis with clearly labelled sensitivity analyses, explaining what question each analysis answers.
Missing Samples and the Risk of Selection Bias
A concentration-level error can be defined, when both prediction and observation are available, as \(d_i=\ln(\widehat{C}_i/C_i)\), where \(\widehat{C}_i\) is the locked predicted concentration and \(C_i\) is the observed concentration. Positive values indicate overprediction and negative values indicate underprediction. The mean of these errors summarizes log-scale bias among the pairs included; its exponent is the geometric mean prediction-to-observation ratio.
That summary describes the observed pairs. It need not describe all scheduled pairs if the chance of obtaining a usable sample is related to the concentration, the prediction error, or participant characteristics. A sample missed for an unrelated logistical reason may have a different implication from samples systematically unavailable at times when concentrations are high or low. The reason for missingness may be unknown, so the observed data alone may not establish that the available pairs are representative.
A complete-pair analysis is often a clear descriptive summary of what was measured, but it does not automatically solve this selection problem. Imputation is not a neutral fix: a concentration imputed from the PBPK model being validated can make agreement look artificially good. Any imputation or weighting method requires a justified model for missingness and must be specified and reported as an analysis choice, not introduced because the primary summary is disappointing.
Worked Example: A Missed Sample in a Profile Comparison
In a hypothetical validation study, five participants each have a scheduled 2-hour concentration. Four usable pairs are available. Their prediction-to-observation ratios are 1.10, 0.90, 1.20, and 0.80; the fifth sample was not collected. Calculate the geometric mean ratio for the observed pairs and explain what it does—and does not—show.
The observed-pair geometric mean ratio is the fourth root of the product: $$ \exp\left(\frac{1}{4}\sum_{i=1}^{4}\ln(\widehat{C}_i/C_i)\right) =(1.10\times0.90\times1.20\times0.80)^{1/4} =0.9504^{1/4} \approx0.9873. $$
Among the four available pairs, the geometric mean prediction is about 0.987 times the observed concentration, indicating a small average underprediction on this ratio scale. The fifth participant contributes no concentration error because there is no observed concentration. The result is not a five-participant profile summary and does not show that the missing participant would have had a similar error. The report should give the denominator (four of five scheduled samples), the reason for the gap if known, and any prespecified sensitivity analysis.
Protocol Deviations: Keep the Tested Question Clear
A deviation can change the question that a comparison answers. If the planned validation question concerns a specified regimen, an incorrect dose or administration time means the observed concentration was not generated under that regimen. Comparing it directly with the prediction for the planned regimen still documents what happened in the study, but an error in that comparison cannot be attributed solely to model performance under the intended conditions.
A useful reporting strategy is to distinguish the analysis of the prespecified validation target from analyses describing the study as conducted. For example, report the planned-regimen analysis using the locked predictions and prespecified eligibility or tolerance rules; separately describe all participants and deviations. If predictions conditional on actual dosing are also produced, state the inputs and method and label the analysis clearly. Such predictions should be generated without using observed concentrations to tune the model. A post hoc prediction using revised inputs is not interchangeable with the original locked prediction.
Do not invent a tolerance after seeing which participants it would exclude. If timing tolerances or rules for major deviations were not prespecified, show the impact using transparent, defensible groupings and identify the analysis as exploratory. Report counts and reasons for every excluded or unavailable comparison. A result should not disappear merely because it disagrees with the model.
Worked Example: An Administration-Time Deviation
In a hypothetical study, the locked prediction for a concentration at 2 hours after a planned 100 mg dose is 10.0 mg/L. The recorded dose was actually administered at 09:00 rather than the planned 08:00, and the sample was collected at 10:00. The observed concentration is 8.0 mg/L. For illustration, a separate model run using the recorded actual dose time, with no refitting and no use of the concentration result, predicts 8.5 mg/L.
Against the locked planned-regimen prediction, the prediction-to-observation ratio is $$ \frac{10.0\ \text{mg/L}}{8.0\ \text{mg/L}}=1.25. $$ The planned-regimen prediction is 25% above the observed concentration. But the dose-time discrepancy means that this comparison combines model error with a mismatch between planned and actual administration.
The separate actual-time prediction has ratio $$ \frac{8.5\ \text{mg/L}}{8.0\ \text{mg/L}}=1.0625. $$ It is about 6.25% above the observation. This sensitivity result suggests that accounting for recorded dose time changes the comparison, but it does not replace the locked result or prove that the model is correct. The report should state that actual-time input was used, that the model was not refitted, and whether this analysis was prespecified. If it was devised after results were seen, it is exploratory rather than a new primary validation result.
Assess Impact, Diagnose Problems, and Choose Comparisons
Assess impact in ways that reveal what the missingness or deviation could change. For concentration-time comparisons, report the number of usable pairs at each scheduled time and whether actual times differ materially from nominal times. For a summary endpoint such as an exposure measure, state whether the available data support its calculation under the prespecified method; do not silently calculate a different endpoint from a truncated profile. If a planned endpoint cannot be estimated reliably, report it as unavailable rather than filling gaps with unvalidated assumptions.
Useful diagnostics include tabulations of missing samples and deviations by participant, time, reason, and study site; comparisons of observed and missingness patterns against available baseline or dosing information; and plots that distinguish scheduled time from actual collection time. Review whether missing samples cluster around a particular phase of the concentration-time profile or whether deviations affect particular subgroups. These checks can identify patterns that make a complete-pair summary difficult to interpret, but they cannot recover unobserved concentrations.
The main approaches answer different questions. Their use should be specified and their limitations stated.
| Approach | What it summarizes | Main caution |
|---|---|---|
| Available-pair summary | Prediction errors for pairs with usable observations | May not represent all scheduled samples if availability is selective |
| Protocol-concordant summary | Performance in the subset meeting prespecified protocol rules | Can narrow the target population; do not define rules using observed errors |
| As-run or actual-input sensitivity analysis | Performance conditional on documented study conduct or actual dosing inputs | Must not be tuned to concentrations; post hoc versions are exploratory |
| Imputation or model-based adjustment | An estimate under additional assumptions about missingness and the analysis model | Results depend on assumptions and may be circular if the tested model supplies the imputations |
Common warning signs include a rising number of unavailable samples at a specific time, unexplained differences between planned and actual dose records, and a sensitivity analysis that changes the conclusion substantially. Also check whether the same protocol rule is applied to all participants and whether the reported denominator matches the records. When data are sparse, missingness is concentrated, or deviation records are unreliable, conclusions about validation performance should be correspondingly limited.
No single missing-data procedure makes a validation study immune to data limitations. The most defensible report separates observed facts, prespecified analysis decisions, and sensitivity assumptions. It states which target the analysis addresses, how many comparisons were available, and whether missingness or deviations could plausibly alter the conclusion. That discipline preserves the distinction between a model’s predictive performance and the conditions under which the study was conducted.
Worked Example: Sensitivity to Outcome-Related Missingness
Consider a hypothetical illustration in which six scheduled log prediction errors would be 0.00, 0.10, 0.20, 0.30, 0.40, and 0.50 if all were observed. Suppose only the first four are available because the final two samples were missed. Compare the mean error among available samples with the mean across all six values in this illustration.
The available-pair mean is $$ \frac{0.00+0.10+0.20+0.30}{4}=\frac{0.60}{4}=0.150. $$ The mean across all six values is $$ \frac{0.00+0.10+0.20+0.30+0.40+0.50}{6} =\frac{1.50}{6}=0.250. $$
The available-pair mean is 0.100 lower. Exponentiating, the corresponding geometric mean ratios are \(\exp(0.150)\approx1.162\) and \(\exp(0.250)\approx1.284\). Thus, if larger errors were systematically more likely to be missing, the observed-pair summary could understate the overprediction in this constructed example. In a real validation study, the missing errors are unknown; this calculation illustrates a possible impact, not a method for filling in missing results.
Check Your Understanding
Answer each question using the principles above. Brief solutions follow the questions.
- A scheduled concentration has no assay result because the specimen was lost. Should it be recorded as zero or as below quantification? What should be reported?
- A participant receives a dose later than planned. Why might the locked planned-regimen prediction remain important, and what would an actual-time sensitivity analysis need to disclose?
- Four observed prediction-to-observation ratios are 1.00, 1.10, 0.90, and 1.20. Calculate their geometric mean ratio.
- Why can excluding all participants with large observed errors create a misleading validation summary?
- What can a plot of missing samples by nominal time reveal, and what can it not establish?
Solutions. (1) Neither: the result is missing, not zero and not below quantification. Report the scheduled time, missing reason if known, and the number of usable comparisons. (2) The locked prediction preserves the test of the planned regimen and prevents post-result substitution. An actual-time analysis should state the recorded inputs, method, whether it was prespecified, and that concentrations were not used to tune the prediction. (3) The geometric mean is \((1.00\times1.10\times0.90\times1.20)^{1/4}=(1.188)^{1/4}\approx1.044\). (4) Excluding by observed error selects the analyzed data based on the outcome, potentially biasing the reported performance in the model’s favor. (5) The plot can show clustering by time that warrants investigation and limits interpretation; it cannot reveal the unobserved concentrations or prove why samples are missing.