Why Validation Data Must Match the Prediction
A QSP model may make several distinct predictions: for example, the mean change in a biomarker in one population, the fraction of patients reaching a clinical response under a specified regimen, or the time course of a tissue concentration. A dataset that is relevant to one of these predictions may be a poor test of another. Selecting validation observations therefore begins with the prediction claim, not with whichever dataset is easiest to obtain.
The intended-use claim specifies what the model is being asked to predict. To test that claim directly, the observations should represent the same target population, intervention, endpoint, measurement method, and time horizon. If one or more of these differ, the comparison may still be useful, but it tests a modified claim or requires additional assumptions. Naming those differences is essential: otherwise, a mismatch in data collection can be mistaken for a model failure—or a weak test can be mistaken for strong validation.
Translate the Claim Into Selection Criteria
Before looking at candidate observations, write the prediction in operational terms. State who or what the prediction concerns; what treatment, dose, schedule, and relevant co-interventions apply; which outcome is predicted; how that outcome is measured; and when it is assessed. Specify the prediction’s form as well: an individual trajectory, a population mean, a response probability, or another summary. A dataset must have a sampling unit and enough context to connect its observations to that target.
A useful representation is to define the prediction target by its conditioning features. Let \(Y\) denote the outcome under a specified measurement method, \(P\) the target population, \(A\) the intervention, \(T\) the time horizon, and \(E\) the endpoint definition. Then the claim concerns a distribution or summary such as:
Here \(M\) identifies the measurement method, and \(\mathcal{L}\) denotes the outcome distribution. A claim about a mean, for example, concerns \(E[Y\mid P,A,E,T,M]\); a claim about a response proportion concerns \(P(Y \text{ meets the response rule}\mid P,A,E,T,M)\). This notation makes clear that an outcome value alone is not enough: who received what, how the outcome was defined and measured, and when it was recorded are part of the target.
The selection question is whether candidate observations estimate the same target, or whether they estimate a related target that can be connected only through an explicit mapping or assumption. Population matching should consider characteristics relevant to the model’s intended use, including disease state, baseline status, age range, and effect modifiers represented in the model. Intervention matching includes dose, route, schedule, duration, adherence where available, and relevant concomitant treatments. Endpoint matching requires the same construct and response rule, not merely a similarly named variable. Measurement matching includes specimen, assay, units, and processing. Time matching requires a common origin and assessment window, not just a similar calendar visit label.
Why Alignment Is a Logical Requirement
Suppose a model predicts a population outcome under a target condition \(C=(P,A,E,T,M)\), while candidate data are collected under a different condition \(C'\). A comparison between the model prediction and the observed data then compares \(\mathcal{L}(Y\mid C)\) with \(\mathcal{L}(Y\mid C')\), not two estimates of one target. A difference could arise from model error, from the changed population or treatment, or from how the endpoint was measured. Without a justified relationship between \(C\) and \(C'\), the comparison cannot isolate those explanations.
This is why directness is not a vague judgment of whether a dataset seems relevant. It is a claim about correspondence. The closer the match on the dimensions that define the prediction, the fewer extra assumptions are needed to interpret agreement or disagreement. When a difference is unavoidable, record it and state what additional evidence or mapping would be required to bridge it.
This argument does not mean that a direct match guarantees a decisive validation result. Samples may be small, measurements noisy, or the observed population unrepresentative. It means that the comparison has the right target; precision, uncertainty, and the strength of evidence still need separate assessment.
Worked Examples of Data Selection
Worked Example: A Tumor-Response Prediction at Week 8
A hypothetical model claim is: among adults with a specified biomarker-positive tumor and baseline measurable disease, predict the distribution of percentage change in target-lesion sum from baseline after one named drug regimen, assessed at week 8. The endpoint is based on the same imaging procedure and lesion rules used to define the prediction.
Candidate A contains 36 adults with the stated disease and biomarker status, treated on the stated regimen. Target-lesion sums were assessed at baseline and at day 56 using the specified imaging and lesion rules. Candidate B contains 80 adults with a different tumor type, treated with the same drug at the same nominal dose, with a week-8 imaging outcome. Candidate C contains 30 adults meeting the population criteria, treated on the stated regimen, with tumor volume measured by a different imaging protocol at day 84.
Candidate A matches the population, intervention, endpoint, measurement method, and time horizon. It is the direct candidate for this claim, assuming the recorded observations and protocol details are reliable. Candidate B has an intervention and nominal time match, but its population differs; the data directly concern another disease population. Candidate C matches the population and regimen, but differs in measurement protocol and timing. It cannot directly test the specified week-8 imaging prediction without a justified measurement and time mapping. Neither B nor C becomes a direct test merely because it has a larger sample or a familiar endpoint label.
If the model predicts a mean percentage change of \(-18\%\) at week 8, the selected observations should be summarized using that same outcome definition and time point. For instance, if the hypothetical observed mean in Candidate A is \(-15\%\), the raw difference is \( -15\%-(-18\%)=3 \) percentage points. This arithmetic identifies a discrepancy in the matched outcome; it does not by itself establish whether the model is adequate, because uncertainty and a prespecified criterion would also matter.
Worked Example: Matching an Assay and Its Units
A hypothetical QSP prediction concerns the plasma concentration of a cytokine at 24 hours after a fixed infusion, measured in pg/mL using assay M. The claim concerns adults with a defined inflammatory condition receiving the stated infusion protocol.
Dataset X contains plasma samples at 24 hours from 22 adults meeting the population definition and receiving the specified protocol. The same assay M was used, and concentrations are reported in pg/mL. Dataset Y contains samples at 24 hours from 40 adults receiving the protocol, but measures cytokine concentration in whole blood using assay N. Dataset Z contains plasma samples measured by assay M in the right units, but collected at 48 hours after a different infusion duration.
Dataset X is aligned on population, intervention, endpoint, method, and time, so it is the direct selection. Dataset Y does not directly measure the claimed plasma endpoint with the specified assay; a whole-blood concentration is not interchangeable with plasma concentration simply because both are reported as concentrations. Dataset Z changes both the intervention exposure history and assessment time, so it tests a different condition.
Suppose the model’s hypothetical predicted mean is \(12.0\) pg/mL and Dataset X has an observed mean of \(13.4\) pg/mL. The observed-minus-predicted difference is:
That difference is interpretable for the specified assay and time point. If a conversion between assay M and assay N has not been established, reporting Dataset Y’s mean beside the prediction as though both measured the same target would conceal an unresolved measurement mismatch.
Worked Example: Selecting Data for a Population Response Claim
A hypothetical claim predicts the proportion of adults with a defined disease who achieve at least a 30% reduction in a specified biomarker from baseline by week 12 under a stated regimen. The prediction is a population response probability, not an individual-level trajectory.
Dataset R includes 20 adults from the target population, treated with the stated regimen. Each person has a baseline and week-12 biomarker value measured using the specified assay. Six meet the response rule. Dataset S includes 100 patients with the same diagnosis and regimen, but reports only the mean biomarker change; individual responses and the response threshold cannot be reconstructed. Dataset T includes 50 adults with individual biomarker measurements and the same response rule, but assesses them at week 8.
Dataset R directly supports comparison with the response-probability claim. Its observed response proportion is:
Thus 30.0% of the 20 observed participants met the stated response rule by week 12. If the hypothetical model prediction is \(0.35\), the observed proportion is 5.00 percentage points lower. This is a descriptive comparison, not proof that the model’s population probability is wrong: sampling uncertainty and the representativeness of the 20 participants remain relevant.
Dataset S cannot directly test this claim because a mean change does not determine the fraction crossing a threshold. Dataset T contains individual values but measures response at week 8, so it addresses a different time horizon. A model prediction at week 8 could be tested with T, but it would not directly test the stated week-12 claim.
Selection Workflow and Practical Diagnostics
A transparent selection process makes the reasoning reviewable and helps prevent an attractive but mismatched dataset from quietly redefining the claim. Record the claim first, then assess candidate data dimension by dimension. Do not combine distinct mismatches into a single label such as “similar population”; specify what matches, what differs, and what consequence follows.
Specify the population, intervention, endpoint and response rule, measurement method, time origin and horizon, and predicted summary or distribution.
For each dataset, identify the sampling unit, population characteristics, treatment details, endpoint definition, assay or measurement process, and assessment schedule.
Mark each feature as matched, mismatched, or unknown. Treat unknown metadata as an unresolved limitation, not as evidence of a match.
Select data that best correspond to the stated target. For any mismatch, state whether a justified mapping exists and how the resulting comparison changes the claim.
Several practical diagnostics can reveal a selection problem before analysis. Check whether the recorded time is relative to the same event as the model’s time origin; whether “response” uses the same threshold and denominator; whether units, specimen, assay, and censoring rules are documented; and whether the dataset’s population exclusions remove a subgroup central to the intended use. Verify that the intervention description includes dose and schedule rather than only a drug name. If essential metadata are absent, the comparison may remain ambiguous even when the headline descriptors appear to match.
The selection record should also state how the observations were used in model development or decision-making. Data used to estimate parameters are not automatically independent evidence for a predictive claim. Keeping the provenance and use of each candidate visible makes it possible to distinguish a direct comparison from one whose interpretation is limited by prior use; the detailed design of holdout data is a separate issue.
Alternatives, Trade-offs, and Common Mistakes
| Selection approach | What it can support | Main limitation |
|---|---|---|
| Directly matched endpoint and conditions | A focused test of the stated prediction target | May be difficult to find, small, or imprecise |
| Mechanistic proxy or surrogate endpoint | A test of a related model component or an explicitly linked intermediate prediction | Does not directly test the claimed clinical endpoint unless the link is justified |
| Related population, regimen, or time point | Evidence about a neighboring condition or a transportability question | Differences can confound interpretation of model-versus-data discrepancy |
| Broad dataset with incomplete endpoint detail | Potentially useful descriptive context | May not permit calculation of the claimed prediction summary |
A common mistake is to choose the largest available dataset and call it the strongest validation set. The faulty reasoning is that a larger sample must provide a better test. Sample size can improve precision for the quantity measured, but it cannot make the wrong endpoint, population, assay, or time point become the target of the claim. The correction is to establish relevance first, then consider whether the matched observations are sufficiently numerous and informative.
Another mistake is to treat a biomarker related to a clinical outcome as a direct substitute for that outcome. A mechanistic link may make the biomarker scientifically useful, but unless the claim itself concerns that biomarker or a justified mapping connects it to the clinical outcome, the data test a different prediction. Likewise, matching the drug name while ignoring dose, schedule, and background therapy may conceal an intervention mismatch.
Direct selection also has limits. A nominal match does not guarantee that the observed participants represent the target population, that the assay is reliable, or that the sample is informative about the predicted distribution. Selection is not a substitute for later assessment of uncertainty, residual patterns, or acceptance criteria. It establishes what the comparison is about and clarifies which conclusions the evidence can support.
Check Your Understanding
Answer each question by identifying the target and the consequence of any mismatch.
- A model predicts a week-12 response proportion using a stated threshold. A candidate dataset reports only the mean change at week 12. Is it a direct test? Why?
- A model predicts plasma concentration at 24 hours using assay A. Candidate observations measure plasma at 24 hours using assay B, with no established relationship between the assays. What is the selection limitation?
- A model predicts a biomarker mean in a defined population under a fixed dosing schedule. Candidate observations match the population and endpoint but use a different schedule. What does the candidate data directly concern?
- For a hypothetical target population and regimen, 9 of 30 observed participants meet the model’s stated response rule at the stated time. Calculate the observed response proportion and identify what further interpretation is needed before judging model adequacy.
- Why is a large sample with a related surrogate endpoint not automatically a stronger direct validation dataset than a smaller, matched dataset?
Solutions. (1) No. A mean change does not determine the proportion meeting a threshold, so the claimed response probability cannot be calculated from those aggregate data. (2) The measurement method differs; without a justified assay mapping, the observations do not directly estimate the assay-A target. (3) They concern the outcome under the different schedule. The intervention mismatch means any difference cannot be attributed solely to model performance for the stated schedule. (4) \(\hat{p}=9/30=0.300\), or 30.0%. This is an observed proportion; sampling uncertainty and whether the participants represent the target population must be considered before judging adequacy. (5) Sample size affects precision for what was measured, while endpoint mismatch means the data do not directly test the claimed outcome. Directness depends on matching the target, not on sample size alone.