Why the Holdout Unit Matters
A validation prediction is informative only if the observation being predicted was not used, directly or indirectly, to choose or calibrate the model. In QSP data, the unit of observation is often not the unit that must be held out: one participant may contribute many time points, one study may include many participants, and one intervention may appear across several studies. Splitting individual rows at random can therefore place related information on both sides of the split.
The appropriate split follows the intended-use claim. A claim about predictions for new participants in a familiar study may call for holding out participants. A claim about predictions in a new study calls for holding out studies. A claim about a new dose or intervention may require holding out intervention groups. The earlier material on selecting validation data for a prediction claim establishes the need to align the validation observations with that claim; here the emphasis is on implementing that choice without leakage.
A useful distinction is between procedural independence and statistical independence. Procedural independence means that the holdout outcomes and information derived from them have not informed model selection, calibration, or analysis decisions. Statistical independence is a property of the data-generating process and may fail when observations share participants, sites, assays, or other sources of dependence. A clean workflow cannot make correlated observations independent; it must split at a level that addresses the dependence relevant to the intended claim.
Choose the Split Level Before Fitting
The split unit should be at least as high as the unit whose information must be new at prediction time. A subject-level split keeps all records from each participant together. A study-level split keeps all participants and records from a study together. An intervention-level split keeps all records for a designated treatment, dose, or regimen together when the claim concerns an intervention not used in model development.
| Prediction claim | Possible holdout unit | Leakage risk if split too low |
|---|---|---|
| New participant in a represented study setting | Participant | Participant-specific responses can appear in both sets |
| New study or site | Study or site | Study-specific assay, protocol, or population patterns can cross the split |
| New dose, regimen, or intervention | Intervention group | Other records from that intervention can reveal its response pattern |
Sometimes more than one grouping matters. If participants receive several interventions, assigning intervention records separately may leave the same participant in calibration and validation. If a study tests several interventions, holding out only one intervention may still leave study-specific information on both sides. Before splitting, map the identifiers and ask which entities are shared across records. When several grouping constraints apply, construct splits that keep the relevant linked groups together, or state clearly which generalization claim the design can support.
Worked Example: Repeated Measurements and Subject Leakage
Worked Example: Repeated Measurements and Subject Leakage
Suppose a hypothetical QSP dataset contains 60 participants, each with five longitudinal biomarker observations. The intended claim is prediction for participants not used in calibration. Consider assigning each of the 300 rows independently, with probability 0.8 to calibration and 0.2 to validation.
For any one participant, all five rows fall in calibration with probability \(0.8^5=0.32768\). All five fall in validation with probability \(0.2^5=0.00032\). The participant appears in both sets in every other case, so
By linearity of expectation, the expected number of participants represented in both sets is
or about 40.3 participants. This expectation does not say that exactly 40.3 participants will be split; the realized count must be a whole number. It shows that row-wise randomization would commonly put the same participant's measurements into both partitions. Such a split does not test prediction for entirely new participants. A participant-level split instead assigns all five observations from each participant to one partition, which matches the stated claim more closely.
The calculation assumes five rows per participant and independent row assignments with the stated probabilities. Unequal numbers of observations or a different assignment scheme change the probability, but not the basic warning: repeated records should not be treated as unrelated rows when the intended validation unit is the participant.
Leakage Can Occur Before the Model Is Fit
A holdout partition is not protected merely because its outcomes are absent from the final parameter-fitting step. Any operation that learns from the full dataset can transfer holdout information into the model-development process. Common examples include estimating scaling constants, imputing missing values, selecting features, choosing transformations, defining subgroups, or setting decision thresholds using both partitions. The correct sequence is to split first, fit these data-dependent operations using calibration data only, and then apply the fixed operations to validation data.
Specify what must be new at prediction time and identify participant, study, intervention, and other relevant identifiers.
Assign complete groups to calibration or validation, preserve the assignment rule and random seed when applicable, and check for duplicated or linked records across partitions.
Fit preprocessing steps, select model structure, estimate parameters, and make analysis choices without using validation outcomes.
Record the model and prediction procedure, generate validation predictions, and compare them with the reserved observations under the prespecified evaluation plan.
Worked Example: Imputation That Leaks Validation Information
In a hypothetical calibration set, a biomarker covariate has observed values 2, 4, 6, and 8 units, and one validation record has a missing value. Suppose median imputation is part of a prediction pipeline.
The calibration median is the midpoint of 4 and 6:
The missing validation covariate should therefore be imputed as 5 units using the calibration-derived rule. If the validation set also contains the observed value 100 units and the median is incorrectly computed after pooling all five records, the pooled median is 6 units. Using 6 for the missing validation record allows a validation observation to alter preprocessing. Even if the change is small here, the operation is not independent of the holdout data. The correction is to estimate the imputation rule using calibration records and apply that fixed rule to validation records. The same principle applies to scaling, feature screening, and other learned transformations.
Worked Example: Holding Out Studies for a New-Study Claim
Worked Example: Holding Out Studies for a New-Study Claim
A hypothetical QSP dataset contains four studies with 40, 35, 30, and 25 participants. The intended claim is prediction in a study not used during model development. A row-wise or participant-wise split across all four studies would leave each study represented on both sides, so it would not directly assess performance in a wholly new study.
A simple design holds out the study with 25 participants and uses the other three studies for development. All participants, time points, assay records, and outcomes from the held-out study remain reserved. Any study-level preprocessing or model choices are derived from the three calibration studies alone. The validation set then contains one complete study, or 25 participants, and the design directly tests transfer to a new study under the observed study conditions.
This design has an important limitation: its evaluation reflects one held-out study, so it may be sensitive to that study's particular population, protocol, or assay. Holding out each study in turn can provide several study-level evaluations, but if results from those evaluations are used to choose the model, they are no longer an untouched final holdout. The intended use and the role of each partition must therefore be declared.
Alternatives, Diagnostics, and Limitations
A single holdout is straightforward and preserves a clear final evaluation, but it may use data inefficiently and its result can depend strongly on which groups were reserved. Grouped cross-validation rotates groups through development and evaluation partitions; it can describe variability across held-out groups and support model development, but repeated use of its results for selection can still overfit the resampling process. A genuinely external dataset or a separately conducted evaluation can provide stronger procedural separation, although differences in population, measurement, or protocol can complicate interpretation. These approaches answer related but distinct questions; none removes the need to choose a split unit aligned with the claim.
Before interpreting validation results, audit the split and workflow. Check that participant, study, and intervention identifiers do not cross partitions where they are required to be held out; inspect duplicate records and derived variables; verify that preprocessing was fitted only on calibration data; and review whether validation results prompted changes to the model or analysis. A validation set repeatedly consulted during development becomes development information, even if its original rows remain labelled “validation.”
Leakage is not always obvious. A derived covariate may encode a post-outcome measurement, a sample identifier may reveal study membership, or a summary statistic may have been calculated using all participants. Conversely, sharing a known baseline covariate across partitions is not automatically leakage if that covariate would be available for the intended prediction and the split unit is appropriate. The relevant question is whether the information would be available at the prediction time and whether its use violates the intended separation.
A split also cannot establish that the model is scientifically adequate, nor can it guarantee performance in populations or interventions absent from the data. Small numbers of independent studies or intervention groups make group-level validation uncertain even when the data are partitioned correctly. Design, provenance, and the scope of the claim must accompany any numerical performance summary.
Worked Example: Holding Out Interventions Without Overclaiming
A hypothetical dataset contains 12 intervention groups, each with 20 participant records. The claim is prediction for interventions not used in model development. The analyst assigns eight complete intervention groups to calibration and four complete groups to validation. Thus 160 records are assigned to calibration and 80 to validation, provided each record belongs to exactly one group.
This partition is aligned with the claim only if the four validation interventions, including their outcomes and derived summaries, are not used to select mechanisms, tune parameters, or revise the analysis. If participants appear under more than one intervention, an intervention-only split may still place a participant's records in both sets. The analyst must then determine whether participant overlap would reveal information unavailable for a new-intervention prediction. If not acceptable, the split must also keep participant-linked records together, or the claim must be narrowed to match what the design actually tests.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A prediction claim concerns new participants from a familiar study. Each participant has six time points. What should generally be kept together, and why?
- For a subject with four rows, each independently assigned to calibration with probability 0.75 and validation with probability 0.25, calculate the probability the subject appears in both sets.
- A researcher standardizes a covariate using the mean and standard deviation computed across calibration and validation records. What is the problem, and what is the correction?
- Why might holding out participants from every study be inadequate for a claim about prediction in a new study?
- After examining holdout performance, an analyst changes the model structure and reports performance on the same holdout set. Is that set still an untouched validation set? Explain.
Solutions. (1) Keep all six rows from each participant in one partition; otherwise participant-specific information can occur in both sets, undermining prediction for new participants. (2) The probability is \(1-0.75^4-0.25^4=1-0.31640625-0.00390625=0.6796875\), or about 0.680. (3) The validation records influence preprocessing. Estimate the mean and standard deviation from calibration records only, then apply those fixed values to validation records. (4) Every study remains represented during development, so study-specific protocols, assays, or population patterns may be shared; hold out complete studies to evaluate a new-study claim. (5) No. The holdout results influenced model selection, so the set has become part of development. A new untouched evaluation set, or an appropriately designed resampling strategy for development followed by a separate final evaluation, is needed for an independent assessment.