What Does a Random Record Split Test?
A concentration-time dataset often contains many records from each participant, and several participants may share a study protocol, site, or assay. Randomly assigning individual records to calibration and validation sets can therefore put measurements from the same participant—and often the same study—in both sets. A validation record may look unused while sharing important information with the records used to develop or tune the model.
The key question is not whether the records were randomly assigned. It is which units are new at the prediction task being evaluated? A record-level split can assess prediction of additional measurements from participants and studies already represented in calibration. It does not, by itself, establish performance for a new participant or a new study. This distinction matters even when no data have been improperly transferred between sets: the split may simply answer a narrower question than its label suggests.
A concentration record is rarely an independent replicate of a new participant. Records from one person share physiology and other participant-level characteristics; records in one study may share protocol, site, assay, and design features. The earlier material on independence and data leakage provides the relevant distinction: separate rows are not necessarily independent units, and information can cross a split through shared people, studies, or data-dependent choices.
Why Shared Groups Can Make Validation Look Easier
Consider a simplified residual model for participant \(i\) and record \(j\):
Here, \(Y_{ij}\) is the observed log concentration, \(f_{ij}\) is the log concentration predicted by the PBPK model before a participant-specific adjustment, \(b_i\) is a participant-level deviation, and \(\epsilon_{ij}\) is a record-level deviation. For this illustration, assume \(b_i\sim N(0,\tau^2)\), \(\epsilon_{ij}\sim N(0,\sigma^2)\), and that deviations are independent across participants and records, with \(b_i\) independent of \(\epsilon_{ij}\). This is a teaching model for dependence, not a claim that every PBPK residual follows this distribution.
If several records from participant \(i\) are in calibration, the fitted model may estimate \(b_i\) from those records. A randomly held-out record from that participant can then benefit from information about the same person. For a new participant, no such participant-specific information is available. The following derivation quantifies this difference under the stated assumptions.
A Short Derivation of the Shared-Participant Advantage
Suppose \(m\) calibration residuals are available for one participant after subtracting \(f_{ij}\). Their average is \(\bar{R}_i=b_i+\bar{\epsilon}_i\), where \(\bar{\epsilon}_i\) has variance \(\sigma^2/m\). Under the normal model and a zero-mean normal prior for \(b_i\), the conditional variance of \(b_i\) after observing that average is:
The first equality follows because the prior precision is \(1/\tau^2\) and the average supplies precision \(m/\sigma^2\); independent normal precisions add. A prediction for another record from that same participant has uncertainty from the remaining uncertainty in \(b_i\), plus its new record-level error. Its expected squared error is therefore \(\sigma^2+\operatorname{Var}(b_i\mid\bar{R}_i)\). For a new participant predicted without participant-specific calibration records, the expected squared error is \(\sigma^2+\tau^2\). Since \(m>0\) implies \(\tau^2\sigma^2/(\sigma^2+m\tau^2)<\tau^2\), the shared-participant prediction can have lower error.
This result describes a model that uses participant-level information estimated from calibration records. If the PBPK predictions are fully fixed, do not use participant-specific estimates, and are not changed by any calibration choices, sharing participants need not lower the validation metric itself. However, records within a participant remain correlated, the validation set still represents those same participants, and the metric's uncertainty and weighting can differ from an assessment on new participants. A random record split is therefore not automatically invalid; its interpretation must match its design and the analysis that produced the predictions.
Worked Example: How Often Do Participants Appear in Both Sets?
Suppose a hypothetical dataset has 10 participants with 8 concentration records each, for 80 records total. Each record is independently assigned with probability 0.8 to calibration and 0.2 to validation. For any one participant, the probability that all 8 records go to calibration is \(0.8^8=0.1678\). The probability that all 8 go to validation is \(0.2^8=0.00000256\). Thus, the probability that the participant appears in both sets is:
On average, \(10(0.8322)=8.322\) of the 10 participants will contribute records to both sets. This is not a guarantee that exactly 8 participants will be shared; it is the expected number under the stated random-assignment scheme. The calculation shows why a nominally large held-out record set can contain very few genuinely unseen participants.
If the intended claim concerns new participants, this split does not reproduce that prediction setting. A participant-grouped split would keep each person's 8 records together, so held-out participants contribute no records to calibration. If instead the intended task is to predict later concentrations for already observed participants, a record split may be closer to the target—but time ordering and access to earlier measurements must still reflect the intended use.
Worked Example: Quantifying the Shared-Participant Advantage
For a hypothetical residual model, suppose the participant-level standard deviation is \(\tau=0.30\) log-concentration units, the record-level standard deviation is \(\sigma=0.20\) log-concentration units, and \(m=6\) calibration records are available for each participant represented in both sets. Under the model assumptions above, the expected squared error for another record from that participant is:
The numerator of the fraction is \(0.09(0.04)=0.0036\), and its denominator is \(0.04+6(0.09)=0.58\). Thus, the squared error is \(0.04+0.0036/0.58=0.04621\), giving an expected root-mean-squared error of \(\sqrt{0.04621}=0.2150\) log-concentration units.
For a new participant without participant-specific calibration information, the expected squared error is \(0.20^2+0.30^2=0.13\), so the corresponding root-mean-squared error is \(\sqrt{0.13}=0.3606\) log-concentration units. In this illustration, the difference arises because the shared participant's deviation is partly estimated from calibration records. It is not a universal numerical correction to apply to PBPK validation results; it depends on the model, assumptions, and target.
Record Counts Also Determine Who Dominates a Metric
Even when predictions are fixed and no participant-specific fitting occurs, a pooled record-level metric weights participants according to their number of records. For example, a participant sampled much more densely can contribute far more to pooled squared error than a participant with sparse sampling. A random record split tends to preserve this imbalance in both sets. Consequently, a pooled metric can describe performance on the observed record mix while giving little weight to participants with fewer measurements.
Worked Example: Pooled Error Versus Equal Participant Weight
Imagine two hypothetical participants in a validation set. Participant A has 90 records, each with an absolute prediction error of 0.10 concentration units. Participant B has 10 records, each with an absolute error of 1.00 concentration unit. The pooled record-level root-mean-squared error is:
concentration units. If instead the two participants are given equal weight by first calculating each participant's mean squared error, the result is:
concentration units. These are different summaries of the same errors, not competing calculations of one uniquely defined quantity. The pooled result answers a record-weighted question; the equal-participant result answers a participant-weighted question. Neither establishes performance on new participants unless the validation design supports that interpretation.
Choosing and Interpreting the Split Unit
A split should be chosen from the prediction target, not from a preference for a particular allocation algorithm. Randomly assigning records is not equivalent to randomly assigning participants or studies. The units held out determine what kind of generalization the validation can examine.
| Split unit | What is held out? | Interpretation and limitation |
|---|---|---|
| Record | Individual measurements; participants and studies may be shared | Can assess additional records in represented groups. Does not by itself assess new-group performance. |
| Participant | All measurements from selected participants | Assesses predictions for participants not used in calibration, within the studies represented by the design. |
| Study | All measurements from selected studies | Assesses predictions in studies absent from calibration. Study count and study differences limit how precisely this can be assessed. |
For example, if the intended use is prediction for a new participant at a familiar site and under a familiar protocol, a participant-grouped split is more informative than a record split. If the intended use is prediction in a new study, holding out participants while allowing the same study in calibration does not test the study-level change. Where participants are nested within studies, a study-level holdout also keeps its participants out. The split should respect the highest shared unit relevant to the intended prediction.
These designs are not interchangeable, and grouped splits have limitations of their own. With few studies or participants, the held-out set may be small or unrepresentative, and a single allocation can produce unstable results. A grouped split also cannot establish performance in every possible new setting. It changes the scope of the evidence rather than guaranteeing broad generalization.
Diagnostics and Common Pitfalls
Check the split at the group level before interpreting a validation metric. Count unique participants and studies in each set, then count the overlap. Inspect records per participant and study, and identify whether the model, parameter estimates, or any preprocessing used information from groups represented in validation. Also make clear whether the reported metric weights records equally or gives groups equal weight.
- Mistake: “The validation records were withheld, so the test is independent.” Withheld rows can still share a participant or study with calibration rows. State the split unit and check group overlap.
- Mistake: “A low random-split error proves accuracy for new participants.” It may reflect prediction of additional records from represented participants. Use a participant-level holdout for a new-participant target.
- Mistake: “Random assignment removes all bias.” Random assignment of records does not remove within-participant dependence or record-count weighting. It also does not undo data-dependent calibration or tuning.
- Mistake: “A grouped split is always better.” A grouped split is appropriate when its held-out group matches the target. It may be unnecessary or poorly matched for a task explicitly limited to additional records from known participants.
If a metric improves sharply under a record split compared with a participant- or study-level split, investigate whether shared information, group heterogeneity, or unequal sampling explains the difference. The contrast is a diagnostic, not a standalone proof of a particular failure mechanism. Conversely, similar metrics do not establish that the groups are independent or that the validation target was correctly represented.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A dataset contains 12 records for each participant. The intended use is prediction for a participant not represented during calibration. Which split unit best matches the target, and why?
- In the residual model, suppose \(m=0\). Write the expected squared error for prediction of a new participant. How does it compare with the same-participant expression when \(m>0\)?
- Why can a pooled record-level RMSE give a different impression from an RMSE that weights participants equally?
- A study-level validation set includes entirely new participants, but all validation participants come from a study represented in calibration. What generalization does this design fail to test?
Solutions. (1) Hold out all records from selected participants. This prevents their measurements from contributing to calibration and aligns the split with prediction for a new participant. (2) With no participant-specific calibration records, the squared error is \(\sigma^2+\tau^2\). For \(m>0\), it is \(\sigma^2+\tau^2\sigma^2/(\sigma^2+m\tau^2)\), which is smaller under the stated assumptions because \(\tau^2\sigma^2/(\sigma^2+m\tau^2)<\tau^2\). (3) Pooled RMSE weights participants in proportion to their record counts, while equal-participant RMSE gives each participant the same weight; the two answer different weighting questions. (4) It fails to test generalization to a study absent from calibration, because the study itself is represented in calibration.