Why the Split Must Follow the Subject
A multi-state subject can contribute several transitions, risk intervals, landmarks, or repeated predictions. These records are not automatically independent observations: they may share measured covariates, a latent propensity to transition, and parts of the same observed history. Cross-validation is intended to assess how a prediction procedure performs on subjects not used to fit it. If one subject’s records are divided between training and validation, the validation records may be unusually familiar to the fitted model.
This is a design problem, not a special choice of prediction metric. A transition-specific Brier score or AUC can be calculated from the predictions produced by a flawed split, but the metric cannot repair information leakage in the split. The key unit to allocate is the subject, while the prediction task still specifies the transition, prediction time, and horizon being assessed.
“Complete history” means all records that could carry information from that subject into model fitting or tuning. It does not mean that every part of a validation subject’s history is forbidden at prediction time. For example, a landmark prediction may appropriately use covariates and past states observed by the landmark if those data would be available when the prediction is made. It must not use later transitions or outcomes to fit the model, select its settings, or construct that prediction.
What a Subject-Level Fold Estimates
Suppose the target is prediction for a new subject from the same population and under the same data-collection and prediction-time rules as the validation subjects. With \(n\) subjects and \(K\) folds, write \(V_k\) for the subject indices in fold \(k\). The training indices for that fold are all subjects not in \(V_k\). Fit the entire prediction procedure using only those training subjects, then generate predictions for the validation subjects.
Here \(\widehat{f}_{-k}\) is the prediction rule fitted without subjects in \(V_k\), and \(L\) is a loss defined for the specified prediction target. This expression assumes one equally weighted loss contribution per subject. If a subject has multiple eligible prediction times, first define how those records contribute: for example, average the losses within each subject and then average across subjects, or weight every prediction opportunity equally. These target different averages and should not be treated as interchangeable.
The basic justification is direct. Each validation subject’s loss is computed using a fitted rule that did not use that subject’s data. If subjects are independent and exchangeable draws from the target population, and the prediction task and evaluation rules match the intended application, these losses provide an assessment of performance on held-out subjects. Cross-validation is not exactly unbiased for the final model fitted on all \(n\) subjects: each fold’s model uses fewer subjects. Its estimate concerns the specified fitting procedure at approximately the fold-specific training size, subject to sampling variation and the chosen loss.
Worked Example: Detecting Leakage From a Row-Level Split
Consider a deliberately simple hypothetical data set with four subjects. Each subject contributes two transition records, and the two records have the same binary outcome in this toy example. Subjects A and B have outcome 1; subjects C and D have outcome 0. A row-level split puts the first record for every subject in training and the second record for every subject in validation. Suppose the fitted rule can memorize a subject identifier and return that subject’s training outcome.
In the validation rows, the memorized outcome matches every recorded outcome: A and B are predicted as 1, while C and D are predicted as 0. Thus the row-level validation accuracy is \(4/4=1\), or 100%. But the rule has not demonstrated that it can predict for a new subject: every validation identifier already appeared in training.
Now keep each subject’s two records together and assess a new subject without a previously observed identifier. A rule that has no information beyond the overall training prevalence has no identifier-specific information. In this balanced example, that prevalence is 2/4 = 0.5. A constant prediction of 1 for all four held-out subjects would be correct for A and B only, giving \(2/4=0.5\), or 50% accuracy. The contrast arises from the split and the identifier-memorizing rule, not from a claim that every row-level split must produce optimistic results. Depending on the records and model, the amount of distortion can vary, and sometimes row-level and subject-level results may coincide.
Build the Entire Procedure Inside Each Fold
The boundary applies to every data-dependent part of the prediction procedure, not only to the final regression or transition model. For each fold, fit any imputation rules, scaling constants, feature-selection steps, baseline transition quantities, and model parameters using training subjects alone. Apply those fitted choices to validation subjects. If an analysis compares tuning settings, the validation outcomes must not be used to choose among them and then also be presented as an untouched assessment.
Specify the transition or state prediction, prediction time, horizon, eligible subjects, and performance metric. Decide how multiple predictions from one subject will be weighted.
Use subject identifiers to ensure that every record from a subject is in exactly one fold. If the data have a higher-level dependence, such as multiple subjects from one household, consider whether that higher level is the appropriate grouping unit.
Estimate the model and every preprocessing step using only subjects outside the validation fold. Select tuning settings within this training set if tuning is part of the procedure.
Use only information available at the intended prediction time. Store each subject’s out-of-fold prediction and compute the prespecified metric using the appropriate outcome and censoring-handling method.
Combine subject-level losses or fold-specific metrics according to the chosen estimand. Report the grouping unit, fold construction, and how repeated prediction opportunities were weighted.
The subject partition alone does not solve other design problems. The outcome definition must respect competing transitions and the prediction horizon; incomplete follow-up must be handled by an appropriate evaluation method; and subjects must be eligible under the stated prediction task. The earlier material on transition-specific metrics supplies those target definitions. Here the concern is that their computation must be based on predictions made without using the validation subject to train or tune the procedure.
Worked Example: Cross-Validated Brier Loss With One Prediction per Subject
Suppose four hypothetical subjects are divided into two folds, with subjects A and B in fold 1 and C and D in fold 2. The outcome \(Y_i\) is whether a specified transition has occurred by a fixed horizon, and the predictions are made by models trained without the corresponding validation fold. Assume all four outcomes are known at that horizon.
| Subject | Validation fold | Observed outcome \(Y_i\) | Predicted probability \(\hat p_i\) | Squared error |
|---|---|---|---|---|
| A | 1 | 1 | 0.80 | \((1-0.80)^2=0.0400\) |
| B | 1 | 0 | 0.30 | \((0-0.30)^2=0.0900\) |
| C | 2 | 1 | 0.60 | \((1-0.60)^2=0.1600\) |
| D | 2 | 0 | 0.20 | \((0-0.20)^2=0.0400\) |
For a binary outcome, the Brier loss is \((Y_i-\hat p_i)^2\). The subject-weighted cross-validated average is therefore
The fold-specific means are \((0.0400+0.0900)/2=0.0650\) and \((0.1600+0.0400)/2=0.1000\). Since the folds contain the same number of subjects, their equally weighted mean is \((0.0650+0.1000)/2=0.0825\), matching the subject-level calculation. The result is an average squared probability error for this binary horizon outcome; it is not a probability that a prediction is correct.
Tuning Requires an Outer Assessment
If the same cross-validation results are used both to choose a tuning setting and to report performance, the reported minimum can be too favorable. This is a selection effect: among several noisy estimates, the smallest is likely to be selected partly because it happened to be low. A common remedy is nested cross-validation. The outer folds assess the complete selection procedure; within each outer training set, inner folds select the tuning setting. The outer validation subjects are excluded from both model fitting and inner selection.
For each outer fold, the selected setting may differ. That is not a defect: the procedure being assessed is “select a setting using inner cross-validation, then fit and predict,” rather than “use one setting chosen with knowledge of all outcomes.” If a final model is later fitted using all available subjects, its tuning is carried out using those subjects’ data; the outer assessment still describes the selection-and-fitting procedure at the outer training size.
Worked Example: Nested Subject-Level Validation
Suppose six hypothetical subjects are assigned to three outer folds, with two subjects per fold. For each outer fold, inner cross-validation is performed only among the four outer-training subjects, and it selects a penalty setting. The selected model is fitted using those four subjects and applied to the two held-out subjects. Assume the six out-of-fold binary outcomes and predictions produce these squared errors, in subject order: 0.01, 0.09, 0.04, 0.16, 0.00, and 0.09.
The outer-fold mean losses are \((0.01+0.09)/2=0.050\), \((0.04+0.16)/2=0.100\), and \((0.00+0.09)/2=0.045\). Since all folds have two subjects, the pooled subject-weighted outer estimate is
This estimates the performance of the procedure that selects a penalty by inner validation and then refits, under the outer-fold training sizes. If the six errors had instead been used to pick the penalty and report the selected model’s performance, they would no longer constitute an untouched outer assessment.
Diagnostics, Alternatives, and Limits
Before interpreting a cross-validated result, check that subject identifiers are disjoint between each training and validation set and that no derived record, duplicate, or linked history crosses the boundary. Verify that preprocessing and tuning are repeated inside each training set. Also check whether every fold contains enough relevant transition outcomes to calculate the intended metric. Rare transitions can make fold-specific metrics unstable or undefined; changing the number or construction of folds may help, but should not be used to conceal inadequate information.
For a metric that is an average of individual losses, pooling the out-of-fold subject losses gives a clear subject-weighted summary. For a pairwise metric such as AUC, the fold-specific estimates may be combined in a way that matches the intended weighting. Simply pooling predictions from different fold-fitted models can compare predictions produced by different fitted scales, including across folds. State whether the summary averages fold-specific AUCs or uses another prespecified combination, and ensure the approach answers the intended question.
A random record-level split is not equivalent to subject-level cross-validation when subjects contribute multiple records. A single subject-level holdout is simpler and can provide a clear evaluation set, but uses fewer subjects for fitting and may yield a noisy estimate. Bootstrap optimism correction, covered in the earlier material on bootstrap optimism correction for multi-state predictions, is another resampling approach; it does not remove the need to identify the independent sampling unit. Subject-level cross-validation is also not a substitute for assessing performance under a different calendar period or population.
A further limitation is that grouping subjects prevents within-subject leakage but does not guarantee independence between subjects. If subjects share a clinician, center, family, or other source of dependence, randomly assigning subjects across folds can still place related information on both sides. The split should then match the intended prediction setting and the relevant independent unit. Finally, cross-validation assesses prediction under the data-generating and follow-up conditions represented in the sample. It cannot by itself establish performance in settings whose case mix, measurement process, or follow-up differs.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions below.
- A subject contributes six transitions. How should those records be assigned in ordinary subject-level cross-validation?
Solution: All six records stay together in one fold. If the subject is in a validation fold, none of the subject’s records may contribute to that fold’s model fitting or tuning. - Can a landmark prediction for a validation subject use covariates measured before the landmark?
Solution: Yes, if those covariates would be available at the intended prediction time. They may be inputs to that subject’s prediction, but the subject’s later outcomes must not be used to fit or tune the model. - Three subjects have losses 0.02, 0.08, and 0.20. What is their equally subject-weighted mean loss?
Solution: \((0.02+0.08+0.20)/3=0.30/3=0.10\). - Why is the smallest inner cross-validation loss among several tuning settings not, by itself, an untouched performance estimate?
Solution: The same noisy results were used to select the setting, so the selected minimum can be favorably low by chance. An outer subject-level fold, excluded from that selection, assesses the full tuning procedure. - If each validation fold has a different fitted model, what should be considered before pooling all out-of-fold predictions to calculate AUC?
Solution: AUC compares predictions across cases and controls. Predictions from different fold-fitted models may not be on directly comparable scales, so a prespecified aggregation of fold-specific metrics may be more appropriate for the target estimand.