Why Put Cross-Validation Inside Cross-Validation?
A joint model can offer several reasonable choices: how the longitudinal trajectory is represented, which association structure links the longitudinal process to the event process, or how a tuning parameter is set. If the same cross-validation results are used both to choose among these options and to report performance, the reported result can be too favorable. The chosen option is, by construction, the one that did best on those particular validation splits, including any good luck in their scores.
Nested cross-validation separates these tasks. Inner folds select a modeling pipeline using only the current training data. Outer folds evaluate the selected pipeline on participants whose outcomes were not used for that selection or for fitting the model being evaluated. The method assesses the performance of a selection procedure, not merely one model chosen in advance.
As in cross-validation for joint-model prediction, the participant is the unit to split: repeated measurements and the event record for one participant must stay together. The additional nesting ensures that the outer evaluation does not also serve as the model-selection contest.
Specify the Prediction Task and Candidate Pipelines
Before splitting the data, define what prediction is being evaluated. For example, the target might be each participant’s event risk by a fixed horizon \(\tau\), using baseline covariates only, or using longitudinal history available through a specified landmark time. These are different prediction tasks. For a dynamic prediction task, the held-out participant’s measurements may be used only if they would be available by the prediction time; measurements after that time must not enter the prediction.
A candidate should be a complete, reproducible pipeline rather than just a label for one association structure. It can specify the longitudinal and event submodels, association structure, tuning choices, and prediction rule. Any data-dependent preprocessing, variable selection, or tuning belongs inside the pipeline and must be learned anew using the relevant training data. Otherwise, information from a validation or test participant can influence the fitted procedure before its prediction is assessed.
Let \(n\) denote the number of independent participants and let \(K_{\text{out}}\) and \(K_{\text{in}}\) be the numbers of outer and inner folds. For candidate pipeline \(m\), let \(S_{m,k}\) be its inner-validation score in inner fold \(k\). If smaller scores are better, the inner estimate is
The score must match the stated prediction target. For example, a Brier score at horizon \(\tau\) evaluates predictions of event status by \(\tau\), with censoring handled appropriately when needed. An integrated Brier score summarizes prediction error across a prespecified time interval. The score, horizon or interval, candidate set, and rule for ties should be fixed before looking at outer test results.
The Nested Procedure
Assign each participant, with all that participant’s longitudinal records and event information, to one outer fold. Use a splitting strategy compatible with the intended validation target and the number of events.
Split that training set into inner folds. Fit each candidate on inner training participants, predict for inner validation participants, and calculate the prespecified score. Select the candidate using only these inner results.
Refit the selected pipeline on all outer training participants. Generate predictions for the outer test participants using only information allowed by the prediction task, then calculate the outer test score.
Repeat for every outer fold and report the prespecified summary of the held-out scores, along with uncertainty and relevant design details when available.
For a survival prediction score that corrects for censoring, the censoring adjustment must not use outer test outcomes to fit or tune the prediction model. A practical design estimates any censoring distribution or weighting model using the outer training data, then applies the resulting evaluation procedure to the outer test data. The validity of such a score still depends on its censoring assumptions; nesting does not remove them.
Worked Example: Selecting an Association Structure Inside One Outer Fold
Suppose a hypothetical outer training set contains 240 participants. Two prespecified joint-model pipelines differ in their association structure: pipeline A uses a current-value association, and pipeline B uses a current-value-plus-slope association. They are compared using three inner folds and a Brier score at 2 years, for which lower is better. The inner scores are:
| Inner fold | Pipeline A | Pipeline B |
|---|---|---|
| 1 | 0.118 | 0.114 |
| 2 | 0.121 | 0.116 |
| 3 | 0.119 | 0.113 |
The mean inner scores are
Pipeline B is selected because \(0.1143<0.1193\). It is then refitted using all 240 outer-training participants. Suppose its Brier score on the separate outer test participants is \(0.126\). That \(0.126\), not the selected inner mean \(0.1143\), is the outer-fold evaluation. It estimates performance on held-out participants under this fold’s sampling conditions; it does not show that the slope association improves prediction in general or establish a causal effect.
Why the Outer Score Addresses Selection Optimism
The reason for nesting can be seen with a short argument. Suppose each candidate has a true prediction error \(R_m\), and its inner estimate is \(S_m=R_m+\epsilon_m\), where the estimation error \(\epsilon_m\) has mean zero. Even if each candidate’s score is unbiased on its own, choosing the smallest observed score tends to select a candidate with a favorable error realization.
For any set of random scores, the minimum is no greater than any particular score. In particular, for every candidate \(j\), \(\min_m S_m\leq S_j\). Taking expectations gives \(E[\min_m S_m]\leq E[S_j]=R_j\). Since this holds for every \(j\),
Thus the best observed inner score is generally optimistic as an estimate of the selected model’s future error. The outer test participants did not determine which candidate won the inner comparison, so their scores provide a separate assessment of the entire selection-and-fitting procedure. The argument explains the direction of selection bias; it does not say that every particular outer score must exceed the inner score.
Outer folds are not independent replications: their training sets overlap, and the selected pipeline can differ across folds. Therefore, the average outer score is an estimate of the performance of the specified procedure when trained on roughly the outer-training sample size. It is not an exact estimate of the performance of one final model refitted on all \(n\) participants, nor does it remove uncertainty from the finite sample.
Worked Example: Summarizing Outer-Fold Scores
In a hypothetical five-fold nested analysis, the held-out integrated Brier scores are \(0.138,\ 0.145,\ 0.132,\ 0.151,\) and \(0.134\). Each fold used its own inner selection and its own outer-training refit. The unweighted mean outer score is
The reported mean is an estimated integrated prediction error of \(0.1400\) under the specified score and time interval for this nested procedure. Lower values indicate better performance on that scale. The fold scores range from \(0.132\) to \(0.151\), so performance varies across the held-out subsets. This range is descriptive, not a confidence interval: the fold scores are dependent and five values alone do not establish a calibrated uncertainty interval.
If fold sizes differ, a participant-weighted pooled summary may be appropriate, provided the score can be computed consistently from the held-out predictions and its censoring adjustment is handled correctly. The aggregation rule should be chosen in advance rather than selected because it gives the most favorable result.
Assumptions, Diagnostics, and Failure Modes
Random participant-level folds are most interpretable when participants are exchangeable draws from the population to which predictions are intended to apply. If the intended setting involves a different population or a change over calendar time, random-fold performance may not represent that target. The split design should reflect the prediction target; nested random cross-validation is not a substitute for evaluation under a distinct deployment setting.
Check the following before trusting the result:
- Participant separation: no participant appears in both training and validation data for a split. Splitting visits or event records independently can leak participant-specific information and produce falsely favorable scores.
- Prediction-time information: every longitudinal measurement used for a test prediction is available at the stated prediction time. Future measurements or post-event information are not predictors.
- Fold adequacy: each training fold has enough participants, events, and follow-up to fit the candidate models and calculate the score. A nominal fold count is not useful if some fits fail or the score is undefined.
- Fitting stability: record convergence failures, boundary estimates, and candidate-selection frequencies. Quietly dropping failed fits or changing the candidate set after inspecting outer results can change the procedure being evaluated.
- Censoring and score definition: use a score estimator whose censoring assumptions are plausible for the data and target. For inverse-probability weighting, for example, inadequate positivity or a misspecified censoring model can destabilize evaluation even when fold separation is correct.
Worked Example: Detecting Leakage in a Dynamic Prediction Split
A hypothetical dataset has 120 participants, each with repeated biomarker values and an event record. The goal is to predict event risk at a 1-year landmark using measurements recorded by that landmark. A proposed split assigns individual visits randomly to folds. Participant 17’s early measurements enter training, while that participant’s later measurements and event record enter validation.
This is not a valid participant-level outer split: the training data already contain information from participant 17. Even if no event indicator is literally copied into the training set, the model can exploit participant-specific patterns learned from that person’s records. The validation score can then overstate performance for genuinely new participants.
The correction is to assign all records for participant 17 to one fold. If participant 17 is in the outer test fold, fit the joint model using other outer-training participants and predict participant 17’s 1-year risk using only the measurements available by the 1-year landmark. Do not use later measurements to construct that prediction. This preserves the stated target: prediction for a held-out participant based on the permitted history.
How Nested Cross-Validation Differs From Alternatives
| Approach | What it does | Main limitation for selection and evaluation |
|---|---|---|
| Single cross-validation | Uses folds to estimate prediction performance or compare candidates. | If the best candidate is selected and the same scores are reported as its performance, selection optimism remains. |
| Nested cross-validation | Uses inner folds for selection and outer folds for held-out evaluation. | Computationally demanding; estimates the specified procedure at a reduced training size, with dependent outer scores. |
| Single holdout set | Uses one set for selection or training and another untouched set for evaluation. | Can be inefficient in small samples; reusing the holdout for repeated model changes compromises its role as an independent evaluation. |
| Bootstrap optimism correction | Uses resampling to estimate and adjust apparent optimism. | Answers through a different resampling design and requires its own careful implementation; it is not the same nested selection procedure. |
Nested cross-validation is useful when the sample is too limited to reserve a large independent test set and model selection is part of the intended workflow. It can be computationally expensive because many joint models must be fitted. It also cannot rescue an overlarge candidate set, poorly specified prediction task, inadequate event information, or unsuitable censoring assumptions. The candidate set and selection criterion should be scientifically defensible and specified before outer results are examined.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Why is the smallest inner-validation score not an appropriate standalone estimate of the selected pipeline’s performance?
- In a dynamic prediction analysis, when may a held-out participant’s longitudinal measurements be used?
- Four outer-fold Brier scores are \(0.12,\ 0.15,\ 0.13,\) and \(0.16\). Calculate their unweighted mean and interpret it.
- What is the unit that should be assigned to folds when each person contributes repeated measurements and an event record?
Solutions. (1) Selection favors candidates with favorable validation noise, so the minimum selected score is generally optimistic; outer test scores evaluate selection separately. (2) Only measurements available by the prediction time specified in the task may be used, not later measurements. (3) \((0.12+0.15+0.13+0.16)/4=0.14\); this is the mean held-out Brier score for the specified target and score, with lower values indicating better prediction on that scale. (4) Assign participants as whole units, keeping each person’s records in one fold.