Why the Two Validation Designs Must Agree
Nested cross-validation has two jobs. The inner design compares candidate analysis pipelines or hyperparameter settings using only an outer training set. The outer design evaluates the selected pipeline on data held out from all fitting and tuning. A valid outer score does not, by itself, guarantee that the inner process selected a model for the intended use. If the inner folds represent one prediction task and the outer folds represent another, the reported evaluation may describe a procedure whose tuning objective was misaligned with its stated target.
The relevant match is not necessarily identical fold construction. For example, the inner folds cannot hold out the same outer test site: that site is unavailable during tuning. Rather, the inner folds should represent the same kind of separation and prediction target as the outer folds. If the target is prediction for a new site, inner validation should generally hold out sites, too. If the target is prediction for new unrelated individuals at sites already represented in training, inner folds should hold out individuals while preserving any required family grouping.
The prediction target must be stated before choosing folds. The introductory material on defining the genomic prediction target distinguishes, for example, predicting a new person from predicting at a site not represented in training. Those settings can differ substantially: site-specific processing, population structure, assay differences, or recruitment can make prediction at a familiar site easier than transfer to a new one.
Specify the Target, Then Match the Design
Let \(O_k\) be the observations in outer fold \(k\), and let \(T_k\) be the corresponding outer training data. Within \(T_k\), let \(V_{kr}\) be an inner validation fold and \(T_{kr}\) its inner training data. For a candidate setting \(\lambda\), such as a regularization strength, the inner process fits on \(T_{kr}\), predicts in \(V_{kr}\), and calculates a prespecified loss. It selects a setting using the inner results. The selected procedure is then refit using \(T_k\) and evaluated on \(O_k\).
Compatibility requires making several choices together. First, specify what is held out: individuals, families, sites, or another deployment-relevant unit. Second, specify which observations may not cross a split. Related family members, repeated samples, and site-specific groups may need to remain together. Third, define the metric and how fold-level results are combined. The metric should reflect the stated target: equal weighting of sites answers a different question from weighting every person equally when site sizes differ.
The outer split must also respect the target. Inner folds are drawn only from the outer training data, so they cannot include outer test observations or use them to make decisions. Scaling, imputation, feature selection, and every other data-adaptive step must be fitted within the relevant training fold, as in a leakage-safe pipeline. Grouping alone does not prevent leakage if a preprocessing operation has already used held-out observations.
State whether predictions concern new individuals from represented sites, new families, unseen sites, or another clearly defined population.
Identify groups that must remain intact, such as families or sites. Apply those constraints within both inner and outer partitions.
Define the loss and whether observations, sites, or another target-relevant unit receive equal weight.
Each inner validation split must be drawn from the outer training set and provide a meaningful analogue of the outer separation. If too few independent groups remain, the design may not support reliable tuning for that target.
Worked Example: New People at Known Sites or New Sites?
Suppose a hypothetical genomic cohort has 600 participants from six hospitals. Participants have binary disease status, and some are relatives. The intended use must be decided before splitting.
Target A: new unrelated participants at hospitals already represented in training. An outer split can hold out participants, stratified by hospital where feasible, while keeping each family wholly on one side. Inner folds within each outer training set should likewise hold out whole families, with hospital representation maintained where possible. This estimates performance for new people under a setting in which training has included data from the hospitals.
Target B: participants at a hospital absent from training. The outer design should hold out an entire hospital. Within each outer training set, inner folds should hold out whole hospitals among the remaining hospitals; family members must also remain together. Holding out individuals at random in the inner loop would allow the same hospitals to appear in inner training and validation, so tuning would favor performance at familiar hospitals rather than transfer to an unseen one.
These are different prediction questions, not merely two ways to partition the same task. If only a small number of hospitals are available, an inner leave-one-hospital-out design may have few validation groups and unstable comparisons. That limitation should be reported; replacing it with random participant folds silently changes the tuning target rather than solving the lack of independent hospitals.
Match the Metric and Its Weighting
Even when both levels hold out the right type of group, they can still be misaligned if they aggregate performance differently. For a continuous outcome, a person-weighted mean squared error (MSE) gives more influence to sites with more participants. An equal-site average of site-specific MSEs gives every site the same influence. Either can be appropriate, but they answer different questions. The inner objective should use the weighting intended for the outer target.
Worked Example: Weighting Changes the Selected Setting
Consider two hypothetical inner validation sites. Site A has 100 participants and site B has 10. Two candidate hyperparameter settings, \(A\) and \(B\), have the following validation MSEs, in squared outcome units:
| Candidate setting | Site A MSE, 100 participants | Site B MSE, 10 participants |
|---|---|---|
| A | 1 | 16 |
| B | 4 | 9 |
If the target gives every site equal weight, the mean site MSE for setting A is \((1+16)/2=8.5\), while for setting B it is \((4+9)/2=6.5\). The inner procedure selects setting B.
If the target instead samples a person from the combined population represented by these two sites, the person-weighted MSE for setting A is \((100\times1+10\times16)/110=260/110=2.364\). For setting B it is \((100\times4+10\times9)/110=490/110=4.455\). This objective selects setting A.
The rankings differ because the weighting defines the target average. Neither calculation is inherently wrong. Choosing the equal-site objective and then reporting a person-weighted outer score—or the reverse—means the tuning and evaluation answer different questions. The site sizes and MSE values here are hypothetical.
Why Compatible Tuning Supports a Meaningful Outer Evaluation
The outer score evaluates the complete procedure, including the choice made by inner tuning. A useful justification for compatibility follows from what the inner comparison is trying to approximate. Suppose the intended deployment loss is an average over a specified population of held-out units, with a fixed loss definition and weighting. Inner validation estimates candidate settings’ performance under an analogous separation from training. Selecting by that objective directs tuning toward the deployment loss. Outer validation then evaluates the selected procedure on separate held-out units drawn according to the outer design.
This reasoning depends on the outer and inner validation units representing the intended target and on the analysis being repeated without information crossing their boundaries. It does not establish that a cross-validation estimate is exactly unbiased in every finite sample. Estimates can be noisy, group counts may be small, and the observed groups may not represent future deployment conditions. Compatibility makes the question coherent; it does not remove sampling uncertainty or distribution shift.
The design relationship can be summarized as follows:
If the first and last parts differ, a carefully executed nested algorithm can still estimate a valid quantity—but it may not be the quantity the analyst intends to report. If the inner design cannot reproduce the outer target using the available outer training data, the limitation must be explicit. A proxy design may be necessary, but it should be chosen as a documented approximation rather than treated as equivalent.
Worked Example: A Multi-Site Outer Evaluation
Consider a hypothetical study with four hospitals, each contributing genomic predictors and a continuous biomarker outcome. The stated target is the expected squared prediction error at a new hospital, with each hospital equally important. An outer leave-one-hospital-out design creates four outer folds. For each fold, one hospital is held out, and the other three form \(T_k\).
Within each \(T_k\), an inner leave-one-hospital-out procedure compares candidate regularization settings. Each inner validation fold contains one of the three training hospitals, and all preprocessing is fitted using only the other two hospitals in that inner split. Family members, if present, remain together. This inner procedure cannot validate on the outer hospital, but it mirrors the unseen-hospital separation using the available training sites.
Suppose the outer-fold hospital-specific MSEs for the selected procedure are 4, 9, 16, and 25 squared biomarker units. Because the target weights hospitals equally, the reported summary is the mean of these four MSEs:
The corresponding square root is \(\sqrt{13.50}=3.674\) biomarker units. It summarizes the root of the equally weighted mean squared error; it is not the arithmetic mean of the four hospital-specific RMSEs. If instead the study target were a person sampled from a population with unequal hospital sizes, an observation-weighted calculation could be more appropriate. The choice must be made consistently in the inner objective and outer summary.
The example assumes that the hospitals are the relevant independent deployment units, that the outer-held-out hospital is not used in any fitting or tuning, and that the available training hospitals permit inner comparisons. With only four hospitals, the estimates may be sensitive to which hospitals are observed, and there are few independent groups. The calculation is valid for the stated folds and weighting, but it does not guarantee stable generalization to all possible hospitals.
Alternatives, Diagnostics, and Limitations
| Design choice | Question it can address | Potential mismatch |
|---|---|---|
| Random individual folds, with family grouping where needed | New individuals from a population represented in training | Does not evaluate transfer to entirely unseen sites if sites appear in both partitions. |
| Site-grouped folds | Prediction at sites absent from training | May be unsuitable for a target restricted to new individuals at known sites; few sites can also make tuning unstable. |
| Equal-group metric aggregation | Average performance across groups given equal importance | Does not represent an average person if group sizes differ and the target weights people equally. |
| Observation-weighted metric aggregation | Average performance for an observation drawn from the represented pooled population | Large groups dominate, which may conflict with a target giving each site equal importance. |
Before running the analysis, inspect fold assignments and confirm that prohibited groups never cross training and validation boundaries at either level. Check that every inner training set contains enough independent groups and outcome variation to fit the candidate pipelines. Also verify that the same metric definition and weighting are used for inner comparisons and the intended outer summary. For example, an AUC calculation may not be defined in a validation fold containing only one outcome class; fold construction must be compatible with the metric’s requirements.
Warning signs include inner folds that mix hospitals when the outer folds hold hospitals out, relatives split between training and validation, or a tuning objective that weights people equally while the reported target weights sites equally. A mismatch need not cause a numerical error or obvious software warning. It can instead produce a plausible result about the wrong prediction task.
A common mistake is to assume that any inner design is acceptable because the outer test set remains untouched. The outer score does remain separated, but the selected hyperparameter can still be optimized for a different setting. The correction is to state the target first, align the inner separation and metric with it as closely as the available outer training data allow, and describe any unavoidable approximation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- An outer design holds out entire hospitals to estimate performance at an unseen hospital. What kind of inner split is the natural match, and why?
- Why can equal-site and person-weighted MSE select different hyperparameter settings when site sizes differ?
- In a study with related participants, what must be checked in both inner and outer partitions?
- Does an untouched outer test set guarantee that the inner tuning objective matches the deployment target?
Solutions. (1) Hold out whole hospitals within each outer training set, so inner tuning compares settings under a separation that approximates transfer to an unseen hospital. (2) Person-weighted MSE gives larger sites more influence, while equal-site MSE gives each site equal influence; as the worked calculation shows, these objectives can rank settings differently. (3) Family members must remain together across each training-validation boundary at both levels, and preprocessing must be fitted using only the relevant training data. (4) No. The outer test set can be untouched while the inner folds still represent a different task; the target, split unit, metric, and weighting must be aligned explicitly.