Why Nest the Validation?
A PRS analysis may compare several score-building methods and many settings within each method. These choices can include a p-value threshold, a shrinkage method, or a method-specific tuning parameter. If the same validation outcomes are used both to choose the best setting and to report its performance, the reported result is generally optimistic: the winning setting may have benefited from random features of those outcomes.
Nested validation separates these jobs. The inner loop chooses a method and its settings using only the outer training data. The outer loop evaluates the resulting selection-and-fitting procedure on individuals whose outcomes did not influence that choice. The target is therefore not the performance of a setting chosen in advance; it is the performance of the procedure that selects a setting from the specified candidates.
What Must Be Inside the Loops?
Let \(\mathcal{C}\) be the candidate set of PRS methods and hyperparameter settings, specified before examining outer-fold results. It may include threshold-based scores and shrinkage-based scores. Let \(M\) be a prespecified validation metric, with its direction specified in advance: for example, larger values may be better, whereas prediction error measures such as mean squared error are better when smaller.
For each outer fold, the inner loop evaluates each candidate on inner validation folds. It selects the candidate with the best average inner-fold metric. Then the complete selected pipeline is fitted on the full outer training set and evaluated on the held-out outer fold. Any choice informed by phenotype data belongs inside the loop: this includes choosing score settings and, where applicable, fitting outcome models or selecting covariate adjustments.
Some parts of score construction may be fixed independently of the validation cohort. For example, GWAS summary statistics from an independent discovery sample can be treated as fixed inputs. But if discovery statistics overlap with the evaluation individuals, held-out outcomes or genotypes can leak into score construction. The overlap must be removed or otherwise handled so that the outer test individuals do not contribute information to their own predictions. For inner-loop evaluation, the same principle applies to inner validation individuals.
Data-adaptive preprocessing also needs care. A transformation estimated from the evaluation cohort, a phenotype-informed variant filter, or an outcome model fitted using held-out outcomes can make validation performance misleading. Estimate such steps from the relevant training partition and apply them to its validation partition. Fixed, prespecified rules that do not learn from the held-out outcomes do not automatically require refitting.
Specify the candidate PRS pipelines, validation metric, fold structure, and any grouping constraints before using outer-fold results.
Keep its outcomes out of inner selection, model fitting, and any data-adaptive step that would use them.
Compare candidates across inner folds and select the one with the best prespecified average inner-fold metric.
Refit the selected pipeline on all outer training data, predict the outer test fold, and calculate its metric.
Summarize the outer-fold metrics according to a prespecified rule, and do not use them to revise the candidates and then report the same results as an untouched evaluation.
Worked Inner and Outer Loops
Worked Example: Selecting a PRS Setting in the Inner Loop
Suppose an outer training set is divided into four inner folds. Three candidates are compared using inner-fold mean squared error (MSE), measured in squared phenotype units. Smaller MSE is better. The values below are hypothetical fold results.
Candidate T is a threshold score with fold MSE values \(0.94, 0.88, 0.91,\) and \(0.87\). Its mean is
Candidate S is a shrinkage score with values \(0.90, 0.85, 0.88,\) and \(0.83\):
Candidate L is a penalized score with values \(0.92, 0.89, 0.90,\) and \(0.86\):
The inner-loop choice is S because \(0.865\) is the smallest average MSE. This is a selection decision, not an estimate of the final out-of-sample performance: the inner-fold results were used to choose S. The selected pipeline is next refitted on all the outer training data before it is evaluated on the outer test fold.
Worked Example: Combining Outer-Fold Performance
Consider three outer folds of equal size. The inner loop selects a candidate separately within each outer training set; it selects T in the first fold and S in the second and third. After refitting each selected pipeline on its outer training set, the respective outer test-fold MSE values are \(1.04, 0.96,\) and \(1.10\) squared phenotype units.
For this prespecified summary, take the unweighted mean of the three fold MSE values. Equal fold sizes make this the same as pooling the squared prediction errors over the test observations:
This estimates the performance, under the stated fold design, of the procedure that selects among T, S, and L using inner validation and then refits on the outer training data. It does not estimate the performance of S alone, even though S was selected in two folds. Nor does it imply that the exact value \(1.033\) will be attained in a new cohort.
If outer folds have different sizes, an unweighted mean gives each fold equal weight. A sample-weighted summary instead weights each fold metric by its test-fold size. Choose the summary to match the intended estimand and state it; do not silently switch between the two.
Why the Outer Loop Protects the Estimate
The source of selection optimism can be seen by writing a candidate's observed inner error as its underlying error plus noise. If \(\widehat L_j=L_j+\epsilon_j\) is the observed error for candidate \(j\), choosing the candidate with the smallest \(\widehat L_j\) tends to favor candidates with unusually negative noise. Even if each \(\widehat L_j\) is unbiased for its own \(L_j\), the minimum is not generally unbiased for the minimum underlying error.
The inequality follows because, for each fixed candidate \(r\), \(\min_j \widehat L_j\leq \widehat L_r\). Taking expectations gives \(E[\min_j \widehat L_j]\leq L_r\); this holds for every \(r\), including one with the smallest \(L_r\). Thus the best observed inner error is prone to optimism.
In an outer fold, the candidate choice and fitted model depend on the outer training data, while the outer outcomes are reserved for evaluation. Conditional on the outer training data, those held-out outcomes have not selected the candidate. This is the central justification for using outer-fold results to assess the whole tuning procedure. It depends on a valid split and on the held-out observations representing the intended prediction setting. Outer folds are not independent replicates: their training sets overlap, so their metrics should not be treated as independent measurements merely because they come from different folds.
Fold Design and Leakage Checks
Worked Example: Keeping Related Individuals Together
Suppose a hypothetical cohort contains 120 individuals from 30 families, with four individuals per family. Randomly assigning individuals to five outer folds could put members of the same family in both training and test data. Because relatives share genetic information, this can make the test task easier than predicting genuinely new families.
Instead, assign whole families to folds. Put six families, or \(6\times4=24\) individuals, in each outer test fold. Each outer training set then contains \(120-24=96\) individuals from 24 families. Divide those 24 training families into four inner folds of six families each: each inner validation fold has 24 individuals, and the corresponding inner training set has 72.
The arithmetic confirms that no family crosses a fold boundary. The evaluation now targets prediction for held-out families under this design, rather than prediction for new individuals whose close relatives may already be in training. If the intended use is prediction in a new site or time period, the split should reflect that target instead.
Before interpreting results, check that each outer test fold was untouched until its predictions were formed; each inner validation fold was excluded from the corresponding inner fit; and all relatives or other dependent groups were kept together when the prediction target requires it. Also check that the candidate set and metric were fixed before examining outer performance. A split that ignores population, site, batch, or time structure can answer a different question from the one intended.
Alternatives, Limitations, and a Common Mistake
| Approach | What it does | Main limitation |
|---|---|---|
| Non-nested cross-validation | Uses the same folds to choose a candidate and summarize its performance. | The winning candidate’s reported performance is subject to selection optimism. |
| Nested cross-validation | Uses inner folds for selection and outer folds for evaluation of the selection procedure. | Computationally more demanding; estimates performance at the outer training-set size and depends on the fold design. |
| Single holdout | Uses one data partition for tuning and a separate reserved test set for evaluation. | Can be valid if the test set stays untouched, but the result may depend strongly on one split and uses data less efficiently for fitting. |
| Independent external validation | Evaluates a finalized pipeline in a separate cohort. | Requires suitable independent data; its population and design may differ from the development setting. |
Nested validation does not make a poor candidate set good, remove bias caused by a mismatched discovery sample, or guarantee transport to another population. Its outer estimate concerns the tuning-and-fitting procedure trained on approximately the outer training fraction of the data, not necessarily a final model trained on every available individual. Outer-fold results may also be noisy, especially with small cohorts; a later assessment of their precision should not be replaced by treating the folds as independent.
Common mistake: “The shrinkage score had the best inner-fold result, so I will report that result as its performance.” This reuses the data that selected the winner, so it can be optimistic. Use the inner result to select, refit the selected pipeline on the full outer training set, and report its outer-fold evaluation. Also, do not repeatedly alter the candidate set after seeing outer results and still describe those results as an untouched evaluation. If a final model is needed after evaluation, run the prespecified tuning procedure on all development data; the outer-fold models themselves are evaluation fits, not automatically the final deployed model.
Check Your Understanding
Try each question, then check the concise solutions below.
- Why should the lowest inner-fold error not be reported as an unbiased performance estimate of the selected candidate?
- In a nested analysis, what data are used to refit a candidate after the inner loop selects it?
- Four equal-sized outer folds have MSE values \(0.80, 1.00, 0.90,\) and \(1.30\) phenotype units squared. Calculate the unweighted mean outer MSE.
- Why might random individual-level folds be unsuitable when the target is prediction for new families?
Solutions. (1) The candidate was selected for a favorable inner result, so that result includes selection optimism. (2) Refit the selected pipeline on all data in that outer training set, not on the outer test fold. (3) \((0.80+1.00+0.90+1.30)/4=4.00/4=1.00\) phenotype units squared. (4) Relatives may be split between training and test data, allowing shared genetic information to make evaluation easier than prediction for genuinely new families; assign entire families to folds.