Nested Cross-Validation for Model Tuning

Why Separate Tuning From Evaluation?

Genetic prediction often requires choices that affect the fitted model: a ridge penalty, a lasso mixing parameter, a clumping-and-thresholding setting, or another tuning value. These choices are commonly made by comparing prediction performance across candidate settings. The comparison itself uses data, so the winning setting may partly reflect random features of the data used for comparison. If the same results are then reported as the model’s evaluation, performance can look better than it would on new individuals.

Nested cross-validation addresses this by assigning two distinct roles to two levels of folds. Inner folds select the hyperparameter. Outer folds estimate the performance of the procedure that includes that selection. The distinction matters: outer test observations must not influence which setting is chosen for their fold.

Definition: In nested cross-validation, the data are divided into outer folds. For each outer fold, its observations are held aside as an outer test set. The remaining observations form an outer training set, which is divided again into inner folds to select a hyperparameter. The selected model is then refitted using the full outer training set and evaluated on the outer test set. The outer-fold evaluations are combined to estimate prediction performance.

Let each observational unit have genetic predictors \(G_i\) and an outcome \(Y_i\). For a quantitative outcome, one possible loss is squared prediction error, \((Y_i-\hat{Y}_i)^2\), measured in squared outcome units. The population prediction risk of a fitted prediction rule is its expected loss for a new individual drawn from the target population. In practice, cross-validation estimates this risk using held-out observations.

The workflow below assumes that the candidate settings and the selection criterion are specified in advance. It also assumes that observations are independent and drawn from the same target population, as in ordinary individual-level random-fold cross-validation. The fold design must reflect the intended prediction target; the procedure’s interpretation depends on that design. For simplicity, the examples use mean squared error (MSE), which is smaller when predictions are better.

The Two Levels of Folds

Suppose there are \(n\) individuals, \(K\) outer folds, and \(J\) inner folds. Each individual appears in exactly one outer test fold. In a given outer iteration, the other \(K-1\) folds form the outer training set. Only this training set is divided into inner folds. Each candidate hyperparameter is assessed by fitting on some inner-training observations and predicting the corresponding inner-validation observations.

For each candidate, the inner validation losses are combined according to a prespecified rule, such as the mean of the inner-fold MSEs when the folds are equal in size. The candidate with the lowest combined MSE is selected. The model with that setting is then refitted using all observations in the outer training set. Its predictions on the outer test set produce that iteration’s outer-fold performance estimate.

Hold out an outer fold.
Set aside one outer fold and do not use its outcomes to select the hyperparameter for this iteration.

Tune within the outer training set.
Use inner folds to evaluate every candidate setting, then apply the prespecified rule to select one.

Refit and evaluate.
Refit the selected model on the full outer training set, predict the held-out outer fold, and calculate its loss.

Repeat and aggregate.
Repeat for every outer fold, so every individual receives an outer-fold prediction from a model whose tuning excluded that individual.

Worked Example: Tuning a Ridge Penalty Within One Outer Fold

Consider hypothetical data on 120 unrelated individuals, with a quantitative phenotype measured in trait units. There are four outer folds of 30 individuals each. In one iteration, 30 individuals are held out, leaving 90 in the outer training set. Three-fold inner cross-validation on those 90 individuals gives the following validation MSEs, in squared trait units:

Ridge penalty	Inner-fold MSEs	Mean inner MSE
0.01	14, 13, 15	14
0.1	12, 11, 13	12
1	13, 12, 14	13

For penalty 0.01, the mean is \((14+13+15)/3=42/3=14\). For penalty 0.1, it is \((12+11+13)/3=36/3=12\). For penalty 1, it is \((13+12+14)/3=39/3=13\). The prespecified rule is to choose the penalty with the smallest mean inner MSE, so the selected penalty is 0.1.

The model is now refitted with penalty 0.1 using all 90 outer-training individuals. Suppose the sum of squared prediction errors for the 30 held-out individuals is 330 trait-units squared. The outer-fold MSE is

\mathrm{MSE}_{\mathrm{outer}}=\frac{330}{30}=11

Thus, for this outer fold, the tuned procedure has an estimated prediction error of 11 squared trait units. The outer test outcomes did not determine the penalty; they were used only for the evaluation.

Why Reusing Tuning Folds Is Optimistic

The optimism comes from selection. Even if each candidate’s validation estimate is unbiased for its prediction risk, the smallest of several noisy estimates tends to be lower than the corresponding true risk. Choosing the candidate because it had the most favorable validation result selects not only for model quality but also, in part, for favorable validation noise.

To see the argument, write \(\hat{R}_m\) for the estimated validation risk of candidate \(m\), and \(R_m\) for its true risk. Suppose each estimate is unbiased, so \(E(\hat{R}_m)=R_m\). For any candidate \(m\), the minimum observed risk cannot exceed its observed risk:

\min_j \hat{R}_j \leq \hat{R}_m

Taking expectations gives \(E(\min_j\hat{R}_j)\leq R_m\) for every \(m\). Therefore,

E\left(\min_j \hat{R}_j\right)\leq \min_m R_m

So the validation risk reported for the selected candidate is, on average, no greater than the true risk of the best candidate in the set. When candidates’ estimates vary and selection can exploit that variation, the reported minimum is typically too favorable. This does not mean every selected model will perform worse on every new sample; it describes the direction of the selection effect across repeated samples.

Worked Example: The Lowest Validation Error Can Reflect Noise

Suppose, hypothetically, that three candidate genetic prediction settings have the same true MSE: 10 squared trait units each. On one set of tuning folds, random variation produces validation estimates of 9.2, 10.1, and 10.4. The selection rule chooses the first setting because

\min(9.2,\ 10.1,\ 10.4)=9.2

The selected setting’s tuning-fold estimate is 9.2, which is 0.8 squared trait units below its true MSE of 10. That difference is not evidence that the model truly has MSE 9.2; in this hypothetical example, all three candidates have true MSE 10. Suppose the selected model’s MSE on an independent outer test fold is 10.3. The two results are consistent with the point: the tuning estimate happened to be favorable, while the independent test result did not reuse that favorable noise. The particular outer result, 10.3, is itself subject to sampling variation.

If the value 9.2 were reported as the final performance estimate, the same data would have both selected the setting and evaluated it. Nested cross-validation instead uses inner-fold results to select the setting and outer-fold results to estimate its performance.

Combining Outer-Fold Results

After the procedure has been repeated for every outer fold, the outer test predictions cover the data once. If all outer folds have the same size, the average of their fold MSEs equals the MSE calculated over all outer predictions. With unequal fold sizes, a pooled MSE weights each fold by its number of test observations:

\mathrm{MSE}_{\mathrm{nested}}= \frac{\sum_{k=1}^{K} n_k\,\mathrm{MSE}_k}{\sum_{k=1}^{K} n_k}

Here \(n_k\) is the number of observations in outer test fold \(k\), and \(\mathrm{MSE}_k\) is that fold’s MSE. This is the total squared error across all outer predictions divided by the total number of predictions. If all \(n_k\) are equal, it reduces to the unweighted mean of the fold MSEs.

Worked Example: Calculating a Nested Cross-Validation Estimate

A hypothetical study has 100 unrelated individuals and uses five equal outer folds of 20 individuals. In each iteration, inner cross-validation selects a ridge penalty using only that iteration’s 80-person outer training set. The resulting outer-fold MSEs, in squared trait units, are 8.4, 9.1, 10.0, 8.9, and 9.6.

Because the folds are equal in size, the pooled nested estimate is the mean of the five fold MSEs:

\mathrm{MSE}_{\mathrm{nested}} =\frac{8.4+9.1+10.0+8.9+9.6}{5} =\frac{46.0}{5} =9.2

The nested cross-validation estimate is therefore 9.2 squared trait units for this procedure under this fold design. The selected penalty need not be the same in all five iterations: each outer training set can favor a different setting. That variation is part of evaluating a procedure that tunes its hyperparameter rather than evaluating one fixed penalty.

What the Estimate Does—and Does Not—Represent

The nested estimate assesses a learning procedure: given a training sample, use inner cross-validation to select a setting, refit the selected model on that training sample, and predict new individuals. It is not simply an estimate for the single hyperparameter that wins after tuning on all available data. Different outer training sets may select different settings, and the outer results summarize the resulting procedure.

There is also a training-size distinction. With \(K\) equal outer folds, each outer model is trained on about \(n(K-1)/K\) individuals, not all \(n\). The estimate therefore reflects performance for the procedure trained at roughly that sample size. After evaluation, a final model intended for use can be tuned using all available development data and refitted; its exact realized performance is not directly observed by that refit. The nested estimate remains an evaluation of the specified procedure, not a guarantee about any one fitted model.

The performance measure and selection rule should match the scientific prediction task. If the task uses another loss or score, that measure must be defined and used consistently for inner selection and outer evaluation, unless the analysis deliberately specifies different criteria. In all cases, the outer results must remain separate from the choices being evaluated.

Common Mistakes

Reporting the best inner-fold result as final performance. This uses the same noisy results to choose and evaluate a setting. The correction is to report outer-fold performance for nested evaluation.
Tuning once before creating outer folds. If the full dataset determines the selected setting, outer test outcomes have indirectly influenced the model-selection decision. Tune separately within each outer training set.
Assuming every outer fold must choose the same setting. Fold-specific selections are expected when training samples differ. The nested estimate evaluates the tuning procedure, not a requirement that one setting always win.
Interpreting the estimate as the guaranteed performance of the final refit. Cross-validation estimates performance under a defined data-splitting and training-size scheme; it does not guarantee an individual model’s future error.

Key takeaway: Keep selection and evaluation separate. Inner folds choose the hyperparameter; outer folds estimate the performance of the full tuning-and-fitting procedure. Reusing tuning folds for final evaluation makes the reported selected performance too favorable on average because the selection favors low validation noise.

Check Your Understanding

Try these questions without looking back.

In one outer iteration, which observations may be used to select a hyperparameter, and which observations estimate performance?
Why can the minimum of several unbiased validation-risk estimates be optimistically low?
Five equal outer folds have MSEs of 7, 8, 9, 10, and 11 squared trait units. What is their pooled nested cross-validation MSE?
Why might different outer folds select different hyperparameter values without indicating that the procedure was applied incorrectly?
What training-sample-size limitation should be kept in mind when interpreting a nested cross-validation estimate?