Why Model Comparison Needs Two Layers
A model comparison often involves more than choosing between fitted algorithms. The analyst may also choose a tuning parameter, such as the number of neighbors in a k-nearest neighbors (kNN) model, or a tree-complexity setting. If the same validation results are used both to make those choices and to claim how well the chosen procedure predicts new observations, the reported performance tends to be too favorable. The comparison has adapted to the particular validation folds.
Nested cross-validation separates these tasks. The inner folds select a model and its tuning settings using only an outer training set. The outer fold, which was not used for those choices, evaluates the selected procedure. Repeating this process across outer folds estimates the performance of the process that includes model selection—not just the performance of a model chosen in advance.
Suppose there are \(n\) observations and the outcome is \(Y\). For classification, one possible loss is the indicator of an incorrect class prediction, \(L(Y,\hat{Y})=I(Y\ne\hat{Y})\). For regression, a common loss is squared error, \(L(Y,\hat{Y})=(Y-\hat{Y})^2\). Cross-validation averages a chosen loss over held-out observations. The metric and loss should be selected for the prediction task; the nested structure does not itself choose them.
The Nested Procedure
Divide the observations into \(K\) outer folds. In turn, hold out each fold as an outer test set and use the remaining observations as that iteration’s outer training set. Within that outer training set, run inner cross-validation for every candidate in the comparison. Select the candidate according to the prespecified inner score, refit it on all outer training observations, and predict the outer test observations.
Set aside one outer fold. Its outcomes must not affect preprocessing choices, candidate selection, tuning, or fitting in this iteration.
Divide the outer training observations into inner folds. Fit each candidate on the inner training portions and score its predictions on the corresponding inner validation portions.
Choose the candidate using the prespecified inner criterion, then fit that candidate to the entire outer training set.
Predict the outer test observations and calculate the chosen loss. Repeat for each outer fold.
Combine the held-out losses to summarize the expected predictive performance of the selection procedure.
For equal-sized outer folds, the mean of the outer-fold losses equals the average loss over all outer-held-out observations. If fold sizes differ, an observation-weighted average is appropriate: multiply each fold’s mean loss by its test-set size, add those totals, and divide by the total number of outer test observations. For classification error, this is the total number of incorrect outer predictions divided by the total number of outer-held-out observations.
The outer folds estimate performance for the entire procedure: trying the specified candidates, selecting one using inner folds, and refitting it. They do not estimate the performance of one universal fixed model if different outer iterations select different candidates. After evaluation, a final deployable model can be selected and fitted using all available development data, but its fitted performance should not be reported as an outer test result.
Worked Examples
Worked Example: Nested Classification Comparison
A hypothetical insurer has 12 independent policy records and wants to compare a kNN classifier, a depth-two classification tree, and a random forest with a prespecified setting. Use three outer folds of four records each. Within each outer training set of eight records, use two inner validation folds of four records each. The table gives the numbers of misclassified records in the two inner validation folds for the first outer iteration. Select the candidate with the lowest mean inner misclassification rate.
| Candidate | Errors in inner fold 1 (of 4) | Errors in inner fold 2 (of 4) | Mean inner error rate |
|---|---|---|---|
| kNN | 1 | 2 | 0.375 |
| Depth-two tree | 2 | 1 | 0.375 |
| Random forest | 1 | 1 | 0.250 |
Method and inner selection. Each inner fold has four observations, so its error rate is its error count divided by four. For kNN, the two rates are \(1/4=0.25\) and \(2/4=0.50\), giving
For the tree, the rates are \(2/4=0.50\) and \(1/4=0.25\), also averaging \(0.375\). For the random forest, each rate is \(1/4=0.25\), so its mean is \(0.250\). The random forest is selected for this outer iteration.
Outer evaluation. Refit the selected random forest on all eight outer training records, then predict the four records in the outer test fold. Suppose it makes one error. Its outer error rate is \(1/4=0.25\). Repeat the complete inner selection and outer evaluation for the other two outer folds. Suppose their held-out error counts are zero and two, respectively. The pooled outer error rate is
The estimated error rate for the full selection procedure is therefore \(0.25\), or 25%. The inner rates in the table are all valid for folds of four: each is an integer multiple of \(1/4\). Common mistake avoided: treating an inner score as the final performance estimate would skip the outer evaluation that accounts for model selection.
Worked Example: Why the Best Inner Score Is Not the Performance Estimate
A hypothetical risk team uses four inner validation folds of 20 policies each to compare three classifiers. The total inner validation count for each candidate is 80. The error counts across those folds are: kNN, 2, 4, 3, and 3; a tree, 4, 3, 4, and 5; and a random forest, 4, 4, 3, and 5. The kNN model is selected. On a separate outer test fold of 20 policies, the selected-and-refitted model makes five errors.
Method and inner calculation. For kNN, the total number of inner errors is \(2+4+3+3=12\), so its mean inner error rate over the 80 validation predictions is \(12/80=0.15\). The tree has \(4+3+4+5=16\) errors, giving \(16/80=0.20\). The random forest also has \(4+4+3+5=16\) errors, giving \(16/80=0.20\). Thus, kNN is selected by the inner results.
The outer test error rate is \(5/20=0.25\). In this particular split, the selected candidate’s inner rate is 0.15 and its outer rate is 0.25. These values illustrate why the inner minimum should not be presented as an unbiased estimate of the selected procedure’s future error. One split need not show this direction of difference; the concern is the selection process across repeated samples.
Why selection tends to make the reused score optimistic. Let \(\hat{R}_j\) be the validation estimate for candidate \(j\), and let \(R_j\) be its underlying expected risk. If the same validation results select the candidate with the smallest estimate and are then reported, the reported quantity is \(\min_j\hat{R}_j\). Since \(\min_j\hat{R}_j\leq\hat{R}_k\) for every candidate \(k\), taking expectations gives \(E[\min_j\hat{R}_j]\leq E[\hat{R}_k]\). If each estimate is unbiased, \(E[\hat{R}_k]=R_k\), so the expected minimum is no greater than the risk of any one candidate, and thus no greater than \(\min_j R_j\). When random validation fluctuations affect which candidate wins, the inequality is often strict. The outer fold avoids reporting that selected inner minimum as the evaluation: it assesses the choice using observations not used to choose it.
Common mistake avoided: calling \(0.15\) the final estimate because it is the lowest inner rate. It is a tuning result. The outer error rate \(0.25\) is the held-out assessment for this iteration; repeated outer folds are needed for the nested estimate.
Worked Example: Nested Comparison Using Squared Error
A hypothetical insurer predicts a continuous claim-cost measure using kNN regression, a regression tree, or a random forest. Within one outer training set, three inner validation folds each contain two policies. The sums of squared prediction errors (SSE) are as follows: kNN has SSE values 8, 2, and 5; the tree has 4, 4, and 4; and the random forest has 2, 4, and 4. Select by mean inner mean squared error (MSE). Then evaluate the selected model on an outer test fold of two policies where actual outcomes are 10 and 14 and its predictions are 11 and 12.
Method and inner calculation. Each fold contains two observations, so its MSE is its SSE divided by two. For kNN, the fold MSEs are \(8/2=4\), \(2/2=1\), and \(5/2=2.5\). Their mean is \((4+1+2.5)/3=2.5\). For the tree, each fold MSE is \(4/2=2\), so its mean is 2. For the random forest, the fold MSEs are \(2/2=1\), \(4/2=2\), and \(4/2=2\), with mean \((1+2+2)/3=5/3\), or approximately 1.667. The random forest is selected.
Outer calculation. The two outer residuals, defined as actual minus predicted, are \(10-11=-1\) and \(14-12=2\). Their squared errors are \((-1)^2=1\) and \(2^2=4\). The outer MSE is \((1+4)/2=2.5\) squared units. The selected model’s inner MSE is approximately 1.667, while this outer fold’s MSE is 2.5; neither number changes the other’s role. Common mistake avoided: averaging fold SSEs without dividing by the number of observations would compare totals rather than mean squared errors.
Conditions and Practical Safeguards
The fold design must reflect how predictions will be used. Randomly assigning independent observations to folds is reasonable when observations are independent and drawn from the same target population. If multiple records belong to one person, policy, or household, splitting related records between training and validation can let information about that unit appear on both sides. Keep such groups together when the intended prediction is for new groups. For time-ordered prediction, preserve the temporal order rather than allowing future observations to train predictions for the past.
Every data-dependent operation belongs inside the appropriate training portion. For example, if predictors are standardized, estimate the means and standard deviations from the inner training fold and apply those values to its validation fold. Within each outer iteration, all tuning—including choices about transformations, predictors, or settings—must use only the outer training data. Then refit the resulting pipeline on that full outer training set before scoring its outer test fold. This prevents information from held-out outcomes or predictor distributions from influencing the evaluated procedure.
The candidate set, loss, fold plan, and tie-breaking rule should be specified before examining outer results. If multiple candidates have identical inner scores, a predetermined rule—for example, preferring a simpler prespecified candidate—makes the selection reproducible. Repeatedly changing the candidate set after inspecting outer results also adapts to those results; the original outer estimate then no longer evaluates the revised process independently.
Check Your Understanding
For each problem, identify the tuning calculation or evaluation step requested before checking the answer.
- Three candidates are compared on three inner folds of 10 observations. Their error counts are: kNN, 2, 3, 4; a tree, 3, 3, 4; and a forest, 2, 2, 3. Calculate each candidate’s pooled inner error rate and identify the selection. (Answer: kNN \(9/30=0.3000\), tree \(10/30=0.3333\), forest \(7/30=0.2333\); select the forest.)
- Four equal-sized outer test folds each contain eight observations. The selected procedure makes 2, 1, 3, and 2 errors, respectively. Calculate the pooled outer error rate. (Answer: \(8/32=0.2500\).)
- Two regression candidates are assessed on two inner folds of three observations each. Candidate A has SSE values 12 and 18; candidate B has SSE values 9 and 15. Calculate each mean inner MSE and identify the selection. If the selected candidate has SSE 14 on an outer fold of four observations, calculate that outer MSE. (Answer: A has mean MSE \((12/3+18/3)/2=5\); B has mean MSE \((9/3+15/3)/2=4\), so select B; outer MSE \(14/4=3.5\).)
- An analyst tries 12 settings using the same validation folds, selects the setting with the lowest error, and reports that lowest error as final performance. Explain the problem and name the corrective evaluation structure. (Answer: the reported minimum is subject to selection optimism because the folds both selected and evaluated the winner; use inner folds for selection and separate outer folds for evaluation.)
- A data set has 100 independent observations. Use five equal outer folds and, within each outer training set, four equal inner validation folds. State the outer test size, outer training size, and inner validation size. (Answer: each outer test fold has 20 observations; each outer training set has 80; each inner validation fold has 20.)