Why Derived-Feature Choices Need Resampling
A clustering or principal component analysis (PCA) step can change a prediction model substantially. The number of clusters determines how finely the data are partitioned; the number of retained components determines how much of the predictor representation is carried into the model. These are choices about the prediction procedure, not merely descriptive settings. If the same validation results are used both to choose among many choices and to report the chosen model's performance, the reported value can be too favorable.
Resampling addresses this problem by separating two tasks: selection, which compares feature configurations, and evaluation, which assesses the selection procedure on data not used to make that choice. For example, inner cross-validation can choose a cluster count or a number of PCA scores, while outer cross-validation evaluates the full process of making that choice. A reserved test set can serve as a separate final evaluation set instead.
For a candidate configuration, a complete prediction procedure includes learning the feature transformation on a training portion, applying that fitted transformation to its validation portion, and fitting and using the prediction model consistently with those features. The earlier material on assigning new records to existing clusters and transforming validation and test records gives the operational principle: a validation record receives features from the transformation learned on its corresponding training data. Resampling repeats that discipline at every split.
Selection and Evaluation Are Different Tasks
Suppose the candidate choices include \(k=2,3,4\) clusters and retaining \(m=1,2,3\) principal components. These choices can be compared by a prediction metric, such as root mean squared error (RMSE) for a quantitative outcome. For validation observations with actual values \(y_j\) and predictions \(\hat{y}_j\), RMSE is the square root of the average squared prediction error. Lower RMSE is preferred when the objective is to predict the outcome on its stated scale.
In \(K\)-fold cross-validation, the available selection data are divided into \(K\) folds. Each fold is held out once, while the other \(K-1\) folds form the training portion. For each candidate, compute a validation score on each fold and summarize those scores according to a prespecified rule. With equal-sized folds, the simple mean of the fold RMSEs is a possible selection score. If fold sizes differ, pooled squared error gives each validation observation equal weight; averaging fold RMSEs instead gives each fold equal weight.
A sound design therefore keeps selection separate from evaluation. In nested cross-validation, each outer split sets aside an outer validation fold. The remaining outer-training data are used for all inner tuning. After selecting a configuration within that outer-training data, fit the selected procedure on those data and score it on the untouched outer fold. Repeat for each outer fold. The outer-fold results assess the tuning-and-fitting procedure, including the fact that it selects among candidates.
A Nested Resampling Workflow
Specify the derived-feature choices, prediction-model choices if relevant, and metric before comparing results. State how ties will be handled.
Set one outer fold aside for evaluation. Do not use its outcomes to choose a feature configuration or tune the prediction model.
Use inner folds formed only from the outer-training data. In every inner split, learn the derived-feature mapping on the inner-training portion and apply it to the inner-validation portion.
Refit the selected configuration using the outer-training data, then predict the outer validation observations and calculate the prespecified metric.
Report the outer evaluation results using a stated aggregation rule. Do not choose a configuration by looking at which outer-fold result is most favorable.
The inner and outer folds play different roles even though both are resampling splits. The inner results determine a choice. The outer results estimate how well the procedure performs when that choice is made without access to the observations being evaluated. Outer evaluation does not guarantee that the estimate equals future performance: its usefulness depends on the data being representative of the intended prediction setting and on the split design matching that setting.
Worked Example: Tuning Cluster Count Within Outer Folds
A hypothetical insurer models the number of service requests made by a policyholder. The outcome is quantitative, and lower RMSE is preferred. Three candidate procedures use \(k=2\), \(k=3\), or \(k=4\) clusters as predictors. In each of three outer folds, a three-fold inner cross-validation compares the candidates. Assume the inner validation folds are equal in size. The inner-fold RMSEs and the corresponding outer-fold RMSEs for the selected procedures are shown below.
| Outer fold | Inner RMSEs for \(k=2\) | Inner RMSEs for \(k=3\) | Inner RMSEs for \(k=4\) | Selected \(k\) | Outer RMSE |
|---|---|---|---|---|---|
| 1 | 5.4, 5.2, 5.5 | 5.1, 5.0, 5.2 | 5.0, 5.2, 5.1 | 4 | 4.8 |
| 2 | 5.2, 5.1, 5.3 | 5.1, 5.2, 5.0 | 4.9, 5.0, 5.0 | 4 | 5.1 |
| 3 | 5.5, 5.4, 5.6 | 5.2, 5.1, 5.3 | 5.1, 5.0, 5.2 | 4 | 4.9 |
Method: inner selection followed by outer evaluation. In outer fold 1, the mean inner RMSE for \(k=2\) is \((5.4+5.2+5.5)/3=16.1/3=5.3667\). For \(k=3\), it is \((5.1+5.0+5.2)/3=15.3/3=5.1000\). For \(k=4\), it is \((5.0+5.2+5.1)/3=15.3/3=5.1000\). The prespecified tie rule selects the larger cluster count, so \(k=4\) is selected.
For outer fold 2, the three means are \(15.6/3=5.2000\) for \(k=2\), \(15.3/3=5.1000\) for \(k=3\), and \(14.9/3=4.9667\) for \(k=4\). Thus \(k=4\) is selected. For outer fold 3, the means are \(16.5/3=5.5000\), \(15.6/3=5.2000\), and \(15.3/3=5.1000\), respectively, so \(k=4\) is selected again.
The outer RMSEs are \(4.8\), \(5.1\), and \(4.9\). With equal-sized outer folds, their mean is \((4.8+5.1+4.9)/3=14.8/3=4.9333\). The outer estimate is therefore about \(4.93\) requests in RMSE units. The inner scores guided the selection; the outer scores evaluated the selected procedure. A common mistake would be to report the best inner score, \(4.9667\), as the performance estimate. It is not the outer evaluation result and answers a different question.
Retained Components and Tie Rules
The same design applies when the tuning choice is the number of retained PCA scores. Candidate values of \(m\) should be specified in advance, and the preprocessing and PCA mapping should be learned within each training portion. The validation portion is transformed using that fitted mapping. A selection rule should also specify how to handle ties or near-ties, rather than choosing after inspecting the evaluation results.
One simple rule is to choose the candidate with the lowest mean inner score and, in an exact tie, choose the candidate with fewer retained components. This favors a more compact representation when the measured scores are identical. Such a rule is a practical decision convention, not proof that the compact candidate will always predict better.
Worked Example: Choosing the Number of PCA Scores
A hypothetical equipment-maintenance team uses PCA scores derived from sensor measurements to predict hours until a machine requires service. Five equal-sized inner validation folds produce these RMSEs for four candidate values of \(m\):
| Retained scores \(m\) | Fold RMSEs | Mean inner RMSE |
|---|---|---|
| 1 | 7, 6, 7, 6, 6 | 6.4 |
| 2 | 5, 6, 5, 5, 4 | 5.0 |
| 3 | 4, 7, 5, 5, 4 | 5.0 |
| 4 | 4, 8, 5, 6, 4 | 5.4 |
Method: minimize mean inner RMSE, then apply the tie rule. For \(m=1\), the sum is \(7+6+7+6+6=32\), so the mean is \(32/5=6.4\). For \(m=2\), the sum is \(5+6+5+5+4=25\), giving \(25/5=5.0\). For \(m=3\), the sum is \(4+7+5+5+4=25\), also giving \(25/5=5.0\). For \(m=4\), the sum is \(4+8+5+6+4=27\), giving \(27/5=5.4\).
The minimum mean is \(5.0\), shared by \(m=2\) and \(m=3\). Under the stated tie rule, choose \(m=2\), which retains fewer scores. If a separate outer fold then gives RMSE \(5.3\) for the selected procedure, that outer score contributes to evaluation; it must not be used to revisit the choice between \(m=2\) and \(m=3\). A common mistake is to select \(m=3\) because one of its individual fold scores, \(4\), is the smallest score in the table. Tuning is based on the prespecified summary across folds, not the most favorable single fold.
Using a Reserved Test Set for Final Evaluation
Nested cross-validation is one way to separate selection and evaluation. Another is to keep a test set untouched while all candidate comparisons are carried out on development data. After selecting a configuration through resampling on the development data, fit the chosen procedure using all development data and evaluate it once on the test set. The test set should not be consulted to revise the number of clusters, retained components, or any other modeling choice.
If the test set is used to choose among several candidates, it has become selection data. Its resulting score no longer has the same role as a final evaluation of a procedure selected independently of it. This distinction matters even if the feature construction itself does not use the outcome: the prediction scores used to choose the configuration do use observed outcomes.
Worked Example: One-Time Test Evaluation After Tuning
A hypothetical benefits administrator uses development data to compare cluster and PCA configurations by cross-validation. The chosen configuration is a model using two PCA scores. Its inner cross-validation mean RMSE is \(5.0\), the lowest among the candidates. The administrator now fits that selected procedure on all development data and evaluates it on a reserved test set of 30 cases. The 30 test residuals consist of 15 values equal to \(+5\) hours and 15 equal to \(-5\) hours.
Method: calculate test RMSE from test residuals. Squaring either \(+5\) or \(-5\) gives \(25\). There are 15 residuals of each kind, so the total squared error is \(15(25)+15(25)=375+375=750\). The test mean squared error is \(750/30=25\), and the test RMSE is \(\sqrt{25}=5\) hours.
The inner mean RMSE of \(5.0\) selected the configuration; the test RMSE of \(5.0\) evaluates the resulting fitted procedure on the reserved cases. Their numerical equality does not make them interchangeable: they were obtained from different data with different roles. The common mistake is to compare several configurations on this test set and then report the lowest test RMSE as though it were an untouched evaluation. The correction is to keep the test set out of selection and use it for the final assessment only.
Practical Checks and Common Traps
Before interpreting a tuned model's reported performance, verify that the split design matches the intended use and that the selection procedure was followed consistently. A correct score calculation cannot fix a design in which evaluation outcomes influenced the feature choice.
- Reporting the minimum inner score as final performance. The inner score was used to select among candidates. Use outer-fold scores or a genuinely reserved test set for evaluation.
- Tuning separately on each outer validation fold. Outer-fold outcomes must remain unavailable while the candidate is selected. Tune within the corresponding outer-training data.
- Changing the candidate grid after seeing evaluation results. Doing so uses evaluation information to make a modeling choice. If the procedure changes, a new independent evaluation is needed for a clean final assessment.
- Ignoring fold sizes. Mean fold RMSE weights folds equally. When fold sizes differ and each observation should have equal influence, aggregate squared errors over all validation observations before taking the square root.
- Choosing a favorable fold or metric after the fact. Prespecify the summary metric and aggregation rule; do not select whichever result makes the model look best.
For \(K\) outer folds with unequal sizes \(n_r\), let \(\operatorname{SSE}_r\) be the sum of squared errors on outer fold \(r\). The pooled outer RMSE weights each evaluated observation equally:
This follows directly from the definition of RMSE: the numerator sums every squared prediction error across all outer validation observations, and the denominator is their total count. If all outer folds have equal size, this pooled calculation is also the square root of the mean of their fold mean squared errors. It is generally not identical to the arithmetic mean of fold RMSEs, because taking a square root before averaging changes the calculation.
Check Your Understanding
For each question, distinguish clearly between the data used to select a configuration and the data used to evaluate it.
- A candidate with \(k=3\) clusters has inner-fold RMSEs \(6.0, 5.0, 5.5\); a candidate with \(k=5\) has \(5.0, 5.5, 5.0\). With equal fold sizes, which has the smaller mean inner RMSE? (Answer: \(k=5\), with mean \(5.1667\) versus \(5.5000\).)
- Four equal-sized outer folds have RMSEs \(4.2, 4.8, 5.0, 4.4\). What is their mean outer-fold RMSE? (Answer: \((4.2+4.8+5.0+4.4)/4=4.6000\).)
- Two PCA candidates have mean inner RMSE \(4.7\): one retains two scores and one retains three. The prespecified rule selects fewer scores in a tie. Which is selected, and should an outer fold be used to break the tie? (Answer: Select two scores; no, the outer fold is reserved for evaluation.)
- Two outer folds have sizes 10 and 20, with squared-error sums 100 and 320. Calculate pooled outer RMSE. (Answer: \(\sqrt{(100+320)/(10+20)}=\sqrt{14}=3.7417\).)
- A team selects the number of clusters using cross-validation, then compares three selected configurations on its reserved test set and reports the lowest test RMSE. Is that test result still an untouched final evaluation? (Answer: No; the test set has been used for selection, so the reported minimum can be optimistic.)