Why Tuning Can Make Performance Look Better Than It Is
A prediction pipeline can have many settings that affect its performance: a regularization strength, the number of latent components, or the relative contribution of different omics blocks. Choosing settings is itself a data-dependent operation. If the same observations both determine which setting looks best and provide the reported performance estimate, the estimate tends to be too favorable.
The concern is not limited to an obvious choice such as selecting the model with the lowest validation error. Integration decisions are also tuning decisions when they are selected based on predictive performance. For example, choosing early rather than late integration, selecting a block-weighting scheme, or choosing how many components to retain can all exploit random features of the data used to compare them.
The candidate configuration should describe the pipeline being compared, not just its final regression or classification model. If each candidate uses different preprocessing, feature learning, modality handling, or integration, those steps must be fitted using only the observations available for that candidate's training split. The earlier material on fold-local preprocessing and feature selection gives the relevant safeguards for those individual steps.
Where the Selection Must Occur
In nested cross-validation, the outer split separates model development from performance evaluation. Within each outer training set, inner resampling compares candidate configurations. The selected configuration is then fitted on the full outer training set, and its predictions are assessed on the outer test fold, which did not participate in tuning.
This procedure estimates the performance of an algorithm that includes tuning; it does not estimate the performance of a configuration chosen after looking at all the outer-fold results. The inner criterion and the outer evaluation criterion should be specified in advance and should correspond to the prediction task. For a loss where smaller values are better, select the candidate with the smallest inner-fold average. For a score where larger values are better, select the largest. A deterministic tie rule should also be set in advance.
The split structure must match the independent units and the intended prediction setting. For example, repeated samples from one person must not be separated across training and validation folds when the prediction target is a new person. If deployment requires prediction in a new cohort, a random split of observations may not represent that target. The earlier material on independent units and cohort-aware splits addresses these design choices. Nested resampling cannot repair a split that does not reflect the target population.
List the model and integration settings to compare, define the score or loss, and set a tie rule before inspecting outer-fold results.
Use inner folds to compare candidates. Fit every data-dependent pipeline step using only the inner training portion for that comparison.
Refit the selected pipeline using the complete outer training set, then calculate its performance on the untouched outer test fold.
Aggregate outer-fold performance according to the prespecified evaluation plan. Do not choose a new configuration using these results and then report the same results as an unbiased evaluation.
Worked Example: Selecting a Penalty Within an Outer Training Set
Suppose a hypothetical study predicts a continuous clinical response measured in response units. The loss is mean squared error (MSE), in squared response units. Within one outer training set, researchers compare three penalty strengths, \(\lambda=0.01\), \(0.1\), and \(1\), using the same four inner folds. Assume the independent units are correctly split, each candidate's preprocessing is fitted within its inner training folds, and all candidates are evaluated on the same inner validation observations.
The inner-fold MSEs are:
| Penalty strength | Inner-fold MSEs (response units squared) | Mean MSE |
|---|---|---|
| \(\lambda=0.01\) | 0.80, 0.90, 1.10, 1.00 | 0.950 |
| \(\lambda=0.1\) | 0.70, 0.80, 0.90, 0.80 | 0.800 |
| \(\lambda=1\) | 0.75, 0.85, 0.80, 0.90 | 0.825 |
For \(\lambda=0.1\), the inner-fold mean is
The other means are \((0.80+0.90+1.10+1.00)/4=0.950\) and \((0.75+0.85+0.80+0.90)/4=0.825\) response units squared. Thus \(\lambda=0.1\) is selected within this outer training set. The pipeline is then refitted on all of that outer training set using \(\lambda=0.1\). Its performance is assessed on the outer test fold, not on the inner folds used to choose it. The value 0.800 is a tuning result, not the outer-fold performance estimate.
Why the Best Observed Score Is Optimistic
The selection effect can be shown with a short argument. Imagine a fixed set of candidate pipelines with true risks \(R_1,\ldots,R_m\), where lower risk is better. Suppose their estimated risks \(\widehat{R}_j\) are unbiased for those fixed risks. The minimum observed estimate cannot exceed any individual estimate, so it cannot exceed their average either:
Taking expectations gives an upper bound for the expected selected estimate. Also, the minimum of a set of numbers is no greater than any one of them, so it is no greater than the smallest of their expectations:
Thus the expected minimum estimated risk is no larger than the smallest true risk. With noisy estimates, the inequality is often strict: some candidates will look unusually good by chance, and selecting the minimum favors those candidates. For a score where larger values are better, the same logic applies to the maximum observed score, which tends to be too high.
This argument is an idealized explanation, not a claim that cross-validation estimates are always unbiased for a single fixed risk. Cross-validation folds can have dependent estimates, and models trained on smaller inner samples can differ from models trained on the full development sample. Those features affect the precise amount of optimism. They do not remove the selection problem: choosing among noisy scores and reporting the chosen score uses the same noise twice.
Worked Example: Quantifying Selection Optimism in a Simple Case
Suppose two hypothetical candidate pipelines have the same true MSE, \(0.20\) response units squared. Each estimated MSE equals its true value plus an independent error that is \(+0.04\) or \(-0.04\), each with probability \(1/2\). Each estimate is unbiased because the error has mean \((0.04-0.04)/2=0\).
The smaller estimate is \(0.16\) unless both errors are \(+0.04\). That event has probability \((1/2)(1/2)=1/4\), in which case both estimates are \(0.24\). Therefore,
The expected selected estimate is \(0.18\), although both candidates' true MSEs are \(0.20\). An independent evaluation that is not used to select between the candidates has expected MSE \(0.20\) for either candidate in this example. The difference, \(0.02\) response units squared, illustrates selection optimism; it is not a universal correction factor.
Integration Choices Are Hyperparameters Too
Tuning is not restricted to a grid of penalty values. Candidate configurations can vary the number of components retained from each omics block, the relative block weights, or the integration strategy. If performance determines which of these is preferred, that choice belongs inside the inner tuning process. The candidate set may combine choices: for example, a block-weighting rule paired with a penalty strength and a component count.
The more candidates that are compared, the more opportunities there are to select a configuration with a favorable noise fluctuation. This does not mean a broad search is invalid. It means that the entire search must occur within each outer training set, and the outer test folds must remain outside that search. A configuration set should be scientifically and computationally defensible; extreme or redundant grids can consume substantial computation without providing useful options.
Worked Example: Evaluating a Tuned Integration Pipeline
Consider a hypothetical three-fold outer evaluation of a multi-omics model for a continuous clinical response. Within each outer training set, three penalty strengths are compared using inner folds. The table gives the mean inner MSE for each candidate, followed by the MSE on the corresponding outer test fold after refitting the selected pipeline. Assume participants are the independent units, the outer folds respect the intended prediction setting, and all preprocessing and integration steps are fitted within the relevant training splits.
| Outer fold | Inner means for \(\lambda=0.01,\ 0.1,\ 1\) (response units squared) | Selected setting | Outer-test MSE (response units squared) |
|---|---|---|---|
| 1 | 0.31, 0.24, 0.27 | \(\lambda=0.1\) | 0.22 |
| 2 | 0.26, 0.28, 0.21 | \(\lambda=1\) | 0.25 |
| 3 | 0.23, 0.20, 0.22 | \(\lambda=0.1\) | 0.19 |
The outer mean MSE is
Each outer fold can select a different setting because it tunes on a different training subset. That is not a contradiction: the object being evaluated is the procedure “tune within training data, then fit,” rather than one fixed penalty value. The mean of the three selected inner means is \((0.24+0.21+0.20)/3=0.217\) response units squared, but it is not the outer performance estimate. In particular, comparing these two averages does not by itself quantify optimism, because the inner and outer evaluations differ in both data use and fitting details.
Diagnostics, Alternatives, and Limits
A useful audit follows the information path for every choice. Ask which observations influenced preprocessing, feature selection, integration, model fitting, and the final choice among settings. If outer-fold outcomes influenced any of these steps, those folds are no longer an independent evaluation of the resulting procedure. Keep a record of candidate settings, inner results, selection rules, and outer results so that the boundary between tuning and evaluation is explicit.
Warning signs include a large gap between the best inner score and outer-fold performance, unstable selected settings across outer folds, or unusually strong performance from a candidate that was chosen after many informal comparisons. These are prompts to inspect the analysis, not automatic proof of leakage: genuine sampling variability, different training-set sizes, and population differences can also produce variation. Non-convergence, inconsistent preprocessing, or different validation observations across candidates can make comparisons hard to interpret and should be investigated.
The main alternatives have different uses and limitations:
| Approach | What it can do | Main limitation for tuning |
|---|---|---|
| Choose and report the best cross-validation score | Compare candidates during model development. | The reported best score is subject to selection optimism because it was used to choose the candidate. |
| Single training/validation split | Separate tuning from an evaluation split if the validation data are not used in selection. | Results can depend strongly on one split; using that split repeatedly to revise choices makes it part of development. |
| Nested cross-validation | Evaluate the complete tuning procedure through repeated outer training and evaluation splits. | It can be computationally expensive, and its estimate reflects the split design and outer training-set size. |
| Fixed configuration chosen in advance | Avoid data-driven selection among configurations for the evaluation. | It may not be the best-performing choice, and it still requires a valid evaluation design. |
Nested cross-validation does not guarantee future performance, correct a mismatch between study and deployment populations, or make a poorly specified candidate grid scientifically appropriate. Its value is narrower and important: the outer observations do not determine the configuration evaluated on them. After evaluation, a final configuration can be selected using the available development data and the chosen pipeline refitted, but the outer-fold estimate remains an estimate for the defined tuning procedure, not a score that should be repeatedly optimized.
Common Mistakes
- Choosing the configuration with the best outer-fold result. That choice uses evaluation outcomes for tuning. The resulting outer scores no longer evaluate the newly selected configuration independently.
- Tuning only the model penalty while selecting integration from the same evaluation results. Integration choices are data-driven settings too, so they must be included in the tuning boundary.
- Calling the best inner score an unbiased performance estimate. It is useful for selecting settings, but selection favors low observed loss. Use the outer evaluation to estimate the tuning procedure.
- Assuming nested cross-validation fixes every leakage problem. It does not help if samples from the same independent unit cross folds, or if data-dependent processing is fitted before the folds are formed.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Why is the smallest inner-fold MSE useful for selection but not, by itself, a clean estimate of the selected pipeline's performance?
- A researcher compares three block-weighting rules using outer-fold scores and then reports the score for the best rule. What is the problem?
- In one outer training set, two candidates have inner mean MSEs of 0.34 and 0.29 response units squared. Which is selected if lower MSE is better, and what must happen before outer evaluation?
- Name two reasons that nested cross-validation may still give an estimate that does not represent the intended deployment performance.
Solutions. (1) The candidate was selected because its noisy observed loss was smallest, so its score has benefited from selection; outer-fold evaluation is needed to assess the tuning procedure. (2) The outer scores have been used for tuning, so reporting the best one reuses evaluation information and is optimistic on average. (3) Select the candidate with MSE 0.29 response units squared, refit that pipeline using the complete outer training set, and evaluate it on the untouched outer test fold. (4) Examples include a split that does not respect the independent units or deployment population, and a mismatch between the study population and the population where predictions will be used.