Why Compare the Inner and Outer Scores?
Hyperparameter tuning selects the configuration that scores best on inner validation data. Because that score helped choose the configuration, it can be optimistic as an estimate of performance on new data. Nested cross-validation provides a separate outer assessment: for each outer split, the selected pipeline is evaluated on data that were not used to choose its hyperparameters. Comparing these two scores can reveal a pattern consistent with selection optimism.
The comparison is meaningful only when the scores are matched: the inner score and outer score must refer to the same outer split, the same prediction target, and the same performance metric. The inner score summarizes the selected configuration’s performance in that split’s inner validation process; the outer score assesses the refitted selected pipeline on that split’s held-out outer fold. They are related, but not identical measurements.
The score direction must be made explicit before interpreting a gap. Accuracy, area under the ROC curve, and many correlation-based scores are usually interpreted as higher-is-better. Log loss and mean squared error are lower-is-better. One can multiply a loss by \(-1\) to use a higher-is-better convention, but the transformation and units should be stated. A difference of 0.04 in accuracy is four percentage points, not a four percent relative increase.
Conditions for a Useful Diagnostic
This is a descriptive diagnostic within a correctly implemented nested validation design, not a stand-alone test with a universal reference distribution. Before comparing scores, check that the design supports the comparison:
- Hyperparameter selection, feature selection, scaling, imputation, and other learned steps use only data available within the relevant training split. The outer-fold outcomes must not influence any selection.
- The inner and outer validation designs target compatible prediction tasks and populations. For example, if the target is prediction for new sites, both levels should respect the site grouping relevant to that target.
- The inner and outer scores use the same metric, the same direction, and compatible aggregation rules. Comparing a mean of inner-fold AUCs with outer-fold accuracy does not yield an interpretable gap.
- The outer score corresponds to the configuration selected in that outer split, refitted using its outer training data. It is not the outer score of a configuration selected using all observations.
- Fold sizes and training-set sizes are recorded. The inner models are trained on subsets of the outer training data, while the outer model is trained on the full outer training portion. That difference can affect scores independently of selection.
Even when these conditions hold, inner and outer scores differ because they come from different validation observations and often different training sizes. Differences in case mix, class prevalence, genomic cohort composition, and finite-sample variation can all contribute. The gap is evidence to inspect, not a pure measurement of selection bias.
Worked Example: Summarizing Matched Gaps Across Repeats
Suppose a hypothetical nested cross-validation analysis uses accuracy, for which larger is better. In each outer split, the analyst records the best inner accuracy and the corresponding outer accuracy. The six matched pairs, spanning three repeats, are:
| Repeat | Outer split | Best inner accuracy | Outer accuracy | Gap \(I_j-O_j\) |
|---|---|---|---|---|
| 1 | 1 | 0.78 | 0.74 | 0.04 |
| 1 | 2 | 0.81 | 0.77 | 0.04 |
| 2 | 1 | 0.76 | 0.72 | 0.04 |
| 2 | 2 | 0.79 | 0.80 | -0.01 |
| 3 | 1 | 0.83 | 0.78 | 0.05 |
| 3 | 2 | 0.77 | 0.75 | 0.02 |
The mean inner score is
The mean outer score is
Thus, the mean matched gap is \(0.790-0.760=0.030\), or 3.0 percentage points of accuracy. The six gaps also average to \((0.04+0.04+0.04-0.01+0.05+0.02)/6=0.030\). The appropriate interpretation is that, in these hypothetical splits, the best inner accuracy was higher on average than the corresponding outer accuracy by 3.0 percentage points. This pattern is consistent with selection optimism, but it does not establish that selection alone caused the difference or that the same gap would apply to a future analysis.
Why Selection Can Make the Best Inner Score Optimistic
A simple model explains the direction of the selection effect. Suppose, just for this justification, that \(H\) candidate configurations have the same underlying score \(\theta\) for the target evaluation, and their inner scores are \(I_h=\theta+\varepsilon_h\), where each \(\varepsilon_h\) has mean zero. The selected inner score is the maximum, \(\max_h I_h\). Since the maximum is a convex function, Jensen’s inequality gives
Therefore, under this simplified equal-performance setup, the expected noise component of the winning inner score is nonnegative. With variable, non-identical inner errors it may be strictly positive. The process has selected not only a configuration but also, partly, one that happened to receive a favorable validation fluctuation.
This argument explains a tendency; it does not produce a correction for an observed gap. Real configurations can have different underlying performance, inner estimates can be dependent, and outer scores have their own sampling variation. Moreover, the outer model is trained on a different number of observations from the inner-fold models. Thus the observed \(I_j-O_j\) combines selection effects with other differences in estimation and evaluation.
Worked Example: Seeing the Winner’s-Noise Mechanism
Consider a hypothetical illustration with three configurations that have the same target accuracy, 0.75. Their observed inner accuracies are 0.84, 0.79, and 0.77. The selected configuration is the first, because 0.84 is the largest inner score. Its inner score can be written as \(0.75+0.09=0.84\); the other two observed deviations are \(0.04\) and \(0.02\).
Suppose the selected configuration’s outer accuracy is 0.76. The matched gap is \(0.84-0.76=0.08\), or 8 percentage points. The outer score is close to the stipulated target accuracy in this illustration, while the winning inner score is 9 points above it. This illustrates how selecting the largest noisy score can create an inner score that looks unusually strong.
The numbers do not imply that a real 8-point gap is all selection optimism. In practice the target accuracy is unknown, and an outer score can also be high or low by chance. The example isolates the mechanism only; it does not supply a formula for subtracting 0.08 from future inner scores.
Reading Patterns Across Splits
Inspect both the overall average gap and the individual matched gaps. A consistently positive gap can flag a potentially consequential winner’s effect, especially when many configurations were searched or inner scores vary substantially across folds. A near-zero average does not prove that tuning was harmless: positive and negative split-specific gaps can cancel. A negative average is possible too; it means outer scores were higher on average in the observed splits, not that selection optimism is impossible.
Repeated cross-validation reuses observations across splits. Consequently, the split-specific gaps should not be treated as independent observations for a routine standard error or significance test. For this diagnostic, report the score definition, the paired inner and outer summaries, the number of splits and repeats, and the spread or range of the gaps as descriptive information. Formal uncertainty quantification requires methods and assumptions beyond the gap itself.
A useful review can proceed in a consistent order:
For each outer split, identify the inner-selected configuration and the exact inner score used to select it.
Pair that inner result with the outer score for the pipeline selected in the same split, using the same metric and direction.
Compute one difference per outer split, then describe its mean and split-to-split pattern across repeats.
Check for leakage, mismatched fold designs, inconsistent aggregation, small or variable validation folds, and differences in training size before attributing the gap to selection.
Worked Example: Interpreting a Gap for a Loss Metric
A hypothetical tuning analysis records log loss, where smaller is better. For one outer split, the best inner log loss is 0.42 and the corresponding outer log loss is 0.48. The deterioration-direction gap is
The selected configuration’s outer log loss is 0.06 higher than its best inner log loss, so performance is worse on the outer fold by 0.06 log-loss units. This is directionally consistent with an optimistic inner result. Before interpreting it, verify that both values are log loss calculated on comparable validation data and that the inner score was aggregated in the same way as the outer score.
If the analyst instead calculated inner minus outer, the result would be \(0.42-0.48=-0.06\). That arithmetic is correct, but its sign is easy to misread because lower loss is better. Defining the sign convention first avoids calling worse outer performance a negative optimism gap.
Alternatives, Limits, and Common Mistakes
The inner-best score answers a tuning question: which candidate performed best under the inner selection rule? The outer score answers an evaluation question: how did the selected procedure perform on held-out outer data? Reporting only the inner winner’s score risks presenting a selection-contaminated result as an unbiased evaluation. Reporting only the outer results provides the central evaluation, while comparing them to the inner winner adds a diagnostic about the tuning process. An independent external validation set can provide a distinct evaluation when it is genuinely untouched, but it does not make the inner–outer gap a universal correction.
This comparison also differs from comparing two models using paired outer-fold results. Here the focus is the relationship between the selected model’s inner score and its corresponding outer score, not a claim that one model is superior to another. A large gap alone does not establish that tuning should be abandoned, nor does it identify which alternative would perform better.
Common mistakes include treating the mean gap as a correction to subtract from every future score; comparing unmatched splits; interpreting a loss difference as if higher were better; and claiming the gap proves leakage. Leakage is one possible failure mode to investigate, but legitimate score variation, training-size differences, and imperfectly matched validation designs can also produce a gap. Conversely, a small gap cannot certify that the pipeline is leakage-free.
Check Your Understanding
Answer each question using the stated score direction. Solutions follow the questions.
- For a higher-is-better score, an outer split has best inner score 0.86 and outer score 0.81. What is the gap, and what does it mean?
- For a lower-is-better loss, the inner value is 0.31 and the outer value is 0.35. Calculate the deterioration-direction gap.
- Why can the highest of many inner scores be optimistic even when each candidate’s score has mean-zero error?
- Across repeated nested cross-validation, some matched gaps are positive and some negative. Why should the mean gap not automatically be treated as a correction or as a formal significance test?
Solutions. 1. \(0.86-0.81=0.05\), or 5 percentage points if the score is a proportion. The best inner score is higher than its matched outer score by 0.05; this is consistent with, but does not prove, selection optimism. 2. \(0.35-0.31=0.04\) loss units, indicating worse outer loss by 0.04. 3. Maximizing selects the largest observed score, so the winner tends to include favorable noise; for equal underlying scores, the expected maximum of mean-zero errors is at least zero. 4. The gaps combine selection effects with split variability and other design differences. Repeated splits reuse observations, so the gaps are not independent replicates; their mean is descriptive and is not automatically a universal correction or a valid significance test.