Uncertainty In Cross-Validation Scores

Why a Cross-Validation Score Needs an Uncertainty Summary

A cross-validation score is an estimate of predictive performance, not a fixed property of a fitted GLM. It depends on which observations are held out, which observations train each fit, and how the held-out predictions are scored. Two partitions of the same data can therefore produce different scores. Quantifying that variation is useful when comparing models, but the variation must be interpreted according to how the scores were generated.

The central difficulty is that the scores from different folds are not generally independent. Their training sets overlap, and their held-out sets are parts of the same observed data. Consequently, the standard deviation across folds describes observed fold-to-fold spread, but dividing it by the square root of the number of folds does not automatically give a valid standard error for the overall cross-validation score.

Fold Scores and the Overall Cross-Validation Score

Fix a data set of $n$ observations and a $K$-fold partition. For fold $k$, fit the GLM using all observations outside that fold, then evaluate a chosen predictive score on the held-out observations. For example, for a binary response, the log predictive score of an observation with outcome $y_i$ and predicted probability $\hat p_i$ is $y_i\log(\hat p_i)+(1-y_i)\log(1-\hat p_i)$. Higher log score is better. Equivalently, one may use negative log score as a loss, for which lower is better. State which convention is used before interpreting a difference.

Definition: Let $s_k$ be the mean held-out score in fold $k$, and let $n_k$ be the number of held-out observations in that fold. The observation-weighted $K$-fold cross-validation score is $$ \hat S_{\mathrm{CV}}=\frac{\sum_{k=1}^{K} n_k s_k}{n}. $$ If all folds have equal size, this equals $K^{-1}\sum_{k=1}^{K}s_k$. When sizes differ, the weighted expression gives each held-out observation equal weight.

This score summarizes predictive performance under the particular cross-validation design. It does not by itself express uncertainty. A fold-level standard deviation can describe how widely the $s_k$ values vary, but that is a different quantity from the sampling standard error of $\hat S_{\mathrm{CV}}$.

Worked Example: A Four-Fold Difference in Binary Log Score

Suppose two logistic GLMs, A and B, are compared using four held-out folds, each containing one observation with $y=1$. On each fold, model B predicts probability $0.5$. Model A predicts, respectively, $0.47619$, $0.49020$, $0.49020$, and $0.48544$. Define $d_k$ as model A's log score minus model B's log score on fold $k$. Since $y=1$, each score is $\log(\hat p)$, so $d_k=\log(\hat p_{A,k}/0.5)$. To the displayed precision, the four differences are $-0.0487912$, $-0.0197946$, $-0.0197946$, and $-0.0295524$. Negative values favor B.

The mean difference is $$ \bar d=\frac{-0.0487912-0.0197946-0.0197946-0.0295524}{4} =\frac{-0.1179328}{4} \approx -0.0294832. $$ Thus the mean held-out log score favors B by about $0.02948$ log-score units per observation.

The deviations from this mean are approximately $-0.0193080$, $0.0096886$, $0.0096886$, and $-0.0000692$. Their squared sum is approximately $$ (-0.0193080)^2+2(0.0096886)^2+(-0.0000692)^2 \approx 0.0005605. $$ The sample standard deviation across the four fold differences is therefore $$ s_d=\sqrt{\frac{0.0005605}{4-1}}\approx 0.01367, $$ and the commonly computed quantity $s_d/\sqrt{4}$ is approximately $0.006835$. The latter is only the standard error one would obtain from a model treating these four differences as independent observations with a common variance. The fold construction does not guarantee that assumption.

Why Fold Scores Are Dependent

In ordinary $K$-fold cross-validation, the fit for fold $k$ uses the observations in every other fold. Thus the training sets for two different folds overlap substantially. The held-out scores also share features of the same data set and can respond similarly to a difficult or unusually easy sample. These sources of dependence mean that the fold scores should not be treated as $K$ independent replicates of an experiment.

The basic variance identity makes the consequence precise. It applies to random fold scores under any probability model for the data and partition, provided their second moments are finite.

Theorem: If $S_1,\ldots,S_K$ have finite variances, then $$ \operatorname{Var}\left(\frac{1}{K}\sum_{k=1}^{K}S_k\right) =\frac{1}{K^2}\left\{\sum_{k=1}^{K}\operatorname{Var}(S_k) +2\sum_{1\leq j<k\leq K}\operatorname{Cov}(S_j,S_k)\right\}. $$ Equal variances and zero pairwise covariances are sufficient, but not necessary, for the variance of the mean to equal $K^{-1}\operatorname{Var}(S_1)$; in general, use the full covariance identity above.

Proof. Write $\bar S=K^{-1}\sum_k S_k$. By bilinearity of covariance, $$ \operatorname{Var}(\bar S) =\operatorname{Cov}(\bar S,\bar S) =\frac{1}{K^2}\sum_{j=1}^{K}\sum_{k=1}^{K}\operatorname{Cov}(S_j,S_k). $$ The diagonal terms are $\operatorname{Var}(S_k)$. For each pair $j<k$, the two off-diagonal terms are $\operatorname{Cov}(S_j,S_k)$ and $\operatorname{Cov}(S_k,S_j)$; these are equal. Grouping the terms gives the stated identity. If all variances equal $v$ and all pairwise covariances are zero, the result reduces to $K^{-2}Kv=v/K$. $\square$

A positive covariance increases the variance of the average relative to the independence calculation. Negative covariance decreases it. In cross-validation, there is no universal covariance value that can be inferred from the number of folds alone.

Worked Example: The Effect of Positive Fold Covariance

Suppose three fold scores have variance $0.04$ each and every pair has covariance $0.02$. These are hypothetical values chosen to illustrate the variance identity, not a general model for cross-validation. The variance of their mean is $$ \operatorname{Var}(\bar S) =\frac{3(0.04)+2\{3(0.02)\}}{3^2} =\frac{0.12+0.12}{9} \approx 0.02667. $$ The corresponding standard deviation is $\sqrt{0.02667}\approx 0.1633$.

If the covariance terms were incorrectly set to zero, the estimated variance would be $0.04/3\approx 0.01333$, giving a standard deviation of about $0.1155$. In this illustration, ignoring positive dependence substantially understates the variability of the average. The calculation shows why the independence assumption matters; it does not establish that fold-score covariance is always positive or equal to $0.02$.

Repeated Cross-Validation: What Repetition Measures

Repeated $K$-fold cross-validation uses new random partitions of the same observed data. Let $\hat S_r$ be the overall score from repetition $r$, for $r=1,\ldots,R$. The repetitions can reveal sensitivity to the random partition. Conditional on the observed data, if partitions are generated independently using the same randomization procedure, the $\hat S_r$ are identically distributed and independent with respect to that partition randomization. Their sample standard deviation estimates the conditional spread across random partitions.

That conditional interpretation is narrower than uncertainty about performance on a new population. Every repetition reuses the same observed outcomes and predictors. The repetition-to-repetition spread does not include the full variation that would result from drawing a new data set, and it does not correct for any selection of the model using the same data.

Result: Conditional on a fixed data set, suppose $\hat S_1,\ldots,\hat S_R$ are independent with a common finite mean and variance $\tau^2$, because the repetitions use independent random partitions from the same procedure. Then the sample variance $$ s_R^2=\frac{1}{R-1}\sum_{r=1}^{R}(\hat S_r-\bar S_R)^2, \qquad \bar S_R=\frac{1}{R}\sum_{r=1}^{R}\hat S_r, $$ is unbiased for $\tau^2$. The quantity $s_R/\sqrt R$ estimates the conditional standard deviation of the average over $R$ independently randomized repetitions.

Proof. Let the common conditional mean be $\mu$. The identity $$ \sum_{r=1}^{R}(\hat S_r-\bar S_R)^2 =\sum_{r=1}^{R}(\hat S_r-\mu)^2-R(\bar S_R-\mu)^2 $$ follows by expanding both sides and using $\sum_r(\hat S_r-\bar S_R)=0$. Taking conditional expectations, the first term on the right has expectation $R\tau^2$. Independence gives $\operatorname{Var}(\bar S_R)=\tau^2/R$, so the second term has expectation $R(\tau^2/R)=\tau^2$. Therefore the expected numerator is $(R-1)\tau^2$, and division by $R-1$ proves $E[s_R^2\mid\text{data}]=\tau^2$. The variance of the average is $\tau^2/R$ by independence, which motivates $s_R/\sqrt R$ as its estimated conditional standard deviation. $\square$

This result concerns randomization over partitions conditional on the data. If the goal is uncertainty about population predictive performance, the data themselves are also random, and this conditional calculation does not capture that additional source of uncertainty.

Worked Example: Summarizing Five Repeated Scores

Suppose five independently randomized repetitions of five-fold cross-validation for a Poisson GLM give mean held-out negative log scores $1.20$, $1.16$, $1.18$, $1.22$, and $1.19$. Lower is better. Their mean is $$ \bar S_R=\frac{1.20+1.16+1.18+1.22+1.19}{5} =\frac{5.95}{5}=1.19. $$ The deviations are $0.01$, $-0.03$, $-0.01$, $0.03$, and $0$. The squared deviations sum to $$ 0.01^2+(-0.03)^2+(-0.01)^2+0.03^2+0^2 =0.0001+0.0009+0.0001+0.0009 =0.002. $$ Thus $s_R^2=0.002/4=0.0005$, $s_R\approx 0.02236$, and $s_R/\sqrt{5}\approx 0.0100$. The observed scores vary by about $0.022$ across random partitions, and the estimated conditional standard deviation of their five-repetition average is about $0.010$, under the independent-random-partition setup. Neither number is, by itself, a standard error for population predictive performance.

Comparing Models and Reporting Uncertainty

For model comparison, calculate paired score differences on the same folds or repetitions. For example, if $s_{A,k}$ and $s_{B,k}$ are scores on the same held-out fold, define $d_k=s_{A,k}-s_{B,k}$. Pairing avoids comparing scores from unrelated partitions, and the average difference directly describes which model scores better under the chosen convention. However, pairing does not make the fold differences independent: their training sets still overlap.

A clear report should identify the scoring rule and direction, the cross-validation design, whether fold sizes were equal, and whether a reported spread is across folds or across independent repetitions. It should label a standard deviation as descriptive unless its inferential assumptions are justified. If repeated scores are used, state that their spread is conditional on the observed data and the random partition procedure. A more ambitious population-level uncertainty analysis requires a method whose target and assumptions are stated explicitly; the fold standard deviation alone does not supply it.

Common Mistake: Treating Folds as Independent Replicates

Incorrect reasoning: “There are $K$ fold scores, so the standard error of the cross-validation mean is the fold standard deviation divided by $\sqrt K$.”

Correction: That formula requires independent, identically distributed scores (or other assumptions that justify the variance calculation). Cross-validation folds share training observations and arise from one data set, so independence is not ensured. The quantity may be reported as a familiar descriptive summary, but it should not be presented as a generally valid confidence measure. Repeated partitions can quantify sensitivity to partition randomization; they do not create new independent data sets.

Key takeaway: Keep three quantities distinct: the overall cross-validation score, the spread of scores across folds, and the variability of the score under a specified repetition or sampling mechanism. Dependence among folds limits naive standard-error calculations, while repeated partitions primarily describe sensitivity to the split on the observed data.

Check Your Understanding

Answer each question using the distinction between fold spread, split variability, and population uncertainty.

Why should held-out fold scores from ordinary $K$-fold cross-validation not automatically be treated as independent replicates?
When fold sizes differ, how should their mean scores be combined to give each held-out observation equal weight?
State the covariance term that is omitted when the variance of an average is calculated under an independence assumption.
What uncertainty does the standard deviation across independently randomized repetitions estimate, conditional on a fixed data set?
Why does repeating cross-validation many times not, by itself, measure all uncertainty about predictive performance in a new population?