Why a Cross-Validation Score Needs an Uncertainty Qualification
A cross-validation score is an estimate, not a fixed property of a model. Its value can change when the observations are divided into different folds: a particular split may place unusually difficult cases together, or the fitted models may differ because their training samples differ. Reporting only the average score hides this variation. Reporting a standard error can help describe it, but only if the standard error is interpreted in light of how cross-validation is constructed.
The central complication is that fold-level losses are generally dependent. In ordinary \(K\)-fold cross-validation, each fitted model uses most of the same observations as the other fitted models. Thus, the familiar calculation “sample standard deviation divided by the square root of the number of folds” is a useful descriptive summary under an independence assumption, but it is not automatically a valid measure of sampling uncertainty. The emphasis here is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
Calculating the Naive Fold-Level Standard Error
For an equal-fold calculation, first find the mean loss, then calculate the sample standard deviation using \(K-1\) in the denominator, and finally divide by \(\sqrt K\). The standard deviation describes how spread out the observed fold losses are. The standard error is intended to describe the variability of their average under a specified sampling model. These are different quantities.
Worked Example: Five-Fold Squared-Error Loss
A hypothetical analyst uses five-fold cross-validation to assess a regularized regression model. The five fold mean squared errors are \(8,10,9,13,\) and \(10\), in squared response units. Assume the folds have equal sizes. Calculate the cross-validation score, the sample standard deviation of the fold losses, and the naive standard error. No parametric distribution for the response or fold losses is assumed.
Method. Compute the arithmetic mean and sample standard deviation across the five fold losses, then divide the standard deviation by \(\sqrt{5}\). This calculation avoids using the population-variance denominator \(K\) when estimating spread from a sample of fold values.
Mean loss. The total is \(8+10+9+13+10=50\), so \(\overline{L}_{CV}=50/5=10\).
Squared deviations. The deviations from 10 are \(-2,0,-1,3,0\). Their squares are \(4,0,1,9,0\), with sum \(14\). Therefore \(s_L^2=14/(5-1)=3.5\), and \(s_L=\sqrt{3.5}=1.8708\) before rounding.
Naive standard error. \(s_L/\sqrt{K}=1.8708/\sqrt{5}=0.8367\) before rounding. The cross-validation score is \(10\), the fold standard deviation is \(1.871\), and the naive standard error is \(0.837\), all rounded to three decimal places.
Check and limitation. The deviations sum to zero, as they should when calculated from the mean. The standard error is smaller than the standard deviation because it divides by \(\sqrt{5}\). It is a fold-based summary, not automatically a valid confidence interval for prediction error on new data; the folds do not meet the independence assumption underlying that interpretation.
Why Fold Losses Are Dependent
In \(K\)-fold cross-validation with \(n\) observations and equally sized folds, each training set contains approximately \(n(K-1)/K\) observations. Two different training sets omit different folds, but they share approximately \(n(K-2)/K\) observations. For example, in five-fold cross-validation, each training set contains about \(4n/5\) observations, and two training sets share about \(3n/5\). The fitted models therefore reuse many of the same data.
The test folds themselves are disjoint, but that fact does not make the fold losses independent. The loss on a fold depends both on which cases are in that fold and on the model fitted using the other folds. An observation that is tested in one fold is used to train the models evaluated on the other folds. Further, those other models have overlapping training data. These connections can induce dependence among the fold losses.
The effect on the variance of the average can be seen directly. If the fold losses have a common variance \(\sigma_L^2\) and a common pairwise covariance \(c\), then
The naive independent-fold formula keeps only \(\sigma_L^2/K\), effectively setting \(c=0\). When the covariance is positive, that formula understates the variance; if covariance is negative, it can overstate it. Overlapping training sets establish a reason to question independence, but they do not determine the covariance’s sign or size in every problem. Therefore, one should not simply assert that every naive standard error is too small by a known amount.
Repeated Cross-Validation: What the Variation Measures
Repeated cross-validation uses new random fold assignments and repeats the full fold procedure. It can show how much a score changes with the random partitioning of the same data. A natural summary is the mean score across repeats, together with the standard deviation and standard error of those repeat-level scores. This can be more interpretable than treating every fold from every repeat as a separate, independent observation.
However, repeats reuse the same original data. They are not independent samples from the population merely because the random fold assignments differ. A standard error computed across repeat averages therefore primarily summarizes variation due to the repeated partitioning and fitting process on the observed data. It does not by itself quantify all uncertainty from drawing a new data set.
Worked Example: Summarizing Four Repeated-CV Scores
A hypothetical analyst runs four repetitions of the same five-fold cross-validation procedure on one fixed data set. The mean loss across the five folds in each repetition is \(12,10,11,\) and \(9\). Calculate the average repeat score, the sample standard deviation across repeats, and the naive standard error across the four repeat scores. Treat the scores as numerical losses; do not assume they are independent samples from a population.
Method. First summarize the four repeat averages as observations. This measures variation among the repeated partitions, while keeping clear that the same original observations were reused in each repetition.
Mean. The sum is \(12+10+11+9=42\), so the average is \(42/4=10.5\).
Sample standard deviation. Deviations from 10.5 are \(1.5,-0.5,0.5,-1.5\). The squared deviations are \(2.25,0.25,0.25,2.25\), summing to \(5\). The sample variance is \(5/(4-1)=5/3\), so the sample standard deviation is \(\sqrt{5/3}=1.2910\) before rounding.
Naive standard error across repeats. Divide by \(\sqrt{4}=2\): \(1.2910/2=0.6455\) before rounding. The average repeat score is \(10.5\), the standard deviation is \(1.291\), and the naive standard error is \(0.645\), rounded to three decimal places.
Check and limitation. The deviations sum to zero, and their squared sum is \(5\). The calculation is correct as a summary of the four repeat scores. Because all repetitions reuse the same data set, \(0.645\) should not be presented as a general sampling standard error for future-data performance without additional assumptions and a suitable sampling argument.
A Dependence-Adjusted Heuristic
One commonly used approximation for repeated \(K\)-fold cross-validation incorporates a term related to the ratio of test-set size to training-set size. In a version of the corrected resampled standard-error calculation, if there are \(R\) repetitions and \(K\) folds per repetition, and \(s^2\) is the sample variance of the \(RK\) fold-level scores, the approximate variance is
For equal folds, \(n_{\mathrm{test}}/n_{\mathrm{train}}\) is approximately \(1/(K-1)\). The added ratio prevents the calculation from treating all \(RK\) fold scores as independent. This is an approximation, not a universal variance identity: its quality depends on the setting and the way the scores are generated. It should be reported as a chosen dependence adjustment, not as a guaranteed correction or an exact confidence interval.
Worked Example: Applying a Dependence Adjustment
A hypothetical validation study uses \(R=3\) repetitions of five-fold cross-validation. Across its 15 fold-level losses, seven are \(8\), one is \(10\), and seven are \(12\). Use the corrected resampled approximation above to calculate an adjusted standard error. Assume equal fold sizes, so the test-to-training size ratio is \(1/4\).
Method. Calculate the sample variance across all 15 fold scores, then multiply it by \(1/(RK)+n_{\mathrm{test}}/n_{\mathrm{train}}\). Take the square root of the resulting approximate variance.
Mean and sample variance. The mean is \((7(8)+10+7(12))/15=(56+10+84)/15=150/15=10\). The squared deviations from 10 sum to \(7(2^2)+0+7(2^2)=56\). Thus \(s^2=56/(15-1)=56/14=4\).
Adjustment factor. Here \(RK=3(5)=15\), so the factor is \(1/15+1/4=4/60+15/60=19/60\).
Adjusted variance and standard error. The approximate variance is \((19/60)(4)=76/60=19/15=1.2667\) before rounding. Its square root is \(\sqrt{19/15}=1.1255\) before rounding. The adjusted standard error is \(1.125\), rounded to three decimal places.
Check and limitation. The mean is 10 because the seven scores below 10 balance the seven scores above 10. The adjustment factor \(19/60\) exceeds the independent-fold factor \(1/15\), so the adjusted standard error exceeds the naive value \(2/\sqrt{15}=0.516\). This particular increase follows from the specified heuristic; it does not establish a universal correction for every cross-validation design.
Practical Interpretation and Common Traps
A cross-validation report should identify what is being summarized. State the loss function, the fold structure, whether the reported dispersion is a standard deviation or a standard error, and whether the standard error is the naive fold-based quantity or a dependence-adjusted approximation. For example, “mean five-fold validation loss \(=10.0\), fold standard deviation \(=1.87\), naive fold standard error \(=0.84\)” is more informative than “score \(=10.0\pm0.84\)” because it makes the calculation and its limitation visible.
Do not confuse variation from random fold assignment with uncertainty from collecting a new sample. Repeating folds on a fixed data set can reveal sensitivity to partitioning, but it cannot manufacture new independent observations. Nor does a small fold standard error prove that the model’s future loss is known precisely. That conclusion would require a justified sampling model and an uncertainty procedure appropriate to the target being estimated.
A further trap is to use the standard deviation where the standard error is requested, or to divide by \(\sqrt{RK}\) while claiming that all repeated fold scores are independent. The former reports spread rather than mean uncertainty under the independent model; the latter ignores reuse of both observations and overlapping training data. Finally, losses from unequal-sized folds may need to be aggregated with fold-size weights to obtain the observation-level average loss. An unweighted mean of fold losses and a pooled observation-level loss need not be the same.
Specify the loss and whether the target is a mean across folds, a pooled loss across held-out observations, or a mean across repetitions.
Use the relevant fold or repeat scores to compute a sample standard deviation, keeping the denominator consistent with the number of values summarized.
If using \(s/\sqrt{K}\), label it as naive and explain that its usual independent-observation interpretation may not hold for overlapping cross-validation training sets.
Describe partition sensitivity or an explicitly chosen approximation; do not turn it into a confidence statement about future performance without further justification.
Check Your Understanding
For each problem, distinguish the arithmetic summary from the assumptions needed to interpret it as uncertainty.
- Four equal-sized folds have losses \(6,8,10,12\). Calculate their mean, sample standard deviation, and naive standard error. Round the last two quantities to three decimals. (Answer: Mean \(=9\), sample standard deviation \(=\sqrt{20/3}\approx2.582\), naive standard error \(\approx1.291\).)
- Three repeated-CV mean scores on one data set are \(14,12,\) and \(13\). Calculate their average and sample standard deviation. What does the spread directly summarize? (Answer: Average \(=13\), sample standard deviation \(=1\); it summarizes variation among these repeat scores on the reused data set.)
- There are \(n=100\) observations and five equal folds. How many observations are in each training set, and how many are shared by the training sets for two different folds? (Answer: Each training set has \(80\) observations; two training sets share \(60\) observations.)
- A repeated five-fold analysis has \(R=2\), fold-score sample variance \(s^2=2.25\), and equal folds. Using the dependence-adjusted approximation in this tutorial, calculate the adjusted standard error to three decimals. (Answer: Adjustment factor \(=1/10+1/4=0.35\); adjusted variance \(=0.35(2.25)=0.7875\); adjusted standard error \(=\sqrt{0.7875}\approx0.887\).)
- Why does having disjoint test folds fail to establish that fold losses are independent? (Answer: The models evaluated on different folds are fitted using overlapping training observations, and observations held out in one fold are used to train other fold models.)