Cross-Validation As A Generalization Estimate

What Does Cross-Validation Estimate?

A validation error is useful only when its target is clear. Cross-validation repeatedly fits a learning procedure on part of the observed data and evaluates that fit on observations left out of that particular fit. The held-out observations can be reused in other fits: an observation used for evaluation once may be used for training elsewhere. The key question is not simply how many times data are reused, but what average prediction risk the resulting held-out losses represent.

The target is generally the average performance of a learning procedure trained on a specified sample size, evaluated on a new case from the population. It is not automatically the error of the final model fitted on all available data. Nor does a held-out average have this interpretation merely because the held-out outcomes were not used to choose the split. The split design must also preserve the relevant distributions of training and evaluation observations.

For squared-error prediction, \(L(y,\hat y)=(y-\hat y)^2\). For binary classification with predicted event probability \(p\) and outcome \(y\in\{0,1\}\), the Brier loss is \(L(y,p)=(y-p)^2\). Other loss functions define other targets. The learning procedure includes any data-dependent operations that affect predictions, such as estimating preprocessing quantities or selecting model terms; those operations must be based on the training sample for the corresponding held-out loss to represent performance without access to the held-out case.

Why Held-Out Losses Can Estimate This Risk

Consider a split that assigns observations to training and evaluation sets independently of all their measured values. Suppose the observations are independent draws from the same population, every training set has size \(m\), and the entire learning procedure for each fit uses only that training set. For a particular fit, its training observations and its held-out observations are then independent samples from \(P\). Conditional on the fitted rule, each held-out case has the same distribution as a new case from \(P\).

This gives a short justification for the estimate. Let \(T_j\) denote training indices for fit \(j\), \(V_j\) its held-out indices, and \(n_j=|V_j|\). The average held-out loss for that fit is $$ \widehat R_j = \frac{1}{n_j}\sum_{i\in V_j} L\bigl(Y_i,f_{D_{T_j}}(X_i)\bigr). $$ Under the conditions just stated, each validation observation is independent of the training data used to construct its prediction and is distributed as a new observation from \(P\). Thus its expected loss, averaging over both the training sample and the new case, is \(R_m\). Averaging these expected losses gives $$ E\left[\frac{1}{J}\sum_{j=1}^{J}\widehat R_j\right] =\frac{1}{J}\sum_{j=1}^{J}E[\widehat R_j] =\frac{1}{J}\sum_{j=1}^{J}R_m =R_m. $$ This is an expectation statement about the validation design and repeated samples from the population. In one realized data set, the calculated cross-validation value is an estimate, not a guarantee that the population risk has been measured exactly.

The argument does not require the different held-out losses to be independent. The same observation can appear in several training sets, and fitted models can share much of their training data. These overlaps make the estimates related. Reuse is compatible with the expectation argument, but it does not turn the held-out losses into independent observations or, by itself, provide a standard error.

The Training Size Is Part of the Target

A procedure trained on fewer observations may perform differently from the same procedure trained on more observations. Therefore, if every fit uses \(m\) observations, the natural target is \(R_m\), not a risk with an unspecified training size. For example, if each fit holds out one fifth of the available sample, the resulting estimate concerns models trained on the remaining four fifths, subject to the sampling and split assumptions. A final model refitted on the entire data set has a different training size, so its risk need not equal the estimated \(R_m\).

If training-set sizes differ, there may be no single \(R_m\) that describes every fit. An equally weighted average of fit-level validation means estimates an average of risks at the corresponding training sizes, when each fit and its validation cases have the required distributions. If instead the goal is to give each held-out case equal weight, use the pooled average of all held-out losses. With unequal validation-set sizes, these two summaries can differ. The averaging rule should match the intended target and be reported clearly.

The same reasoning applies to the loss scale. A mean squared error is in squared outcome units; a root mean squared error is its square root and is not itself the average of case-level squared errors. For a probability prediction, the average Brier loss summarizes squared probability errors. A metric must be calculated consistently across held-out predictions before its interpretation is stated.

When the Split Changes the Population Being Evaluated

The expectation argument depends on more than keeping held-out outcomes out of the fitting step. A split based on predictor values can produce held-out cases whose predictor distribution differs from that of a new population case. It can also change the distribution of training samples. In that situation, the held-out average may estimate performance for the split-induced distributions rather than \(R_m\) for the original population.

A split independent of all observations is a straightforward condition that supports the stated interpretation when the observations are independent and identically distributed. Other designs may be appropriate, but they require a clear argument that training and evaluation cases represent the distributions in the target risk, or an explicit target defined for the design-induced distributions. Where evaluation cases have been sampled with unequal probabilities, appropriate weighting may be needed to recover a population-level target.

Practical Interpretation and Common Traps

Cross-validation is useful because it allows observations to contribute to fitting in some fits and to evaluation in others, rather than reserving every evaluation case permanently. That reuse makes efficient use of a data set, but it does not remove the need to define the prediction setting. Independent, identically distributed observations are a suitable basis for the simple risk argument above. Repeated entities or ordered observations can violate that basis if the split lets related records or future information cross the evaluation boundary. In such cases, the validation design needs to reflect the intended deployment setting.

For an actuarial analysis, a risk estimate is meaningful only when the cases used for validation resemble the cases on which predictions will be used, and the metric matches the decision. For example, squared error may be appropriate when the magnitude of numerical prediction errors matters, whereas a probability loss may be more informative when assessing predicted event probabilities. The estimate still has sampling variability, and a favorable value does not prove that a model will perform equally well under population or operating conditions that differ from those represented in the validation design.