What Generalization Performance a Split Estimates

What Does a Prediction-Error Number Describe?

A regression model can have a small average error on one set of observations and still perform poorly on the cases where it will actually be used. The key question is not only how large is the error? It is also for which population, under which sampling design, and for a model trained on how much data? A test-set or cross-validation result is useful only when its answer matches the prediction task of interest.

For example, a model evaluated on a random sample of patients from several clinics may be intended for new patients at those same clinics. That evaluation does not automatically tell us how well it will predict at a new clinic, or in a future population with different characteristics. The split is part of the definition of what is being assessed.

Define the Target Before the Estimate

Suppose \(X\) denotes a new case’s predictor values and \(Y\) its outcome. A fitted regression model, based on training data \(D\), gives a prediction \(\hat f_D(X)\). To describe predictive performance, we must specify a loss function: a numerical measure of the discrepancy between the observed outcome and its prediction. For squared-error loss, the loss is \((Y-\hat f_D(X))^2\); for absolute-error loss, it is \(|Y-\hat f_D(X)|\).

With squared-error loss, risk is the target-population mean squared prediction error. Its square root is the root mean squared error, which is on the same scale and in the same units as \(Y\). With absolute-error loss, risk is the target-population mean absolute error, also in the units of \(Y\). The chosen loss matters: two models can rank differently under squared and absolute error because squared error gives more weight to large misses.

There is a second quantity to keep distinct from the risk of one fitted model. A learning procedure is the rule used to fit a model from training data. If \(D\) itself is regarded as a random sample of a stated size, the procedure’s expected predictive risk averages over both the possible training samples and new target-population cases. This quantity describes the procedure at that training-sample size; it is not the conditional risk of a particular fitted model.

When a Holdout Average Estimates Target Risk

Suppose a model has been fitted using training data \(D\), and a separate test set contains \(m\) cases. If the test cases are independent draws from the target population and the model is held fixed, the average test loss estimates the risk of that fitted model. This statement depends on the test cases representing the target population and being separate from the data used to fit the model.

For this average to be an unbiased estimate of \(R_P(\hat f_D)\), assume that, conditional on \(D\), the test cases are independent and identically distributed from \(P\), the fitted model is not changed using their outcomes, and the expected loss is finite. Under those conditions, each test loss has conditional expectation \(R_P(\hat f_D)\). Linearity of expectation then gives:

This is a justification of the estimate’s target, not a claim that any one test-set average equals the risk exactly. A finite test set varies from sample to sample, so its average loss has sampling variability. This tutorial focuses on what the estimate represents; the size and uncertainty of an evaluation are separate questions.

Sampling and Splitting Define the Population Represented

A test set need not represent the intended target population merely because it was kept out of model fitting. A random split of an appropriate sample can support a target-population interpretation when the original cases are sampled in a way that represents the target. But a split cannot repair a mismatch between the available data and the intended future cases. If some groups are overrepresented, some settings are absent, or observations are selected from a different time period, the average test loss may describe that particular mix instead.

The distinction is especially clear when test cases are deliberately sampled in different proportions from their proportions in the target population. An unweighted average then gives more influence to groups that were sampled more heavily. If group-specific losses are available and the target group proportions are known, a weighted average can instead match the target composition. That adjustment is justified only if the within-group test cases represent their corresponding target groups.

The split design also defines which kinds of new cases are being represented. A random division of cases drawn from the same settings is aimed at new cases from that mixture of settings. If the intended task is prediction in future periods, a test set from a different period may better represent that task. If the task is prediction at a new site, holding out cases from sites that also appear in training does not directly estimate performance at unseen sites. These are different target populations or sampling conditions, so their risks need not be equal.

What Cross-Validation Represents

Cross-validation evaluates predictions for cases left out of the data used to fit the model that predicts them. Its result is therefore associated with the way cases are assigned to training and held-out portions, and with the size and composition of those portions. In ordinary \(K\)-fold cross-validation, each fitted model uses approximately \((K-1)/K\) of the available cases. The pooled average of held-out losses describes performance for models trained on subsets of roughly that size, under the relevant data-splitting design.

This makes cross-validation a way to assess a learning procedure at a particular training-set size, rather than a direct test of the exact model later fitted to all available data. The final model may use more cases and therefore differ from the cross-validation fits. How much its risk differs is not determined by the cross-validation average alone.

A Practical Check on the Intended Interpretation

Before interpreting a holdout or cross-validation result, specify four things: the target cases, the loss function, the sampling or splitting design, and the training-set size associated with the evaluated models. This prevents a numerical estimate from being treated as more general than its design supports.

Common Mistake: Treating “Held Out” as “Representative”

A common but incorrect argument is: “These cases were not used to fit the model, so their average error estimates performance on any future cases.” Being held out addresses whether those outcomes were used to fit the evaluated model; it does not establish that the cases represent every population where the model might be used. A test set sampled only from one clinic, for instance, cannot by itself establish performance at clinics with different patients or practices.

The correction is to name the population supported by the sampling design and limit the interpretation to it. Likewise, a cross-validation result should be tied to the fitting procedure and training size represented by its folds. A valid estimate answers a specific predictive question; a different target population calls for an evaluation design that represents that population.