Independent Holdout Validation

Why Reserve Individuals for Testing?

A prediction model can appear to perform well on the same individuals used to estimate its parameters. That performance is not a fair estimate of how well the fitted model will predict new individuals: the model has already been adapted to the development data. A holdout design addresses this by separating individuals used for development from individuals reserved for evaluation.

In genetic prediction, the predictor may include a polygenic score, such as \(S_i=\sum_{j=1}^{m}\beta_jG_{ij}\), where \(G_{ij}\) is individual \(i\)'s genotype dosage at variant \(j\), and \(\beta_j\) is its specified weight. The prediction model might use this score alone or combine it with other predictors. This tutorial concerns how to evaluate an already specified model using a train–test split, not how to choose or construct the score.

The test set must be held out from decisions, not merely from the final parameter-fitting step. If its outcomes help choose predictors, tune a model, set a classification threshold, or choose among candidate models, the resulting test performance is no longer a clean evaluation of those decisions. A model that has already been specified can still require parameter estimation; those parameters are estimated using training individuals, then the resulting fitted model is evaluated on the test individuals.

Specify the Split and the Evaluation Target

Before making the split, state the prediction target and the population in which performance is meant to apply. The earlier material on prediction targets and validation populations distinguishes these questions: a test set estimates performance for the population it represents, not automatically for every population in which a score might later be used. A random split of one cohort generally evaluates prediction in a population like that cohort, subject to how the cohort was sampled.

The model and evaluation plan should also be clear in advance. Specify the predictors, the form of the model, the outcome, and the performance measure. For a quantitative outcome, one possible measure is root mean squared prediction error. For a binary outcome, one simple measure is classification accuracy at a stated probability threshold. These measures answer different questions; no single measure summarizes every aspect of prediction.

Choose the split at the level of the independent observational units. In a simple setting where individuals are independent and sampled from the target population, randomly assigning individuals to training and test sets can give both sets a similar population composition. If observations have important dependencies, the split must respect those dependencies; a naive individual-level split may not provide an independent evaluation. The appropriate design also depends on whether the intended use is prediction in a similar population or prediction in a distinct setting.

Fit on Training Data, Predict on Test Data

Let \(Y_i\) denote the observed outcome for individual \(i\), and let \(\hat{Y}_i\) denote the model's prediction. For example, a linear prediction model might be specified as \(\hat{Y}_i=\hat{\alpha}+\hat{\beta}S_i+\hat{\boldsymbol{\gamma}}^\mathsf{T}\mathbf{C}_i\), where \(S_i\) is a genetic score and \(\mathbf{C}_i\) is a vector of other predictors. If the intercept and coefficients are to be estimated from the cohort, estimate them using training individuals only. Then apply those fitted coefficients to the test individuals.

Any data-dependent transformation needed to apply the model must also be determined using training data and then applied consistently to the test set. For example, if a predictor is centered using its training-set mean, use that same mean when transforming test individuals. If the model's coefficients and transformations were fixed in advance from an external source, apply those fixed quantities as specified rather than re-estimating them with test outcomes.

For quantitative outcomes, define the prediction error for individual \(i\) as \(e_i=Y_i-\hat{Y}_i\). A common summary is the test-set mean squared error, with root mean squared error reported in the outcome's units:

MSE is in squared outcome units; RMSE is in the original outcome units. Smaller values indicate smaller typical squared errors for the test individuals, but interpretation depends on the trait scale and the target population. A test score is a sample-based estimate, not a guarantee of performance for every future individual.

Binary Outcomes and a Prespecified Decision Rule

For a binary outcome, a model may produce a predicted probability \(\hat{p}_i\). If classification accuracy is the chosen measure, a decision rule is needed to turn probabilities into predicted classes. For example, classify an individual as positive when \(\hat{p}_i\geq 0.50\), and negative otherwise. That threshold must be fixed without using test outcomes. Accuracy is the fraction of test individuals whose predicted classes match their observed classes; it depends on the threshold and the distribution of outcomes in the test population.

Why the Holdout Estimate Works—and Its Limits

The basic justification is that the test individuals are not used to fit the fixed model, so their outcomes can provide new information about its prediction errors. More formally, let \(D_{\text{train}}\) be the training data and let \(f\) be the fitted model resulting from them. Define the loss for a new individual as \(L(Y,f(X))\), where \(X\) denotes that individual's predictors. The target risk for this fitted model is the expected loss in the intended population, conditional on the training data.

If the test individuals are independent draws from that target population and the model \(f\) is fixed without using their outcomes, each test loss has this same conditional expectation. Provided the loss has a finite expectation, linearity of expectation gives:

Thus, under these assumptions, the average test loss is unbiased for the expected loss of this fitted model in the target population, conditional on the training data. This argument does not say that a particular test-set estimate will equal the population risk; sampling variation remains. Nor does it show that the model will perform equally well in a population with a different outcome distribution or predictor distribution.

Using the test set repeatedly to choose among models changes the situation. For example, suppose several candidate models have test loss estimates \(T_1,\ldots,T_K\), and the model with the smallest estimate is selected. The selected estimate is \(\min_k T_k\). Since \(\min_k T_k\leq T_j\) for every candidate \(j\), it follows that \(E[\min_k T_k]\leq E[T_j]\) for every \(j\), and hence \(E[\min_k T_k]\leq\min_k E[T_k]\). The smallest observed test loss can therefore be optimistically low relative to the best candidate's true expected loss. The test outcomes have become part of model selection, rather than an independent evaluation of a fixed choice.

Common Mistakes and Practical Interpretation

• Calling training performance validation. Performance calculated on the individuals used to fit parameters is not a held-out estimate. Report clearly which individuals contributed to each calculation.
• Making choices after checking test outcomes. Changing predictors, model settings, or a classification threshold in response to test results uses the test set for development. The final reported test performance then reflects that reuse.
• Assuming a random split answers every population question. A random split generally evaluates performance for a population like the one sampled. It does not establish transportability to a different ancestry, clinical setting, or sampling scheme.
• Reporting a metric without its definition or units. State how predictions were evaluated. For example, RMSE for a quantitative trait is reported in trait units; classification accuracy should be accompanied by the rule used to assign classes.
• Treating one estimate as certainty. A finite test set gives an estimate with sampling variability. A single number does not imply identical performance for every individual or future sample.