Cross-Validation for Genetic Prediction

Why Use Cross-Validation?

A genetic prediction model can look promising when its performance is measured on the same people used to estimate its weights. That apparent performance does not, by itself, tell us how well predictions will work for new members of the intended population. An independent holdout set addresses this question by reserving people who are not used to fit the model. Cross-validation uses the available sample more than once: each person is held out for prediction in turn, while the remaining people are used to fit the model.

The result is an estimate of the performance of a prediction procedure, not a guarantee about a particular future sample. In genetic prediction, that procedure might include fitting weights for a polygenic score or fitting a model that uses a score and other predictors. The population and prediction target still matter: cross-validation within one sample does not establish performance in a different ancestry group, clinical setting, or time period.

Constructing the Cross-Validation Estimate

Let \(D\) denote the data used for validation, containing outcomes \(y_i\) and predictors such as genetic data \(g_i\), for \(i=1,\ldots,n\). Divide the observation indices into folds \(I_1,\ldots,I_k\). For fold \(j\), fit the specified prediction procedure using all observations outside \(I_j\). Write the resulting prediction function as \(\hat f_{-j}\). For every \(i\in I_j\), the out-of-fold prediction is \(\hat y_i^{(-j)}=\hat f_{-j}(g_i)\). The superscript indicates that the model producing this prediction excluded fold \(j\).

For a quantitative outcome, a common summary is the cross-validated mean squared error (MSE), the average squared difference between each observed outcome and its out-of-fold prediction. Its square root is the root mean squared error (RMSE), expressed in the outcome's units.

Here \(j(i)\) is the fold containing observation \(i\). Each person's outcome is compared with exactly one out-of-fold prediction in a single run of ordinary k-fold cross-validation. The MSE is in squared outcome units; the RMSE is in the same units as the outcome. These summaries are useful for illustrating cross-validation arithmetic; other performance measures may be appropriate for other prediction targets.

The interpretation requires a clear target and a defensible split. The held-out observations should represent the population to which the estimate is intended to apply, and the observations treated as separate units should be suitable for the chosen splitting scheme. Each held-out observation must be excluded from the model that predicts it. If genetic weights or other model components are estimated using a separate external data set, the validation interpretation is conditional on those supplied components; if they are estimated from the validation sample, they must be estimated anew within each training fold.

Worked Examples

Fold-Wise Scores and Pooled Out-of-Fold Predictions

There are two common ways to report cross-validation performance. One is to calculate a metric separately within each fold and average the \(k\) fold-specific values. The other is to collect all out-of-fold predictions, then calculate the metric once across the \(n\) observations. The report should say which was used. For squared error, the pooled MSE equals a fold-size-weighted average of fold MSE values. If all folds have equal size, it also equals their simple average. Pooled RMSE is the square root of pooled MSE, and is generally not the simple average of fold RMSE values.

This difference is not merely a rounding issue. Fold-wise averaging weights folds equally, even if their sizes differ; pooling weights observations equally. Moreover, applying a nonlinear transformation such as a square root before averaging can change the result even when fold sizes are equal. If the intended summary is typical error per observation, a pooled metric is often natural. If equal weight for each fold is intended, report the fold-wise average and name it as such.

Thus the pooled MSE is a weighted average with weights proportional to fold size. The pooled RMSE is \(\sqrt{\sum_j n_j\mathrm{MSE}_j/n}\), whereas the unweighted mean fold RMSE is \(\sum_j\sqrt{\mathrm{MSE}_j}/k\). Because these are different operations, they need not agree.

Choosing the Fold Count and Repetition Scheme

There is no universally best value of \(k\). Common choices are five or ten folds, but the appropriate choice depends on sample size, computational cost, and the prediction procedure. With smaller \(k\), each model is trained on a smaller fraction of the available observations, and each validation fold is larger. With larger \(k\), training sets are closer to the full sample size, but each fold is smaller and more model fits are required. For very small samples, leave-one-out cross-validation sets \(k=n\), fitting \(n\) models; it can be computationally demanding, and its estimate can be sensitive to individual observations.

Repeated k-fold cross-validation creates new valid partitions and repeats the fit-and-predict process. Averaging results across repetitions reduces sensitivity to one arbitrary fold assignment. It does not create new participants or independent validation studies: the same people recur across repetitions, so the repeated results should not be treated as independent observations. Repetitions also cannot repair a mismatch between the validation sample and the population where predictions will be used.

For reproducibility, report \(k\), the number of repetitions, how folds were assigned, and whether the reported metric is pooled or an average of fold-specific metrics. In genetic prediction, also state the population and outcome used for validation. A performance estimate is interpretable only in relation to those design choices.

Common Mistakes and Why Cross-Validation Matters

• Scoring the training data as if it were held out. Predictions from a model fitted on the same individuals do not serve as out-of-fold predictions. Use the model that excluded the person's fold.
• Calling every average a pooled score. An unweighted mean of fold RMSE values is not generally the RMSE from all out-of-fold predictions. State the calculation and its weighting.
• Assuming more folds must mean a better estimate. Increasing \(k\) increases training-set size but also changes validation-fold size and computational burden; no choice is best in every design.
• Treating repetitions as additional independent participants. Repeated splits reuse the same sample. Repetition checks sensitivity to partitioning; it does not multiply the information in the validation population.
• Interpreting internal validation as transportability. Cross-validation estimates performance under the validation sampling and splitting design. It does not establish performance in a different target population.

Cross-validation is useful when a separate holdout would leave too little data for model fitting or when performance should be checked across multiple partitions. Its strength is systematic reuse of the sample for training and held-out prediction. Its limits are equally important: the estimate belongs to a specified prediction procedure, training-set size, validation population, and scoring rule.