Repeated Random Holdout Validation

Why Repeat a Random Holdout?

A single train-test split gives one evaluation of a regression procedure, but its score can depend on which observations happen to land in the test set. A different random split may produce a different score. Repeating the split makes it possible to summarize performance across several partitions rather than relying on just one particular division of the available data.

In repeated random holdout validation, the data are randomly divided into a training set and a test set multiple times. The regression procedure is fitted using the training set in each repetition, and its predictions are evaluated on that repetition’s test set. The resulting scores are then summarized, commonly by their arithmetic mean. This method is also called repeated random subsampling validation.

Definition: In repeated random holdout validation, each repetition creates a random, disjoint training set and test set from the available observations. The regression procedure is fitted anew on the training set, evaluated on the test set, and assigned a performance score. Repeating this process gives a collection of holdout scores to summarize.

Procedure and Score Aggregation

Let \(B\) be the number of repetitions. In repetition \(b\), write \(T_b\) for the training set and \(V_b\) for the test set. Within that repetition, no observation belongs to both sets. Fit the chosen regression procedure using only \(T_b\), then calculate predictions for the observations in \(V_b\). If the test set contains \(m_b\) observations, its mean squared error is

\operatorname{MSE}_b = \frac{1}{m_b} \sum_{i\in V_b}(y_i-\hat{y}_{i,b})^2.

Here \(y_i\) is the observed outcome and \(\hat{y}_{i,b}\) is the prediction for observation \(i\) from the procedure fitted in repetition \(b\). The root mean squared error is \(\operatorname{RMSE}_b=\sqrt{\operatorname{MSE}_b}\), and the mean absolute error is the average of \(|y_i-\hat{y}_{i,b}|\) over that test set. Choose a metric suited to the prediction task and use the same metric in every repetition.

For a metric \(S_b\) calculated in repetition \(b\), the simple aggregate is the mean score:

\bar{S} = \frac{1}{B}\sum_{b=1}^{B}S_b.

For error metrics, smaller values indicate better predictions. The units depend on the metric: RMSE and MAE have the units of the outcome, while MSE has squared outcome units. An average of RMSE values is not generally the same as the square root of the average MSE. These are different summaries and should be named accurately.

Set the evaluation plan.
Specify the regression procedure, the training and test proportions, the performance metric, and the number of repetitions before examining the results.

Generate a random split.
Divide the observations into disjoint training and test sets. Use a new random split for the next repetition.

Fit and evaluate.
Fit the procedure on that repetition’s training observations and calculate its predictions and test score on that repetition’s test observations.

Summarize the scores.
Report the individual scores or a suitable aggregate, such as their mean, with the metric and its units.

Worked Example: Calculating and Averaging Holdout Scores

Suppose a hypothetical data set contains independently sampled cases from the population to which predictions will be applied. A regression procedure is evaluated using two random holdout splits, each with three test cases. The table gives observed outcomes and predictions from the model fitted on that split. Outcomes and predictions are measured in points.

Repetition	Observed test outcomes	Predicted outcomes	Prediction errors
1	12, 9, 15	10, 10, 12	2, −1, 3
2	8, 13, 11	7, 15, 9	1, −2, 2

For the first split, the squared errors are \(2^2=4\), \((-1)^2=1\), and \(3^2=9\). Therefore,

\operatorname{MSE}_1 = \frac{4+1+9}{3} = \frac{14}{3} \approx 4.667\text{ points}^2, \qquad \operatorname{RMSE}_1 = \sqrt{\frac{14}{3}} \approx 2.160\text{ points}.

For the second split, the squared errors are \(1\), \(4\), and \(4\), so

\operatorname{MSE}_2 = \frac{1+4+4}{3} = 3.000\text{ points}^2, \qquad \operatorname{RMSE}_2 = \sqrt{3} \approx 1.732\text{ points}.

The mean of the two RMSE scores is \((2.160+1.732)/2=1.946\) points. This summarizes the average RMSE across these two splits. By contrast, the mean MSE is \((4.667+3.000)/2=3.833\) points squared, whose square root is about \(1.958\) points. The two summaries differ because taking a square root and averaging do not generally commute.

The calculations evaluate predictions on observations not used to fit the corresponding model: the within-split separation condition is met. The hypothetical cases were stipulated to be independent draws from the intended target population, supporting use of random splits for this prediction task. In an actual application, the sampling design and target population would need to support that interpretation.

What Repeated Splits Do and Do Not Guarantee

A new random split does not create new observations. An observation can be in the test set in several repetitions, in the training set in several repetitions, or never be used for testing. Test sets from different repetitions can overlap, and the fitted models also use many of the same observations. Consequently, repeated holdout scores are not evaluations on entirely separate data sets.

A short calculation makes the coverage issue clear. Suppose every repetition independently assigns each observation to the test set with probability \(q\), and there are \(B\) repetitions. For a particular observation, the probability of not appearing in any test set is \((1-q)^B\). Thus its probability of appearing in at least one test set is

P(\text{tested at least once}) = 1-(1-q)^B.

This calculation assumes independent random assignments across repetitions and the same test probability \(q\) each time. It describes test-set coverage; it is not a confidence interval or a measure of uncertainty in the aggregated score.

Worked Example: How Often Might an Observation Be Tested?

Suppose each split assigns about 25% of observations to the test set, and the procedure is repeated four times. For a particular observation, use \(q=0.25\) and \(B=4\). The probability of never being tested is

$$ (1-q)^B = (1-0.25)^4 = 0.75^4 = 0.3164. $$

Therefore, the probability of being tested at least once is \(1-0.3164=0.6836\), or about 68.4%. The expected number of test-set appearances is \(Bq=4(0.25)=1\), but that does not mean every observation appears exactly once. Some may appear multiple times, while others are never tested. This differs from K-fold cross-validation, which assigns each observation to a test fold once per complete K-fold run.

Repeated Holdout Compared With Repeated K-Fold

Both methods repeatedly fit a regression procedure and evaluate it on observations held out from that fit. Their test-set construction is different. In repeated random holdout, each repetition independently draws a training-test partition at the chosen proportions. The test sets across repetitions need not cover every observation and may overlap substantially.

In repeated K-fold cross-validation, the observations are divided into \(K\) folds for a complete run. Each fold serves as the test fold once, while the other folds form the training set. Repeating K-fold cross-validation creates new fold assignments and repeats that full cycle. Thus, within each complete run, each observation is tested once; repeated random holdout has no such coverage guarantee.

Feature	Repeated random holdout	Repeated K-fold cross-validation
Test-set construction	Random test set drawn separately in each repetition	Each run partitions observations into \(K\) folds
Test coverage within one run	Some observations may be tested more than once or not at all	Each observation is tested once per complete run
Training proportion	Chosen directly for each split	Determined by the number of folds
Typical summary	Mean of the holdout scores	Mean of the fold scores, often summarized across repeats

Repeated random holdout can be useful when a particular training fraction and test fraction are desired, or when it is convenient to generate many random partitions. Repeated K-fold provides structured coverage of the observations within each run. Neither method eliminates the need to choose a validation design that matches the intended prediction setting.

Worked Example: Choosing a Validation Design

Suppose a hypothetical prediction task has 120 independent observations and the goal is to assess a procedure trained on roughly 80% of the available observations. One option is repeated random holdout with a 80% training fraction and 20% test fraction, repeated several times. Each repetition directly matches the desired fraction, but some observations may recur in test sets and others may not be tested.

Alternatively, five-fold cross-validation uses about 80% of the observations for training in each fold and about 20% for testing. Across one complete run, all 120 observations are tested once. Repeating the five-fold procedure changes the fold assignment and gives another set of scores. Both designs have approximately the same training fraction here, but their test-set coverage structures differ.

In either design, the regression procedure must be refitted within each training set. The test observations for a split must not influence that split’s fitted predictions; otherwise the score no longer evaluates predictions on held-out cases. Also check that the split matches the data structure: randomly separating rows is not suitable when the intended task requires prediction for future time periods or for entirely new groups. A split should reflect the cases on which predictions will actually be used.

Interpreting the Aggregate and Avoiding Pitfalls

The mean holdout score describes the average score across the particular random partitions used, for a procedure trained on the chosen fraction of the data. It is not automatically the prediction error of a final model fitted using all available observations, nor does it guarantee the same performance in every future setting. The training-set size matters: a procedure trained on a smaller fraction may perform differently from one trained on the full data set.

The score is meaningful for a target population only when the split design and data collection support that connection. For ordinary randomly sampled, independent cases, random partitions may be appropriate. If observations are clustered, repeated measures on the same person, or ordered in time, a split that puts related or future information in both training and test sets can give an evaluation that does not match the real prediction task. Define the target prediction task before choosing the split.

Keep the metric and its units visible when reporting results. For example, “mean RMSE across 20 random holdouts was 3.4 points” identifies the summary, number of repetitions, and scale. Reporting just “the error was 3.4” does not say which metric was used or how it was aggregated.

Common pitfall: “There were 20 holdout scores, so the data were tested on 20 independent test sets and the average is the error of the final full-data model.” This reasoning is incorrect. Random holdout test sets can overlap, so the scores are not evaluations on independent samples, and each fitted model was trained on only the chosen training fraction. The mean summarizes those repeated holdout evaluations; it does not by itself establish the performance of a full-data fit or quantify uncertainty.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

In repeated random holdout validation, what must be true of the training and test sets within each repetition?
A test set has squared prediction errors of 1, 9, and 6. Calculate its MSE and RMSE, including the appropriate units if the outcome is measured in kilograms.
Why is the mean of several RMSE scores generally different from the square root of the mean of the corresponding MSE scores?
With a test fraction of 0.20 and five independent repetitions, calculate the probability that a particular observation is never selected for testing.
State one difference between test-set coverage in repeated random holdout and in one complete K-fold cross-validation run.
Why should a random row-wise split be reconsidered when predictions are intended for future time periods or new groups?