Repeated Random-Split Validation

Why Repeat a Random Split?

A single random train-test partition gives one estimate of predictive performance. A different partition of the same data may give a different estimate: some observations that are difficult to predict may land in the test set in one split but not another. Repeated random-split validation makes this sensitivity visible by fitting and evaluating the same learning procedure on several independently generated partitions.

The result has two parts that should be reported separately: the average test error across splits, and the variation in test error from split to split. The average describes the central result under the chosen splitting procedure. The variation describes how sensitive that result is to the random partition. Neither quantity, by itself, is a confidence interval for performance on a new population.

The loss \(L_r\) must match the performance question. For regression it could be mean squared error (MSE) or mean absolute error; for classification it could be misclassification rate. A smaller value is better for these loss measures. Define the metric, the training fraction, and the random-split procedure before comparing results. When test-set sizes differ, also decide whether the target is the average error for a randomly selected split or the error pooled over all test predictions. Those are not generally the same quantity.

What the Mean and Standard Deviation Describe

The mean \(\bar L\) summarizes the errors observed under the chosen split procedure. The standard deviation \(s_L\) summarizes their dispersion in the same units as the loss. For example, if test error is a proportion, its standard deviation is also measured in proportion units; if the metric is an MSE in squared response units, the standard deviation is in those squared units.

Repeated splits reuse the same observed data. Consequently, \(s_L\) is a measure of split sensitivity conditional on the data set and procedure; it is not a measure of all uncertainty about future predictive performance. In particular, repeating the split does not create new independent observations from the population, remove limitations in the observed data, or correct a poorly chosen model procedure. More repetitions can make the estimated split-distribution mean less affected by Monte Carlo randomness, but they do not make the original data set larger.

If the split assignments are independently generated, then, conditional on the fixed data and a deterministic fitting procedure, the split losses are independent draws from the split-randomization distribution. In that setting, \(s_L/\sqrt{R}\) describes the Monte Carlo standard error of the estimated mean over that distribution. It does not measure uncertainty from drawing a different data set from the population. If the fitting procedure itself uses randomness, that randomness must also be specified: either hold it fixed to isolate split variation, or repeat it as part of the procedure whose variability is being summarized.

Worked Examples

Why the Sample Variance Uses \(R-1\)

The sample variance formula uses \(R-1\), rather than \(R\), because the same data are used to calculate both the sample mean and the deviations from it. The following justification shows the adjustment when the split losses can be treated as independent draws from a distribution with mean \(\mu_L\) and variance \(\sigma_L^2\). This is a statement about repeated split randomization under that setup; it does not assert that the split losses describe all uncertainty in predictive performance.

To verify the identity, write \(L_r-\bar L=(L_r-\mu_L)-(\bar L-\mu_L)\), square, and sum over \(r\). The cross-product is zero because \(\sum_{r=1}^R(L_r-\mu_L)=R(\bar L-\mu_L)\). Taking expectations, the first sum on the right has expectation \(R\sigma_L^2\). The second term has expectation \(R\operatorname{Var}(\bar L)=R(\sigma_L^2/R)=\sigma_L^2\), using independence and equal variances. Therefore:

Dividing by \(R-1\) thus makes the sample variance unbiased for the split-randomization variance under these assumptions. Dividing by \(R\) instead would systematically underestimate that variance on average. This justification does not turn \(s_L\) into a population uncertainty measure; it explains the denominator used to summarize variation across the repeated splits.

Common Traps and Practical Interpretation

• Calling split variation population uncertainty. A small \(s_L\) means the observed data set produced similar results under the chosen partitions. It does not establish that the same error will hold in a new population.
• Ignoring test-set size. The unweighted mean of split rates and the pooled rate answer different questions when test sizes differ. State which one is being reported.
• Confusing averaging scales. Mean RMSE is the average of the split-specific square roots of MSE. It generally differs from the square root of average MSE.
• Assuming more repetitions fix a weak evaluation design. Increasing \(R\) gives a more stable summary of the selected split procedure. It does not undo an unsuitable split fraction or make the available observations more representative.
• Leaving the procedure unspecified. The reported result depends on the metric, training fraction, split mechanism, and fitting procedure. If the goal is to assess a full modeling procedure, each split must evaluate that procedure as it would actually be applied using training data.

Repeated random-split validation is most informative when its summary is read as a pair: the average error indicates typical performance under the specified partitioning rule, while the standard deviation indicates how much that estimate changes across partitions. A low mean with a high standard deviation signals that the central estimate alone hides substantial split sensitivity. A low standard deviation with a high mean indicates stable but poor performance under the chosen metric.