Random Splits and Reproducible Seeds

Why a Random Split Needs a Record

A holdout evaluation depends on which observations are assigned to training and which are reserved for testing. Two analysts can use the same data and the same regression method yet obtain different test scores if their partitions differ. Random assignment gives a principled way to form a partition, while recording the random seed and other details allows the assignment to be reproduced.

Reproducibility is useful for checking calculations, sharing an analysis, and tracing how a result was obtained. But a reproducible split is not automatically a representative split, and a seed is not a device for improving a model. In particular, searching for a seed that produces a favorable test score turns the test set into part of the model-selection process.

What Random Assignment Means

Suppose there are \(n\) distinct observations and a test set of fixed size \(m\). One common procedure randomly permutes the observations, assigns the first \(m\) to the test set, and assigns the remaining \(n-m\) to training. If the permutation is uniform, every subset of \(m\) observations has the same chance of being the test set. There are \(\binom{n}{m}\) such subsets, so the probability of any particular test subset is \(1/\binom{n}{m}\).

This statement describes the randomization procedure, not a guarantee that every realized split will resemble the full data set. For example, a small test set may by chance contain unusually high or low outcomes. A random split also does not repair problems in how the data were collected or make observations independent when they are not. Its usefulness depends on the design and the target population to which prediction performance is meant to apply.

Under a uniform fixed-size split, each observation has probability \(m/n\) of being in the test set: the test set contains \(m\) of the \(n\) observations, with no observation favored by the assignment mechanism. That symmetry is a property of the random procedure. It does not mean that the realized test cases are guaranteed to be typical.

How a Seed Makes a Split Reproducible

A computer’s pseudorandom generator is deterministic once its initial state is specified. Setting a seed initializes that state. If the same generator then receives the same inputs and is used in the same order, it produces the same sequence of pseudorandom values. A split routine that uses that sequence to permute observations will consequently produce the same membership assignment.

The seed is therefore best understood as a reproducibility control, not as a description of a particular test set that will be the same in every software package. Different software, different versions, or different randomization routines may use different algorithms. Even within one program, other random-number operations performed before the split can change which portion of the sequence the split routine uses.

For a reproducible analysis, record more than just the seed. At minimum, preserve the data version and row order, the test-set size or proportion, the split procedure, the software and relevant implementation, and the seed. If the analysis is rerun after data rows have been added, removed, or reordered, the same seed need not assign the same named observations to the test set.

Test Scores and the Cost of Searching

For regression, one common test-set measure is mean squared error (MSE). If the test set contains \(m\) observations, let \(y_i\) be the observed outcome and let \(\hat{y}_i\) be the prediction for test observation \(i\), produced by a model fitted using the training data. Then

Each residual \(y_i-\hat{y}_i\) is measured in the outcome’s units, so test MSE is measured in squared outcome units. A smaller test MSE is a better score on that particular test set. It is not, by itself, proof that the model will have lower prediction error on future cases.

The intended role of the test set is to provide an evaluation that was not used to choose the model or otherwise steer the analysis. If many seeds are tried, each split produces a different training set and test set, and therefore a different fitted model and test score. Choosing the seed with the smallest test MSE uses those test results to make a selection. The selected score is then likely to look better than the performance one would obtain from a split chosen without looking at its score.

The inequality follows because every score is at least as large as the minimum; adding those \(k\) inequalities and dividing by \(k\) gives the result. If the scores differ, the inequality is strict. Thus selecting the minimum from several noisy evaluations systematically favors a low observed value. The selected score need not be lower in every possible set of trials, but the selection rule favors low scores by construction. This is a form of selection bias, sometimes described as a winner’s curse.

A Practical Reproducibility Procedure

A simple record-keeping routine makes the randomization auditable without treating the seed as a tuning parameter.

When a Split Looks Unfavorable

A test score can vary because the test observations differ, the training observations differ, or both. A poor-looking score is not a reason to keep changing the seed until a better one appears. First check that the split was implemented as intended and that the prediction calculation is correct. If the analysis plan allows a different evaluation strategy, choose that strategy based on the study’s goals rather than on which seed produces the most attractive result.

Randomization is valuable because it avoids deliberately assigning observations according to their outcomes or predictions. Reproducibility is valuable because it lets another analyst reconstruct the assignment. These benefits do not imply that one random split is uniquely correct. The split estimates performance for a particular training-and-test partition, and its result can vary with the partition.

Common Mistake

Wrong reasoning: “I tried several seeds and kept the one with the lowest test MSE, so this is my best unbiased estimate of predictive performance.” The seed may indeed be recorded, and the selected partition may be reproducible, but the score was used to select that partition.

Correction: Choose the split procedure and seed without looking for a favorable test score, then report the resulting evaluation with its context. Reproducibility means that the assignment can be repeated; it does not remove the optimism created by selecting an evaluation because it scored well. Keep the distinction clear: the seed controls a pseudorandom sequence, while the evaluation’s credibility depends on how the test results were used.