Why Reproducible Resampling Matters
A train/test split or a set of cross-validation folds is part of the experiment, not a disposable implementation detail. If the split changes between candidate models, a difference in validation scores can reflect different validation observations rather than a genuine difference between models. Reproducing the resampling design lets an analyst inspect, rerun, and explain the comparison.
In R, setting a random seed makes a sequence of random-number operations repeatable under specified conditions. It is useful, but it is not the whole reproducibility plan. The safest practice is to save the actual row identifiers assigned to each partition or fold, then reuse those assignments. A seed helps regenerate a design; a saved design records what was actually used.
Seeds: What They Reproduce
R's random-number generator has an internal state. A call such as set.seed(731) initializes that state. If the same random operations are then performed in the same order, on the same R setup and inputs, the generated sequence can be reproduced. For example, sampling row numbers after setting the seed will ordinarily give the same result when the code is rerun under those same conditions.
A seed does not mean that every future operation in an analysis will use the same random values regardless of what happens in between. An extra random draw before the split advances the generator and changes the later result. Nor does a seed, by itself, guarantee identical results across different R versions, random-number-generator settings, packages, data ordering, or parallel computing configurations. In particular, R's sampling behavior has changed across versions. Record relevant environment details when exact regeneration matters.
Place the seed immediately before the random operation it is intended to control. Avoid repeatedly resetting the same seed inside a loop: doing so can restart the random sequence at each iteration rather than producing fresh draws. For model comparisons, create the folds once and apply that saved assignment to every candidate model. A stochastic fitting algorithm may also use random numbers; its seed is a separate concern from the fold assignment.
Worked Example: Repeating a Random Split
A hypothetical analyst has 10 ordered records and wants a training sample of 7 records and a test sample of 3. The analyst wants to confirm that the split-generation code is repeatable on the current R setup.
Method. Set the seed immediately before sampling row positions, then repeat the same operation after resetting the seed. This tests whether the split-generation step is reproducible under the current conditions.
R code: n = 10; set.seed(731); train_a = sample.int(n, size = 7); test_a = setdiff(seq_len(n), train_a); set.seed(731); train_b = sample.int(n, size = 7); identical(train_a, train_b)
The sample size is 7, so the complement has \(10-7=3\) row positions. The sampled positions are distinct because the default sampling is without replacement. Resetting the seed before the second call restores the initial random-number state; the operation and its inputs are unchanged. Therefore identical(train_a, train_b) returns TRUE on the same setup. The test positions also match because each is computed as the complement of the corresponding training positions.
The method avoids assuming that a seed number alone documents the final split. The analyst should save the resulting row identifiers, not just the code that generated them. A different R version or a changed sequence of random calls could otherwise produce different positions even though the seed is unchanged.
Store the Assignment, Not Just the Seed
Once a split has been generated, retain identifiers that refer to the original observations. Integer row positions are adequate only while row order is unchanged. A stable claim ID, policy ID, or other unique identifier is safer if records may later be sorted, filtered, or joined. Before applying a saved assignment, check that its identifiers still match the intended data and that each appears exactly once where required.
For cross-validation, store a fold label for every analysis observation, or store the validation-row identifiers for each fold. A fold-label vector is compact and makes it easy to derive the training and validation indices. If grouped or time-ordered resampling is required, the saved assignments must also preserve that design; a seed cannot make an inappropriate random split appropriate.
Choose a stable observation identifier and confirm that it is unique. Record the data order if integer positions will be used.
Set the seed immediately before creating the split or folds, and create the assignment using the intended validation design.
Confirm every required observation is assigned, no observation is assigned to two folds, and fold sizes or constraints are as intended.
Save identifiers, assignment labels, seed, relevant environment details, and the data version. Use the same saved assignment for every model being compared.
The following example uses an explicit assignment so every fold membership can be inspected directly. In a live analysis, the labels might be created by a seeded random procedure, but after creation the saved labels—not a fresh sampling call—should drive the model comparison.
Worked Example: Recording and Checking Cross-Validation Folds
A hypothetical portfolio contains 12 observations identified by row positions 1 through 12. It will be evaluated with three folds. The saved fold-label vector is fold_id = c(1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3), where the label at each position gives that observation's validation fold.
Method. Store the assignment and check each validation set and its complement. In R, the validation positions for fold \(k\) are obtained with which(fold_id == k); the training positions are those with a different label.
Fold 1 contains positions 1, 4, 7, and 10. Fold 2 contains positions 2, 5, 8, and 11. Fold 3 contains positions 3, 6, 9, and 12. Each fold has \(4\) observations, and \(4+4+4=12\), so the labels cover all observations. The sets do not overlap because each position has exactly one label.
For fold 2, the validation set is \(\{2,5,8,11\}\), and the training set is its complement, \(\{1,3,4,6,7,9,10,12\}\), containing \(12-4=8\) observations. The same construction gives 4 validation and 8 training observations for each fold.
R code: fold_id = c(1, 2, 3, 1, 2, 3, 1, 2, 3, 1, 2, 3); fold_sets = lapply(1:3, function(k) which(fold_id == k)); lengths(fold_sets) returns c(4, 4, 4). Saving fold_id along with the observation identifiers makes the assignments reusable. The common mistake avoided is regenerating folds separately for each model, which would make the score comparison depend on different validation observations.
Using Shared Folds to Compare Models
When candidate models are evaluated on the same folds, each model faces the same validation observations in each fold. This controls a source of variation in the comparison. The fold assignment should be generated once, saved, and passed into each model's evaluation code. Do not infer that two models were evaluated on shared folds merely because both used the same number of folds.
Suppose \(K\) folds partition the \(n\) observations, and let \(L_{jk}\) be the average loss for model \(j\) on validation fold \(k\). If the folds have equal size, the average of the fold losses is also the loss averaged over all observations, provided each observation contributes one loss and the metric is an arithmetic mean:
Here, \(F_k\) is the set of validation observations in fold \(k\), \(m\) is the common fold size, and \(\ell_{jr}\) is the loss for model \(j\) on observation \(r\). The last equality holds because the folds partition the observations, so every observation's loss appears exactly once in the sum. If folds have unequal sizes, an unweighted average of fold-level means does not generally equal the observation-level mean; weight each fold mean by its validation-set size when a pooled mean is intended.
Not every metric can be averaged this way. In particular, RMSE applies a square root after averaging squared errors. Averaging fold RMSE values is generally different from calculating one pooled RMSE from all squared errors. State which calculation is being used and keep it consistent across candidate models.
Worked Example: Comparing Two Models on the Same Saved Folds
A hypothetical six-record data set has IDs Q1 through Q6. Its saved fold labels are c(1, 2, 1, 2, 1, 2). Two models produce the following residuals, defined as observed value minus predicted value: Model A has c(1, -2, 0, 1, -1, 2); Model B has c(0.5, -1, 1, 2, -0.5, 0). Compare their average fold RMSE and pooled RMSE.
Method. Use the same saved fold labels for both models. For each fold, square its residuals, average the squared residuals, and take the square root to obtain fold RMSE. For pooled RMSE, first combine all squared residuals, divide by 6, and then take the square root.
Fold 1 contains Q1, Q3, and Q5; fold 2 contains Q2, Q4, and Q6. For Model A, fold 1 squared residuals are \(1^2=1\), \(0^2=0\), and \((-1)^2=1\). Their mean is \((1+0+1)/3=2/3\), so fold 1 RMSE is \(\sqrt{2/3}\approx0.8165\). Fold 2 squared residuals are \((-2)^2=4\), \(1^2=1\), and \(2^2=4\). Their mean is \(9/3=3\), so fold 2 RMSE is \(\sqrt{3}\approx1.7321\). The unweighted average fold RMSE is \((0.8165+1.7321)/2=1.2743\), rounding to four decimals.
For Model B, fold 1 squared residuals are \(0.5^2=0.25\), \(1^2=1\), and \((-0.5)^2=0.25\). Their mean is \(1.5/3=0.5\), giving RMSE \(\sqrt{0.5}\approx0.7071\). Fold 2 squared residuals are \((-1)^2=1\), \(2^2=4\), and \(0^2=0\). Their mean is \(5/3\), giving RMSE \(\sqrt{5/3}\approx1.2910\). The average fold RMSE is \((0.7071+1.2910)/2=0.99905\), or \(0.9991\) to four decimals.
For the pooled calculation, Model A has total squared error \(2+9=11\), so pooled RMSE is \(\sqrt{11/6}\approx1.3540\). Model B has total squared error \(1.5+5=6.5\), so pooled RMSE is \(\sqrt{6.5/6}\approx1.0408\). Both calculations favor Model B because its RMSE is lower, but their numerical values differ.
Common mistake and correction. Treating Model A's average fold RMSE, \(1.2743\), as its pooled RMSE is wrong: the correct pooled RMSE is \(1.3540\). The square root is nonlinear, so taking a square root within each fold and then averaging does not give the square root of the pooled mean squared error. Use the metric definition requested, and apply the same calculation to both models.
What to Preserve for a Defensible Rerun
A useful resampling artifact should make it possible to answer both “what was assigned?” and “what did this assignment apply to?” At minimum, preserve:
- Stable observation identifiers and, if using row positions, the exact data ordering.
- The train/test membership or the fold label for every observation.
- The seed and the code that generated the assignment, as supporting provenance.
- The R version, relevant random-number-generator settings, package versions, and data version or a suitable data fingerprint.
- Any constraints defining the resampling design, such as grouping or chronological ordering.
Save the design in a file that can be loaded without regenerating it. For example, an R list containing the observation IDs, fold labels, seed, and R version can be written with saveRDS(design, "resampling_design.rds") and later read with readRDS("resampling_design.rds"). After loading, check that the identifiers match the data being analyzed and that the assignment has the expected number of observations and folds. Saving a design does not remove the need to prevent data leakage: preprocessing must still be fitted using only the training portion of each resample.
Check Your Understanding
Work through these original problems. Assume lower loss is better where model losses are given.
- A data set has 18 ordered rows. A split uses 13 training rows and the remaining rows for testing. How many test rows are there, and what should be saved in addition to the seed? (Answer: 5 test rows; save the actual row identifiers or indices for both partitions, plus the data order or stable IDs.)
- A fold-label vector for 9 observations is c(1, 2, 3, 1, 2, 3, 1, 2, 3). Give the validation IDs in fold 2 and the training IDs for that fold. (Answer: Validation IDs 2, 5, 8; training IDs 1, 3, 4, 6, 7, 9.)
- Two models have fold mean squared errors 4 and 9 on two equal-sized folds. What is the average fold RMSE, and what is the pooled RMSE? (Answer: Average fold RMSE \(=(\sqrt{4}+\sqrt{9})/2=2.5\); pooled RMSE \(=\sqrt{(4+9)/2}=\sqrt{6.5}\approx2.5495\).)
- Three equal-sized folds give Model A losses \(0.12, 0.18, 0.15\) and Model B losses \(0.14, 0.16, 0.19\). Calculate each average fold loss and identify the lower-loss model. (Answer: Model A \(=(0.12+0.18+0.15)/3=0.15\); Model B \(=(0.14+0.16+0.19)/3=0.1633\); Model A is lower.)
- An analyst sets the same seed before generating folds for each of two models but adds a random operation before generating the second model's folds. Why may the assignments differ, and what is the robust correction? (Answer: The extra random operation advances the random-number state; generate folds once, save the assignment, and reuse it for both models.)