Why Random Holdout Splits Need Care
A random holdout split assigns records to separate data partitions so a model can be fitted and assessed on different rows. In a three-way split, the training partition is used to fit the model, the validation partition can support model choices, and the test partition is reserved for evaluation. A split is not valid merely because the code ran: every eligible row must be assigned once and only once, and the assignment must be reproducible when that is required.
This tutorial concerns ordinary row-level random assignment. It assumes that the rows are appropriate units to assign independently for the prediction task. If records are repeated observations of the same entity, ordered in time, or subject to another structural constraint, an unrestricted row-level random split may not be appropriate. Matching partitions to the prediction target must come first.
Constructing a Random Split in R
The safest basic pattern is to sample the row indices, not separate columns of the data. If the data frame has \(N\) rows, the indices are the integers from 1 through \(N\). Randomly permute those indices, then take consecutive sections of that permutation for the three partitions.
In R, sample.int(N, size = N, replace = FALSE) returns all \(N\) row indices in a random order, with no replacement. The arguments make the intended operation explicit: draw \(N\) integers from 1 through \(N\), without selecting any index twice. If the desired counts are \(n_{\mathrm{train}}\), \(n_{\mathrm{valid}}\), and \(n_{\mathrm{test}}\), they must be nonnegative integers satisfying
For example, after defining the counts and the permutation, the allocation can be written as follows. Using seq_len() also handles a partition of size zero, because seq_len(0) is an empty integer vector.
perm <- sample.int(N, size = N, replace = FALSE)
train_id <- perm[seq_len(n_train)]
valid_id <- perm[n_train + seq_len(n_valid)]
test_id <- perm[n_train + n_valid + seq_len(n_test)]
These vectors contain row positions. The corresponding data can then be selected with data[train_id, ], data[valid_id, ], and data[test_id, ]. Applying the same indices to the complete data frame keeps each row’s predictors and outcome together.
Worked Example: Allocating Six Rows Without Overcommitting
A hypothetical claims study has six eligible rows. An analyst proposes four training rows, two validation rows, and two test rows. Check whether the requested allocation is possible, then make a valid split with at least one row in each partition. For the valid allocation, suppose the seeded random draw produces the permutation \(4,1,6,2,5,3\).
Step 1: Check the requested counts. Their sum is \(4+2+2=8\), which exceeds the available six rows. Therefore, keeping all three original counts is impossible.
This does not mean that four training rows are impossible whenever both holdout partitions must be positive. Four training rows could be retained by changing the holdout counts to one validation row and one test row, since \(4+1+1=6\). For this example, use the alternative allocation of three training rows, two validation rows, and one test row.
Step 2: Allocate the permutation. The first three positions, \(4,1,6\), are training indices. The next two, \(2,5\), are validation indices. The final position, \(3\), is the test index.
Step 3: Check the result. The sizes are \(3,2,1\), and the combined indices are \(4,1,6,2,5,3\). Sorting them gives \(1,2,3,4,5,6\), so every row appears exactly once. The common mistake avoided here is to treat a failed total of proposed counts as proof that a particular training count cannot be used under any allocation.
Why the Random Permutation Gives a Uniform Partition
When every permutation of the \(N\) row indices is equally likely, taking fixed-length sections of the permutation gives a uniform random partition among partitions with the specified counts. This result assumes that the sampling routine produces a uniform permutation and that the counts are fixed in advance.
To see why, fix one particular partition of the rows into training, validation, and test subsets of the required sizes. Within any one partition, the training indices can appear in \(n_{\mathrm{train}}!\) orders, the validation indices in \(n_{\mathrm{valid}}!\) orders, and the test indices in \(n_{\mathrm{test}}!\) orders. These orders can be arranged independently within the three sections of the full permutation. Thus the number of full permutations that produce this particular partition is
There are \(N!\) equally likely full permutations. Therefore, the probability of producing that particular partition is
This quantity is the same for every partition with those sizes, so the procedure does not favor one such partition over another. The argument justifies the allocation method; it does not establish that the split is suitable for every prediction target. Suitability depends on the structure of the records and the intended use of the evaluation.
Reproducibility With a Seed
R uses a pseudorandom number generator: a deterministic sequence of values that behaves like random draws for practical purposes. Calling set.seed() sets the generator’s state. If the seed, random-number-generator settings, R environment, and sequence of random operations are the same, the generated permutation is reproducible.
Set the seed immediately before the random operation that defines the split. A random operation inserted before sampling can consume values from the generator and change the resulting permutation. Reproducibility is also not the same as permanently preserving a split: R versions or random-number-generation settings can affect generated results. For an analysis that must be auditable, record the seed and the R setup, and save the resulting row indices.
Worked Example: Repeating a Seeded Three-Way Split
A hypothetical portfolio contains nine records. The analyst wants five training rows, two validation rows, and two test rows. Show how to create the split, repeat the sampling operation, and check reproducibility without depending on a particular printed permutation.
Step 1: Confirm the counts. The requested counts sum to \(5+2+2=9\), equal to the number of records.
Step 2: Draw and allocate indices. In R, use set.seed(914); perm1 <- sample.int(9L, size = 9L, replace = FALSE). Then set train_id1 <- perm1[seq_len(5L)], valid_id1 <- perm1[5L + seq_len(2L)], and test_id1 <- perm1[7L + seq_len(2L)]. Their lengths are 5, 2, and 2; the index values come from a permutation of 1 through 9.
Step 3: Repeat from the same initial state. Run set.seed(914); perm2 <- sample.int(9L, size = 9L, replace = FALSE). With the same R setup and no intervening random operation, identical(perm1, perm2) returns TRUE. Reallocating the second permutation at the same cut points also gives identical partition indices.
Step 4: Verify coverage. The three partition sizes sum to 9. Since each permutation contains every index from 1 through 9 once and the sections do not overlap in their positions, the sections together contain every row once. A direct check is still useful in production code. The common mistake avoided is assuming that a seed alone guarantees identical results after the code, R setup, or random-number settings have changed.
Checking That Rows Appear Once and Only Once
Partition checks should test more than the sum of the partition sizes. For example, a duplicate index and a missing index can leave that sum unchanged. A robust check compares the sorted combined indices to the complete sequence of valid row indices and separately checks for duplicates.
For integer row indices in R, a concise check is identical(sort(c(train_id, valid_id, test_id)), seq_len(N)). It is also helpful to check the partition sizes and duplicate count explicitly:
length(train_id) + length(valid_id) + length(test_id) == N
anyDuplicated(c(train_id, valid_id, test_id)) == 0L
identical(sort(c(train_id, valid_id, test_id)), seq_len(N))
The final comparison catches both omissions and out-of-range indices. The duplicate check identifies repeated assignment directly. These checks apply to row positions from 1 to \(N\); if the data have been filtered or reordered, \(N\) and the indices must refer to the data object that is actually being partitioned.
Worked Example: Keeping Outcomes Aligned With Their Rows
A hypothetical dataset has seven claim records, with row positions 1 through 7. A seeded draw gives the index permutation \(7,2,5,1,6,3,4\). Create partitions of four training rows, two validation rows, and one test row, and explain how to preserve the outcome attached to each claim.
Step 1: Allocate indices by position. The first four indices give train_id = c(7, 2, 5, 1). The next two give valid_id = c(6, 3). The last gives test_id = 4.
Step 2: Check counts and coverage. The partition sizes are \(4,2,1\), and their sum is 7. Combining the indices gives \(7,2,5,1,6,3,4\); sorting yields \(1,2,3,4,5,6,7\). There are no repeated indices, so the split is complete and disjoint.
Step 3: Subset complete rows. If the data frame is named claims, use claims[train_id, ], claims[valid_id, ], and claims[test_id, ]. Each selection carries all columns from the selected original row, including its outcome. Do not sample the predictor columns and outcome column separately: doing so could pair a predictor record with another claim’s outcome. The common mistake avoided is to check that row counts look right while losing the correspondence among variables.
Implementation Choices and Common Traps
A useful implementation keeps the allocation logic visible and makes the checks part of the analysis rather than an informal visual inspection. For instance, after creating the three index vectors, an analyst can use stopifnot() to stop execution if an essential condition fails:
stopifnot(length(train_id) == n_train)
stopifnot(length(valid_id) == n_valid)
stopifnot(length(test_id) == n_test)
stopifnot(anyDuplicated(c(train_id, valid_id, test_id)) == 0L)
stopifnot(identical(sort(c(train_id, valid_id, test_id)), seq_len(N)))
These checks are especially valuable after changing counts, filtering rows, or adapting code to another dataset. Confirm that the object being indexed has the same row order and row count used to generate the indices. If rows were removed after the split, the saved indices may no longer identify the intended records.
- Incorrect total count. Check that the three requested counts sum to \(N\) before sampling. A random procedure cannot make an impossible allocation valid.
- Sampling columns independently. Sample row indices once, then subset the entire data frame so predictors and outcomes remain paired.
- Relying only on the seed. A seed helps reproduce a draw under matching conditions; saved indices provide a direct record of the actual assignment.
- Checking only partition sizes. Equal-to-\(N\) total size does not rule out both a duplicated row and an omitted row.
- Using row-level randomization automatically. A valid index partition can still be unsuitable if the prediction target requires respecting entity or time structure.
Check Your Understanding
For each problem, identify the allocation or check needed. The final answer is provided so you can verify your reasoning.
- A dataset has ten rows. A planned split uses six training rows, two validation rows, and two test rows. A permutation is \(8,1,5,2,10,4,7,3,9,6\). Give the three index vectors and verify coverage. (Answer: Training \(8,1,5,2,10,4\); validation \(7,3\); test \(9,6\); the sorted combined indices are \(1\) through \(10\), with no duplicates.)
- A dataset has twelve rows, and the counts are seven training, three validation, and two test rows. If a seeded permutation is repeated exactly in the same R setup, what should be true of the two permutations and their partition vectors? (Answer: Both permutations should be identical, and slicing them at positions 7 and 10 produces identical partition vectors of lengths 7, 3, and 2.)
- A six-row dataset is assigned proposed counts of four training, two validation, and two test rows. Is the allocation possible? Give two valid alternatives, each with positive validation and test counts. (Answer: No, because the proposed counts sum to 8, greater than 6. Valid alternatives include \(4,1,1\) and \(3,2,1\).)
- For \(N=6\), an analyst reports training indices \(1,2,4\), validation indices \(3,5\), and test indices \(5,6\). What is wrong, and which row is omitted? (Answer: Row 5 is assigned twice, so the partitions are not disjoint; row 5 is not omitted, but no row is omitted. The repeated assignment makes the total number of listed indices 7 rather than 6.)
- A data frame has eight rows. An analyst sets a seed, draws eight indices, then filters out one row from the data frame and applies the original indices to the shortened object. Why is the saved split no longer safe to apply as written? (Answer: The indices refer to positions in the original eight-row object, while the filtered object has a different row count and potentially different row-to-index correspondence. Recreate or explicitly map the split for the filtered data before subsetting.)