Why Reproducibility Matters
A random forest uses random choices, including which observations and predictors are available to a tree. Consequently, two fits with the same data and tuning parameters can differ if they use different random-number streams. That variation is not automatically a problem: it is part of how the method works. The problem arises when a result cannot be rerun or when two candidate models are compared under uncontrolled differences.
In R, setting a seed is an important part of controlling randomness, but it is only one part of reproducible analysis. A rerun can still change if the data order, training split, sampling settings, package version, random-number generator, or parallel computation differs. A professional workflow records these choices rather than relying on the seed alone.
What a Seed Controls
R’s random-number generator produces a sequence of values from an initial state. The command set.seed() resets that state. If the generator, software behavior, and sequence of random operations are also the same, resetting it to the same seed reproduces the sequence. A forest fit can be viewed as a function of both the data and that sequence:
where \(U_1,U_2,\ldots\) are the random values used during fitting. If the same seed produces the same sequence and the same operations use it, the fit is repeatable. If the data, settings, or software’s use of random numbers changes, matching the starting seed does not ensure matching output.
This explains why a seed is not a universal identifier for a model. It identifies an initial random state in a particular computational context. It does not specify the training data, package implementation, sampling scheme, tuning choices, or random-number generator. Nor does it guarantee that parallel computation will produce bitwise-identical results across different environments.
A Reproducible Workflow
Treat randomness at each stage as a separate decision. A split can be reproducible while the forest fit is not, or vice versa. A useful workflow records which seed applies to each stage and preserves the split or resampling assignments used for comparison.
List any randomized data split, resampling, subsampling, or forest fit. Do not assume that one seed set at the top of a long script makes every later result easy to audit.
Record the source, version, row-selection rules, preprocessing, and factor levels. Save split or fold assignments when comparisons must use exactly the same observations.
Record the implementation and its arguments, including the number of trees, predictor-sampling setting, tree-size controls, sampling method, sampling fraction, replacement choice, and any class-sampling or weighting settings used.
Choose a seed for each randomized stage, set it immediately before that stage, and record the random-number-generator settings and software versions.
Repeat the script in the documented environment and compare the saved split, settings, predictions, and chosen performance summaries. State whether the goal is exact equality or agreement within a stated tolerance.
When using R, the base command RNGkind() reports or sets the random-number generator, normal generator, and (in current R versions) sampling method. For example, a script can explicitly use RNGkind(kind = "Mersenne-Twister", normal.kind = "Inversion", sample.kind = "Rejection") before calling set.seed(). The sampling-method argument is version-dependent; older R releases do not accept it. If an analysis must be repeated across R versions, record the version and check the relevant documentation rather than assuming that the same seed implies the same sampled rows.
Record the R version and attached package versions, for example with R.version.string, packageVersion("ranger"), and sessionInfo(). Also record the actual forest arguments, not merely the call’s defaults: defaults can change between package versions, and an omitted argument is harder to audit than an explicit choice. For parallel fitting, record the number of threads and the package’s documented random-number behavior. A single-threaded rerun can be a useful reference when parallel execution is not known to be reproducible in the chosen setup.
Worked Examples
Worked Example: Repeating a Train–Test Split and Fit
A hypothetical claims study contains 240 eligible records. The analyst wants a 25% test set and will fit a forest to the remaining records. Show how to make the split and fit repeatable, and identify what must be recorded.
Method. Control the split and fit as separate random stages. Suppose the data frame is named dat, and the analyst uses a 25% test fraction.
Step 1: Calculate the split sizes. The test-set size is \(0.25 \times 240 = 60\) records. The training-set size is \(240 - 60 = 180\) records.
Step 2: Control the row selection. With the row order fixed, the analyst can use set.seed(731), then test_id = sample(seq_len(nrow(dat)), size = floor(0.25 * nrow(dat))). Here, floor(0.25 * 240) = 60. Define the training IDs as setdiff(seq_len(nrow(dat)), test_id); this leaves \(240 - 60 = 180\) rows. Save both sets of IDs so the test observations do not silently change during later comparisons.
Step 3: Control the forest fit separately. After the data and settings are fixed, use set.seed(842) immediately before the fit. For example, an explicitly configured ranger call should record its formula, data, number of trees, predictor-sampling and tree-size controls, sampling settings, thread count, and package version. The seed for the fit is not a substitute for the split seed: resetting it before the forest does not restore a split that was made differently.
Step 4: State the reproducibility conditions. The rerun requires the same 240 records in the same order, the same split IDs, the same preprocessing and factor levels, the same forest settings, compatible R and package versions, and the same random-number behavior. The example avoids the common mistake of reporting one seed as if it fully described the analysis.
Check. The split contains 60 test records and 180 training records, which sum to the original 240. The two separate seeds make the roles of split randomness and forest randomness explicit.
Worked Example: Comparing Configurations on Fixed Folds
A hypothetical pricing analysis compares three forest configurations using five fixed validation folds. Each fit uses 400 trees. The fold-level validation RMSE values are: configuration A, 12, 11, 13, 10, 14; configuration B, 11, 10, 12, 11, 10; and configuration C, 10, 12, 11, 10, 14. Show how to compare the candidates without letting different fold assignments confound the comparison, and calculate the mean RMSE for each.
Method. Use the same five fold assignments for every configuration, so each candidate is evaluated on the same held-out records in each fold. Use a documented seed schedule for the randomized fits. One possible schedule is seed = 10000 + 100 * fold_id + config_id, where fold IDs run from 1 to 5 and configuration IDs from 1 to 3. Reset the seed immediately before each fit. This schedule assigns a distinct seed to each fit; it does not guarantee that different configurations use identical tree-level random choices.
Step 1: Count the fitting workload. There are \(5 \times 3 = 15\) fits. At 400 trees per fit, the total is \(15 \times 400 = 6{,}000\) fitted trees.
Step 2: Check the seed schedule. For configuration B in fold 3, the seed is \(10000 + 100 \times 3 + 2 = 10302\). For configuration C in fold 5, it is \(10000 + 100 \times 5 + 3 = 10503\). These values are distinct, and the same fold assignment remains in use across candidates.
Step 3: Calculate the mean RMSE. Configuration A has total RMSE \(12 + 11 + 13 + 10 + 14 = 60\), so its mean is \(60 / 5 = 12\). Configuration B has total \(11 + 10 + 12 + 11 + 10 = 54\), so its mean is \(54 / 5 = 10.8\). Configuration C has total \(10 + 12 + 11 + 10 + 14 = 57\), so its mean is \(57 / 5 = 11.4\).
Conclusion. Configuration B has the lowest mean RMSE among these three candidates, at 10.8. These values are hypothetical and illustrate the comparison procedure; a real analysis should also report its evaluation design and uncertainty or variability where appropriate.
Common mistake. Creating new random folds separately for each configuration mixes the effect of the configuration with the effect of which observations were held out. Fixed folds remove that avoidable difference. The seed schedule makes the individual fits repeatable under the stated conditions, but it does not make the fold-level RMSE values identical across configurations.
Worked Example: Documenting Subsampling and the Computing Environment
An analyst fits a hypothetical regression forest to 500 training records. The intended specification is 600 trees, a sampling fraction of 0.70 without replacement, 4 candidate predictors at each split, a minimum node size of 10, and one thread for a reference run. Write down the sampling calculation and the information needed to reproduce the run.
Method. Make the sampling and computational choices explicit, then record the software environment. In a package such as ranger, the corresponding arguments include num.trees = 600, sample.fraction = 0.70, replace = FALSE, mtry = 4, min.node.size = 10, and num.threads = 1. The precise call and defaults should be checked against the installed package version.
Step 1: Calculate the per-tree sample size. With 500 training records and sampling without replacement, \(500 \times 0.70 = 350\) records are sampled for each tree when the package applies the stated fraction to the training rows. Thus, each tree uses 350 sampled records, leaving \(500 - 350 = 150\) training records not selected for that tree’s sample. This calculation describes the intended sampling design; it is not a claim about a particular fitted model’s predictions.
Step 2: Set the fit’s random state. In the script, select the random-number settings, then run set.seed(946) immediately before the forest call. Keep the data, response and predictor definitions, preprocessing, factor levels, and the stated forest arguments fixed for the rerun.
Step 3: Save the audit details. Record the seed, R version, ranger version, sessionInfo(), thread count, sampling fraction, replacement choice, and all tree controls. Preserve the analysis code and a stable copy or identifier for the input data. If the package, R version, or hardware changes, describe that change rather than implying that a matching seed guarantees exact equality.
Check. The sample size is \(350\) per tree under the specified fraction and no-replacement design. The example avoids the common mistake of documenting “70% sampling” but leaving out the training-set size or whether sampling is with replacement.
Comparability Is More Than Repeating a Run
Reproducibility and comparability are related but distinct. Reproducibility asks whether a documented analysis can be repeated. Comparability asks whether differences between candidate fits can reasonably be attributed to the changes being studied rather than to uncontrolled changes elsewhere.
For a tuning comparison, keep the data split or folds fixed across candidates. Record each candidate’s settings and use a pre-specified rule for random seeds. A separate seed for every fit is easy to audit; using the same seed for several candidates can also be a deliberate design choice, but it does not guarantee identical random operations when candidates consume random numbers differently. Neither choice replaces fixed validation observations or explicit reporting of the evaluation procedure.
Likewise, do not silently change defaults when moving from one forest implementation to another. A package may define a control differently or use a different default. Record the implementation and settings, and interpret any change in results in light of those changes. If execution is parallel, document the number of threads and test whether the chosen software setup reproduces the outputs at the precision required. Do not assume all parallel implementations handle random-number streams identically.
Common Reproducibility Traps
- Recording only the seed. A seed does not record the data, split, package, sampling scheme, or forest controls. Save those alongside it.
- Setting the seed too early. Unrelated random operations between the seed and the fit can consume random values and change the fit’s stream. Reset the seed directly before the randomized stage.
- Recreating the split for each candidate. Different validation observations make candidate results harder to compare. Preserve one split or set of folds for the comparison.
- Assuming defaults are stable. Defaults and implementation details may differ across package versions. Record explicit arguments and software versions.
- Promising exact equality across environments. Changes in R, packages, random-number behavior, or parallel execution can prevent bit-for-bit identity. State the conditions and level of agreement expected.
Check Your Understanding
For each problem, identify the reproducibility or comparison issue and calculate the requested quantity.
- A study has 320 records and reserves 20% for testing. How many records belong in the test and training sets? State one additional item, beyond the seed, that should be saved. (Answer: 64 test records and 256 training records; save the row IDs or equivalent split assignment.)
- A forest comparison uses four fixed folds and two configurations, with 250 trees per fit. How many fits and fitted trees are involved? (Answer: 8 fits and 2,000 trees.)
- Using the seed rule \(7000 + 100 \times \text{fold ID} + \text{configuration ID}\), what seed is assigned to configuration 2 in fold 4? (Answer: 7402.)
- A training set has 800 records, and each tree samples 60% without replacement. Under the stated sampling fraction, how many records are sampled and how many are not sampled for each tree? (Answer: 480 sampled and 320 not sampled.)
- Two candidates are assessed on different randomly generated validation folds, and a seed is recorded for each forest. Is their comparison controlled for validation membership? Give the correction. (Answer: No; use the same saved fold assignments for both candidates, while recording fit seeds and settings separately.)