Why Reproducibility Matters in Boosting
A boosted-model comparison is an experiment: the data split, random sampling within the fit, software implementation, and tuning settings all contribute to the result. If any of these change between runs, a different validation score may reflect a changed experiment rather than a meaningful model improvement. Reproducibility means that another run, with the same specified inputs and computational conditions, can recreate the split, fit, and reported outputs.
This matters particularly when boosting uses row subsampling or column subsampling. A seed controls a pseudorandom-number generator, but it does not document the whole experiment. The data order, package version, preprocessing, class weights, model parameters, and relevant parallel-computing settings must also be preserved. The goal is not merely to make a script run twice; it is to make the experiment identifiable and auditable.
Seeds, Randomness, and the Data Split
In R, set.seed() initializes the state of R's pseudorandom-number generator. If the same seed is set before the same sequence of random-number operations, the same sequence of pseudorandom values is generated under the same RNG settings. That is why a seeded call to sample.int() can recreate a random partition.
A split is itself an experimental input. Create it once, save the row identifiers assigned to each partition, and use those saved assignments for every model being compared. Do not rely only on rerunning a split command: a change in row order, filtering, or RNG settings can assign different observations to training and validation data even if the code still contains the same seed.
Worked Example: Recreating and Saving a Train/Test Split
A hypothetical study has 20 independent policy records in a data frame called dat. The analyst wants a 75% training partition and a 25% test partition. Show the split-size calculation, generate the partition in R, and identify what must be saved for later model comparisons.
Step 1: Calculate the partition sizes. The training size is \( \lfloor 0.75 \times 20 \rfloor = \lfloor 15 \rfloor = 15 \) records. The test size is \(20 - 15 = 5\) records. Thus all 20 records are assigned, with 15 in training and 5 in test.
Step 2: Generate the random indices. With the rows in a fixed order, the following code creates the split:
set.seed(2407)
train_idx <- sample.int(nrow(dat), size = floor(0.75 * nrow(dat)))
test_idx <- setdiff(seq_len(nrow(dat)), train_idx)
Here, nrow(dat) is 20, so the requested training sample size is 15. setdiff() assigns the remaining \(20 - 15 = 5\) row positions to the test partition. The specific positions selected are the realized random split; save them rather than assuming that a later sample call will be adequate documentation.
Step 3: Save identifiers and assignments. Suppose dat has a unique column named record_id. Save the selected IDs and the seed with the split:
split_record <- list(seed = 2407,
train_id = dat$record_id[train_idx],
test_id = dat$record_id[test_idx])
saveRDS(split_record, "split_record.rds")
The saved artifact records which 15 records and which 5 records were used, even if row order changes later. A check that the partition is complete is to verify that the training and test ID sets do not overlap and contain 20 unique IDs in total. The method avoids the common mistake of treating a seed as a substitute for preserving the actual split.
Record the Fit, Not Just the Seed
The fit can use additional random draws. For example, a gbm model with a bag fraction below 1 samples observations during boosting. If the data, parameter values, and random-number state are held fixed, repeating the same fit in the same computational environment should reproduce its predictions. Set the fit seed immediately before each fit, and record the model call rather than relying on memory or an informal note.
A compact experiment record should include at least:
- Data version or saved data artifact, row ordering, response definition, and unique row identifiers.
- Saved training, validation, and test assignments, along with the seed and method used to create them.
- Preprocessing steps, predictor definitions, factor-level handling, and any class weights or observation weights.
- Model family, loss or distribution, number of trees, interaction depth, learning rate, bag fraction, and other fitted settings.
- R version, package versions, RNG kind, and relevant hardware or parallel settings.
- Predictions or other outputs used to calculate the reported comparison.
R's sessionInfo() records R, attached packages, and platform details. packageVersion("gbm") records the installed version of that package, while RNGkind() reports the RNG configuration. A project lockfile, such as one maintained with renv, can additionally record package dependencies for restoring an environment. Keep the seed values named by purpose—for example, seed_split and seed_fit—so it is clear which random operation each controls.
Worked Example: Verifying a Repeated Stochastic Fit
An analyst fits a hypothetical binary-outcome boosting model to the same saved training rows twice. The model uses 100 trees, interaction depth 2, shrinkage 0.05, and a bag fraction of 0.70. Assume there are 100 training rows. Show the reproducible fitting procedure and specify a check that verifies the rerun.
Step 1: Record the fixed inputs. The training set has 100 rows, the bag fraction is 0.70, and the stated settings are 100 trees, depth 2, and shrinkage 0.05. With a 0.70 bag fraction, the intended sampled proportion at a boosting iteration is 70% of the training data, or \(0.70 \times 100 = 70\) rows. The precise sampled rows are random and are controlled by the fit's RNG state.
Step 2: Fit twice from the same RNG initialization. The following code uses the same data and model call both times:
set.seed(811)
fit1 <- gbm::gbm(y ~ ., data = train, distribution = "bernoulli",
n.trees = 100, interaction.depth = 2, shrinkage = 0.05,
bag.fraction = 0.70, train.fraction = 1,
verbose = FALSE)
set.seed(811)
fit2 <- gbm::gbm(y ~ ., data = train, distribution = "bernoulli",
n.trees = 100, interaction.depth = 2, shrinkage = 0.05,
bag.fraction = 0.70, train.fraction = 1,
verbose = FALSE)
Resetting the seed before the second fit puts the generator at the same initial state. Because the data and call are also unchanged, both fits receive the same sequence of pseudorandom draws in the same R and package environment.
Step 3: Compare predictions on the same rows. For a fixed set of 10 validation rows, obtain response-scale predictions and check them:
p1 <- predict(fit1, newdata = validation, n.trees = 100,
type = "response")
p2 <- predict(fit2, newdata = validation, n.trees = 100,
type = "response")
identical(as.numeric(p1), as.numeric(p2))
The expected check in this same-environment example is TRUE: there are 10 predictions from each fit, and each corresponding numeric value is identical. If the result is not TRUE, check first that the data, row order, call, seed placement, package version, and RNG configuration really match. This check avoids the trap of saying that two runs are reproducible merely because both completed successfully.
Why Reusing a Seed Works—and Where It Stops
A pseudorandom-number generator is deterministic: its internal state advances according to a fixed rule. Initializing it at a given seed therefore determines the subsequent sequence. If two fits start at the same state and consume random values in the same order, they receive the same random choices. Conditional on an implementation that is otherwise deterministic, that produces the same fit and predictions.
This is a conditional statement. A seed does not freeze the data, the function call, or the computing environment. A changed package version may use a different algorithm; a changed data order can alter the random choices' association with records; and parallel execution or hardware-specific calculations can affect exact results. Some implementations offer deterministic settings, but those settings must be checked for the specific implementation and version. Recording a seed is necessary for many stochastic fits, but recording the rest of the experiment is necessary for an auditable comparison.
For model comparisons, use the same saved data partition so that differences are not caused by testing on different records. If stochastic fitting is part of the question, record each fit's seed as well. These are distinct controls: the split seed determines which records belong to each partition, while the fit seed initializes random operations during model fitting. If class weights were used, record their values and how they were assigned; changing weights changes the fitted experiment even when every seed is unchanged.
Preserve a data snapshot or version, stable row identifiers, and the order or preprocessing that produces the modeling table.
Generate the split once, record its seed, and save the row IDs in each partition for reuse across model fits.
Record the package, response definition, predictors, weights, hyperparameters, and fit seed for every model configuration.
Record R and package versions and RNG settings, then rerun the fit and compare predictions or other saved outputs.
Worked Comparison: Preserve the Evidence Behind a Result
Worked Example: Auditing Two Boosting Configurations
A hypothetical analyst compares two regression-boosting configurations on the same four saved validation records. The observed values are 10, 12, 8, and 15. The saved predictions are A: 9, 12, 10, and 14; B: 10, 11, 9, and 14. The experiment record identifies a common saved split, separate fit seeds 901 and 902, and the full settings for each configuration. Verify the validation RMSE values and explain what a repeat run must reproduce.
Step 1: Calculate configuration A's errors. Subtract observed values from predictions: \(9-10=-1\), \(12-12=0\), \(10-8=2\), and \(14-15=-1\). Squaring gives \(1, 0, 4,\) and \(1\); their sum is \(1+0+4+1=6\). The mean squared error is \(6/4=1.5\), so RMSE is \(\sqrt{1.5}=1.2247\) to four decimal places.
Step 2: Calculate configuration B's errors. The errors are \(10-10=0\), \(11-12=-1\), \(9-8=1\), and \(14-15=-1\). Their squares are \(0, 1, 1,\) and \(1\), summing to 3. The mean squared error is \(3/4=0.75\), and RMSE is \(\sqrt{0.75}=0.8660\) to four decimal places.
Step 3: Check the comparison. The RMSE difference is \(1.2247-0.8660=0.3587\) to four decimal places, so configuration B has the lower RMSE on these four records. For this result to be reproducible, a rerun must recover the same four validation IDs, the same observed values and predictions, and the same recorded model settings and software environment. These predictions are supplied hypothetical outputs for illustrating the audit calculation; they are not claimed as outputs from a particular package fit.
The common mistake is to save only the two rounded RMSE values. They cannot reveal whether the models used the same validation records or how the values were calculated. Saving predictions, row IDs, and the experiment record makes both the arithmetic and the modeling comparison checkable.
Common Reproducibility Traps
- Resetting the seed in the wrong place. Setting a seed once at the start of a long script does not ensure that a later fit begins at the same RNG state if earlier code consumes random values. Set a documented fit seed immediately before the fit when exact repeatability is intended.
- Assuming a seed saves the split. A seed does not identify which records were assigned to each partition if the data or row order changes. Save the actual IDs.
- Changing the environment silently. A package update or different RNG configuration may change results. Record versions and the RNG kind; use a lockfile when restoring package dependencies matters.
- Comparing models on different partitions. A score difference then reflects both model changes and data changes. Reuse the saved partition when the purpose is to compare fits.
- Claiming universal bit-for-bit identity. Parallel computation, implementation changes, or hardware can affect exact reproducibility. State the conditions under which the rerun was checked.
A concise record can be maintained alongside the fitted object: save the split IDs, named seeds, exact model calls, weights, package and R versions, RNG kind, and predictions used in the comparison. This makes it possible to separate a genuine model-setting change from an accidental change in the experiment. These practices are aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
For each case, identify the reproducibility issue or calculate the requested quantity.
- A data frame has 40 rows. An analyst requests a 70% training split using floor(0.70 * nrow(dat)). How many rows are in training and test, and what should be saved in addition to the seed? (Answer: 28 training rows, 12 test rows; save the row IDs assigned to each partition.)
- A stochastic fit is run twice with the same fit seed, but the second run uses a different row order and a new package version. Is the seed alone sufficient to guarantee identical predictions? State two additional items to control or record. (Answer: No; preserve the same data ordering or saved row assignments and record or restore the package version. A seed alone does not guarantee identical predictions.)
- A validation set has observed values 4, 7, and 9 and predictions 5, 5, and 10. Calculate RMSE to four decimal places. (Answer: Errors are 1, -2, and 1; squared errors sum to \(1+4+1=6\), MSE is \(6/3=2\), and RMSE is \(\sqrt{2}=1.4142\).)
- Two boosting models are compared. The analyst saved separate fit seeds and all hyperparameters, but recreated the random test split after filtering two records from the data. What is the main problem, and what should be used instead? (Answer: The test membership may have changed, so the scores may no longer be a like-for-like comparison; use the saved test row IDs and verify that the intended records remain available.)
- A rerun produces slightly different results despite matching the seed, data, and model call. Name two environment-related checks before concluding that the model settings changed. (Answer: Check the R and package versions and the RNG configuration; also investigate parallel or hardware-specific computation settings.)