Why Reproducible Randomness Matters
Regression workflows sometimes use randomized procedures: bootstrap resampling, permutation procedures, or randomly assigned cross-validation folds, for example. These procedures can produce slightly different outputs when run again, even when the data and model are unchanged. That variation is part of the computational method, not necessarily an error. But if an analysis is being checked, shared, or rerun, it is useful to know whether a difference comes from a changed analysis or simply from a new random draw.
A random-number seed provides a way to repeat a pseudorandom sequence under specified conditions. It helps make a randomized step traceable and repeatable. It does not make a random procedure inherently stable, prove that its result is accurate, or guarantee that two different software environments will produce identical results.
In a bootstrap analysis, for instance, the generator supplies the random selections used to construct resamples. The regression calculations on each resample may be deterministic; the variability in the results arises because the selected resamples vary. Setting a seed controls the starting point for those selections. The usual substantive model assumptions still matter for interpreting a regression analysis, but they are separate from this computational reproducibility step.
Setting and Documenting a Seed
Set the seed before the randomized operation whose output you want to reproduce. If a workflow contains more than one randomized step, be clear about whether the seed is set once before the sequence of steps or reset before each step. These choices can produce different random draws. A record that says only “seed = 42” is incomplete if it does not identify the software context and the randomized procedure to which that seed applies.
A useful record names the seed, the random operation, and the relevant computational context. For example, note that a seed was set immediately before generating 1,000 bootstrap resamples for a named regression coefficient. Also record the software and package versions and any settings that affect the random procedure. The same numerical seed used with a different generator, software version, or random-number implementation may not produce the same sequence.
A seed is most useful when the analysis is rerun from the beginning in a clean session, using the documented data, code, software, and settings. Running extra randomized commands before the target operation can consume values from the generator and change the draws that follow. Similarly, setting a seed midway through a workflow does not reproduce random steps that already occurred.
Worked Example: Tracing Bootstrap Resamples
Consider a hypothetical data set with four observations, where \(x\) is measured in hours of practice and \(y\) is a test score in points: \((1,2), (2,3), (3,5), (4,4)\). The fitted simple regression slope for the original data is calculated from the usual slope formula.
For a bootstrap demonstration, draw four observation pairs with replacement for each resample, then calculate the slope for that resample. Suppose that, in a hypothetical run using a documented seed, the first two resamples select indices \((1,2,3,4)\) and \((1,1,3,4)\). The first resample is the original set of pairs, so its slope remains \(0.800\) points per hour.
For the second resample, the selected pairs are \((1,2), (1,2), (3,5), (4,4)\). Here \(\bar{x}=2.25\) hours and \(\bar{y}=3.25\) points. Substituting into the slope formula gives:
The seed determines the sequence of selected indices for this specified run, so repeating the same workflow under the same conditions can reproduce these resamples and slopes. The seed does not change the observed slope of \(0.800\), nor does it guarantee that the slopes across bootstrap resamples will be close to one another. In a real analysis, many resamples would be generated; these two are shown only to make the process concrete.
Checking Exact Reproducibility
A direct reproducibility check is to rerun the randomized step from a clean start with the same seed and settings, then compare the relevant outputs. Depending on the procedure, those outputs might include the resampled indices, the estimated coefficients, the reported interval endpoints, or the assigned folds. Comparing only a final rounded result can conceal differences in intermediate computations, so retain and compare outputs at an appropriate level of detail.
A useful check is to compare both the random inputs and the resulting analysis outputs when feasible. If the resamples differ, the random-number sequence or its use differs. If the resamples match but the fitted coefficients do not, investigate other parts of the workflow, such as changed data or model settings. This comparison helps isolate where a rerun diverged; it is not a substitute for checking that the procedure itself is appropriate.
Worked Example: Rerunning a Bootstrap Interval
Suppose a hypothetical regression workflow bootstraps a slope measuring the association between weekly study hours and quiz score. The analysis uses 2,000 resamples and reports a 95% percentile interval. A run with seed 731 records a slope estimate of \(1.42\) points per hour and interval endpoints of \(0.38\) and \(2.49\) points per hour.
The analyst restarts the workflow from a clean session with the same data, code, seed, software, package versions, and resampling settings. The rerun gives the same estimate and endpoints: \(1.42\), \(0.38\), and \(2.49\). This exact agreement is evidence that the documented workflow reproduces these outputs under those conditions. It does not establish that the model is well specified or that the interval has perfect coverage.
Now suppose a rerun instead gives endpoints of \(0.36\) and \(2.51\), while the displayed estimate remains \(1.42\). The small difference is not explained by ordinary rounding if both runs used the same underlying computations. The analyst should check whether the seed was set at the same point, whether a preceding random operation consumed generator values, and whether the data, software, and procedure settings really matched. The output difference is a diagnostic clue, not by itself evidence that one interval is more correct.
Checking Stability Across Random Draws
Exact reproducibility and stability answer different questions. With a fixed seed and unchanged conditions, the goal is to reproduce the same computation. To assess sensitivity to random draws, deliberately repeat the procedure with different seeds while keeping the data, model, settings, and number of random repetitions fixed. The outputs should not be expected to match exactly; the purpose is to see how much they vary.
For a bootstrap estimate based on \(B\) resamples, one simple stability summary is the range across repeated runs: the largest reported value minus the smallest. You can also report the values from each run and compare them on the scale relevant to the analysis. Do not treat a narrow range as proof that the model is correct; it only indicates limited variation across the particular randomized runs examined.
Worked Example: Comparing Bootstrap Runs
Suppose a hypothetical analysis estimates the change in a regression outcome associated with one additional hour of training. The analyst runs the same bootstrap procedure with 2,000 resamples using three different seeds. The resulting estimated 95% percentile intervals, in outcome points per hour, are:
| Seed | Lower endpoint | Upper endpoint |
|---|---|---|
| 14 | 0.41 | 2.48 |
| 29 | 0.39 | 2.52 |
| 67 | 0.42 | 2.47 |
The range of lower endpoints is \(0.42-0.39=0.03\) points per hour. The range of upper endpoints is \(2.52-2.47=0.05\) points per hour. These small differences suggest that the reported endpoints are fairly stable across these three runs, at the displayed precision. The endpoints are not identical, which is expected because the runs use different random draws.
This check does not show that every possible run would yield nearly identical endpoints, nor does it establish that the model is appropriate. It indicates only that, for these three runs with the stated settings, the endpoint variation was small relative to the units and scale being reported.
A Randomized Validation Example
Randomness can also enter through a split into training and validation data or through the assignment of observations to cross-validation folds. A seed can reproduce one assignment, but a single assignment may give an incomplete picture of how results depend on the split. Comparing several seeds provides a practical check of that dependence, provided the same analysis settings are used each time.
Worked Example: Comparing Randomized Fold Assignments
A hypothetical analyst evaluates a regression model using five-fold cross-validation on the same data. Each fold assignment is randomized, and the analysis reports the mean validation root mean squared error (RMSE) in outcome units. Three runs, each using five folds and the same model settings, give RMSE values of \(6.2\), \(6.5\), and \(6.3\) points.
The mean of the three run-level RMSE values is:
Their range is \(6.5-6.2=0.3\) points. These calculations summarize variation across the three randomized fold assignments; they are not a confidence interval for generalization performance. If the assignments produced much wider variation, the analyst would have reason to investigate how sensitive the reported validation summary is to the random split. A recorded seed for each run makes the assignments traceable, while the comparison across seeds addresses stability.
Common Mistakes and Practical Limits
- Assuming one seed guarantees every rerun everywhere. The same seed may not produce the same results across different software versions, generators, or parallel-computing settings. Record the computational context as well as the seed.
- Setting the seed after the random operation. A seed cannot control draws that have already occurred. Set it before the step you want to reproduce.
- Confusing repeatability with stability. Matching outputs under the same seed checks repeatability. Similar outputs across different seeds check stability. Neither check alone establishes that the analysis is statistically sound.
- Changing several things while comparing runs. If data, model settings, resample count, and seed all change, it is difficult to identify the source of output differences. To examine seed sensitivity, change the seed while holding the other settings fixed.
- Treating a seed as evidence of randomness quality. A seed is an initialization value, not a diagnostic that the generator or the statistical procedure is appropriate.
In parallel computations, the assignment of random-number streams to tasks can depend on software and settings. A seed may therefore fail to reproduce results if the parallel configuration changes. When a workflow uses parallel random operations, document the relevant configuration and verify it by rerunning the analysis in the intended environment.
Check Your Understanding
Answer these questions using the distinctions and procedures in this tutorial.
- What does a seed control in a pseudorandom procedure, and what conditions are needed for the same sequence to be reproduced?
- A bootstrap analysis is run twice with the same seed but different package versions. Why might the outputs differ?
- Explain the difference between checking reproducibility with the same seed and checking stability across different seeds.
- A seed is set after a randomized fold assignment has already been created. Can it reproduce that assignment? Explain.
- Two bootstrap runs produce slightly different interval endpoints under different seeds. What can the difference tell you, and what can it not establish?