Assessing Random-Seed and Replicate Variability

Why Compare Independent Replicates?

A simulation can produce different numerical summaries even when its model, inputs, and code have not changed. Random draws vary from run to run, so a simulated event proportion or mean is an estimate with Monte Carlo variation. A discrepancy between runs is therefore not automatically evidence of a changed model or a coding defect. The review question is whether the difference is plausible given the simulation’s expected random variation, and whether other evidence points to a deterministic difference.

The earlier material on checking simulation outputs against model intent concerns whether outputs are consistent with the intended model. Replicate assessment adds a related but distinct check: hold that intended model and its inputs fixed, vary the random draws, and quantify how much the reported summaries change. This does not replace checks of dose events, model components, or output logic. It helps separate stochastic variation from changes in configuration, implementation, or execution.

A replicate is one run of a specified simulation procedure. Its random seed initializes a pseudorandom-number generator; changing the seed commonly changes the sequence of generated values. A different seed is not, by itself, a proof that two runs are statistically independent. Independence depends on how the generator and streams are managed. Reproducible review therefore records the seed, generator and stream settings, simulation inputs, code version, and run configuration.

Replicate-Level Quantities and Assumptions

The same procedure means that the model, parameter values, input data or covariates, simulation size, output definition, and relevant options are held fixed. Only the intended random draws differ. If a run changes sample size, a parameter, a covariate distribution, or code, then its difference from another run is not attributable solely to random-seed variation. Such differences may be intentional, but they require a different comparison.

If each replicate summary has finite variance \(\tau^2\), independence gives the variance of the replicate average. The sample variance \(s_T^2\) estimates \(\tau^2\), so \(s_T/\sqrt{R}\) estimates the Monte Carlo standard error of \(\bar{T}\).

The variance equality uses independence: covariances between distinct replicates are zero. If replicates are correlated, that equality does not hold and the usual standard error can understate or overstate uncertainty. Also distinguish \(R\), the number of independent replicates, from the number of simulated subjects or draws within each replicate. More subjects within one run may reduce that run’s noise, but do not provide the same information about between-replicate variation as additional independent runs.

Event Proportions: Expected Variation Across Seeds

Suppose each replicate generates \(M\) independent, identically distributed Bernoulli outcomes with a common event probability \(p\). The simulated proportion in one replicate is \(\hat{p}=K/M\), where \(K\) is the number of events. Its expectation is \(p\), and its variance is \(p(1-p)/M\). Because \(p\) is usually unknown, substituting \(\hat{p}\) gives the estimated standard error for that replicate’s proportion. The common-probability and identical-distribution assumptions matter; independence alone does not justify this binomial formula.

If independent Bernoulli outcomes instead have different event probabilities \(p_i\), then \(\hat{p}=M^{-1}\sum_{i=1}^M X_i\) has variance \(M^{-2}\sum_{i=1}^M p_i(1-p_i)\). The single binomial standard-error formula is not generally appropriate in that case. In a simulation review, first decide whether the outcomes are genuinely generated under a common probability or whether covariates or other individual-level features make their probabilities differ.

Continuous Summaries and a Short Derivation

For a continuous output, the replicate statistic might be the mean apparent clearance among simulated subjects, a concentration quantile, or the fraction meeting a threshold. Calculate the same statistic in every replicate. The sample standard deviation \(s_T\) describes the spread of individual replicate summaries; the Monte Carlo standard error \(s_T/\sqrt{R}\) describes the estimated precision of their average. These answer different questions and should not be reported interchangeably.

The standard error formula follows directly from the variance of a sum. If each replicate statistic has variance \(\tau^2\), then independence makes every cross-product covariance zero. Thus the variance of the sum is \(R\tau^2\), and dividing that sum by \(R\) divides its variance by \(R^2\). The result is \(\tau^2/R\). Replacing the unknown \(\tau^2\) with the sample variance gives the estimated standard error above. This derivation also shows why the formula needs independent replicates with a common distribution and finite variance.

Comparing Two Independent Runs

To compare two implementations or configurations, calculate the same summary in independent replicates from each. Let \(\bar{T}_A\) and \(\bar{T}_B\) be the averages across \(R_A\) and \(R_B\) replicates, with sample standard deviations \(s_A\) and \(s_B\). If the two sets of replicates are independent, the estimated standard error of their difference is \(\sqrt{s_A^2/R_A+s_B^2/R_B}\). This comparison is meaningful only if both sets target the same intended quantity; if a setting is deliberately changed, the expected change must be considered separately.

Diagnostics, Alternatives, and Limitations

A useful review separates reproducibility checks from statistical comparisons. First, rerun the same configuration with the same seed and generator settings. If the software is designed for deterministic replay, the outputs should reproduce to the documented numerical tolerance. Failure to reproduce is a reproducibility issue to investigate; it is not ordinary variation between independent seeds. Next, run distinct seeds using the same fixed configuration and compare replicate-level summaries. Record enough metadata to make either check interpretable.

Inspect the full set of replicate summaries, not just their average. A single extreme replicate, a steadily drifting sequence, unexpectedly identical outputs across distinct streams, or a spread that changes markedly between batches may indicate a problem. Check logs for warnings, failed runs, different sample sizes, missing outputs, accidental seed reuse, and stream overlap. Also verify that summary calculations use the same units, time points, and denominator in every replicate.

When there are only a few replicates, \(s_T\) is a noisy estimate of the true between-replicate spread. Highly skewed or heavy-tailed replicate statistics may also make a normal approximation for the average unreliable at small \(R\). A conventional interval based on a normal approximation, \(\bar{T}\pm 1.96\,\widehat{SE}(\bar{T})\), is only an approximate 95% interval when replicate averages are sufficiently close to normal and the standard error estimate is reliable. It is not a probability statement about a fixed parameter, and it should not be used mechanically with very few or pathological replicates.

The main alternative to independent-seed comparison is a paired comparison using common random numbers: align corresponding random inputs across two implementations, then analyze within-pair differences. Pairing can reduce noise when matched draws are positively correlated, but requires deliberate stream control and is not the independent-replicate calculation presented above. Repeating the exact same seed is useful for checking replay, not for estimating variation across independent simulations. In either approach, numeric agreement alone does not verify that the simulated model matches its intent.