Why Pair Simulations of Candidate Designs?
When two candidate MAMS designs are simulated separately, their estimated operating characteristics can differ partly because the designs differ and partly because each simulation happened to draw different random trial histories. A direct comparison can be more precise if both designs are exposed to the same underlying random conditions. This technique is called common random numbers (CRN).
For example, suppose the quantity of interest is the probability that a design rejects a null hypothesis under one specified scenario. Simulating each design on the same replication-specific random inputs creates a pair of outcomes. The difference within each pair measures how the designs differed under those shared inputs. Averaging these paired differences estimates the difference in operating characteristics.
The sign convention matters: with \(D_r=Y_{Br}-Y_{Ar}\), a positive average difference means design B has a larger value of the outcome than design A. For rejection probability, that means a higher simulated rejection rate; it does not by itself establish that the higher rate is desirable. The same setup works for other simulation outputs, such as expected sample size, where a negative difference can mean that design B uses fewer participants on average.
What Makes a Common-Random-Number Comparison Valid?
The goal is not simply to set the same seed twice. The coupling must preserve the correct marginal simulation for each candidate design. For every replication, design A’s simulated trial must have the distribution it would have under a valid simulation of A, and likewise for B. The shared inputs create dependence between the two simulated outputs; they must not introduce invalid dependence among observations within either simulated trial.
A practical construction is to index random inputs by replication and by their role in generating the trial: for example, by patient index, treatment assignment, outcome type, and random event. When designs share a control arm or a trial-generation mechanism, corresponding inputs can be shared. If sample sizes differ, a shared sequence can supply the first \(n\) patient-level inputs to a design requiring \(n\) patients; additional inputs are generated for the design requiring more patients. If allocation or stage structure differs, map inputs by a prespecified common source, such as patient-level potential outcomes or arm-specific outcome streams, rather than assuming that the same position in two code loops has the same meaning.
The comparison is usually conducted separately within each scenario on the scenario grid. A scenario specifies the parameter values and other conditions under which the operating characteristic is being assessed. Replications should be independent across \(r\), even though the two candidate outputs within each replication are deliberately dependent. The simulation code for each candidate must also correctly implement its own design; shared randomness cannot repair an incorrect design implementation.
Why Pairing Can Reduce Monte Carlo Noise
The estimator of the difference in mean operating characteristics is the average paired difference. Its uncertainty depends on the variance of a difference, not on the separate variances alone.
Here \(s_D\) is the sample standard deviation of the \(R\) paired differences. The equality between the paired mean difference and the difference of the two sample means holds because both means use the same \(R\) replications.
The justification for CRN is the variance identity
If the coupling makes the two outputs positively correlated, the covariance term reduces the variance of the paired difference. If outputs are independent within pairs, the covariance is zero and pairing offers no variance reduction. If they are negatively correlated, the paired difference can be noisier than a difference based on independent simulations. Thus CRN is a strategy to try and assess, not a guarantee of improved precision.
For comparison, with independent simulations of \(R\) replications per design, the estimated difference has estimated standard error
where \(s_A^2\) and \(s_B^2\) are the sample variances within the two independent sets. With paired simulation, use \(s_D/\sqrt{R}\), not this independent-sample formula. The covariance induced by CRN is part of the paired variance.
Worked Example: Paired Difference in Rejection Probabilities
Suppose 1,000 paired simulations compare two hypothetical MAMS designs under the same null scenario. Let \(Y_A=1\) or \(Y_B=1\) indicate that the corresponding design rejects at least one null hypothesis. The paired results are: both reject in 230 replications; only A rejects in 70; only B rejects in 90; neither rejects in 610.
The estimated rejection probabilities are \(\hat{p}_A=(230+70)/1000=0.300\) and \(\hat{p}_B=(230+90)/1000=0.320\). With \(D=Y_B-Y_A\), the differences are \(+1\) in 90 replications, \(-1\) in 70, and 0 in the remaining 840. Therefore,
The paired sample second moment is \(\sum D_r^2/R=(90+70)/1000=0.160\). Using the sample-variance correction,
In this hypothetical simulation, B’s estimated rejection probability is 2.00 percentage points higher than A’s under the null scenario. The paired Monte Carlo standard error of that difference is about 1.26 percentage points. It describes simulation noise in estimating the contrast; it is not a clinical effect estimate.
For comparison, treating the two rejection indicators as independent gives an estimated standard error of \(\sqrt{[0.300(0.700)+0.320(0.680)]/1000}=0.0207\). The paired outputs tend to move together: their sample covariance is approximately \((230/1000)-(0.300)(0.320)=0.134\), which is positive. That positive covariance explains why the paired standard error is smaller here.
Constructing and Checking the Coupling
A shared seed is not a complete description of a coupling. Software may consume random numbers in different orders when candidate designs have different stage rules, allocation ratios, or sample sizes. The same seed can then produce inputs that are not meaningfully aligned, and a code change can silently alter the pairing. A reproducible comparison should document how shared inputs map to the simulated trial and should keep the marginal simulation for each design correct.
Name the operating characteristic, the scenario, and the direction of subtraction. For example, define the contrast as rejection probability for B minus rejection probability for A.
Identify which inputs represent genuinely shared trial conditions and how they are indexed across designs. Use independent inputs where the two designs require distinct random events.
For each replication, generate both candidate outputs from the corresponding coupled inputs. Use a new independent set of inputs for the next replication.
Calculate \(\bar{D}\) and \(s_D/\sqrt{R}\). Also inspect the pairwise relationship and compare the paired variance with an independent-simulation benchmark or pilot estimate.
Worked Example: Comparing Expected Sample Size
In a small, hypothetical pilot with five paired replications, two designs have the following total sample sizes. The inputs are coupled by replication, and each design’s own stopping and allocation rules are applied to its simulated trial.
| Replication | A: participants | B: participants | \(D=B-A\): participants |
|---|---|---|---|
| 1 | 420 | 400 | -20 |
| 2 | 500 | 470 | -30 |
| 3 | 380 | 370 | -10 |
| 4 | 610 | 580 | -30 |
| 5 | 490 | 460 | -30 |
The estimated difference in expected sample size is
The differences have mean \(-24\) and deviations \(4,-6,14,-6,-6\) participants. Thus \(s_D^2=(16+36+196+36+36)/(5-1)=80\) participants squared, and \(s_D=\sqrt{80}=8.94\) participants. The paired Monte Carlo standard error is \(8.94/\sqrt{5}=4.00\) participants. In this pilot, B used an estimated 24 fewer participants on average than A. Five replications are far too few for a stable final comparison; the calculation illustrates the paired method, not a design-selection conclusion.
The close movement of the two sample-size outputs across replications is consistent with positive within-pair association. That association can make the differences less variable than the two sample sizes considered separately. The actual amount of benefit should be checked with an adequately sized simulation.
Worked Example: When Common Inputs Do Not Help
Consider 100 hypothetical paired simulations of two binary operating-characteristic outcomes. Both designs reject in 0 replications, only A rejects in 40, only B rejects in 40, and neither rejects in 20. Each estimated rejection probability is \(0.400\), so \(\bar{D}=0\).
Here \(D\) is \(+1\) 40 times, \(-1\) 40 times, and 0 20 times. Its sample variance is
The independent-simulation standard error at these estimated rates would be \(\sqrt{[0.400(0.600)+0.400(0.600)]/100}=0.0693\). In this example the paired outputs are negatively associated: if one design rejects, the other does not. Pairing has increased rather than reduced the estimated noise. The result does not invalidate the marginal rejection-rate estimates, but it shows why analysts should inspect the paired differences instead of assuming CRN worked well.
Diagnostics, Alternatives, and Limitations
Useful checks include verifying that the marginal simulation results for each design agree with independently run simulations or established analytic special cases, and inspecting the sample covariance or correlation of paired outputs. For a binary outcome, the counts in the four joint categories—both positive, only A positive, only B positive, neither positive—show directly how often the designs agree. For a continuous outcome, a plot or summary of \(Y_A\) against \(Y_B\), alongside the distribution of \(D\), can reveal whether pairing is producing consistent differences or occasional large contrasts.
If CRN produces little benefit, consider whether the random inputs were aligned by meaningful trial features and whether the chosen coupling induces positive association for the quantity being compared. A different valid coupling may perform better, but it must preserve each candidate’s marginal distribution. If the paired variance is larger than an independent benchmark, independent simulation may be preferable for that contrast.
| Approach | How the comparison is estimated | Main consideration |
|---|---|---|
| Common random numbers | Average the within-replication differences | Can improve precision when paired outputs have positive covariance; the coupling must preserve both marginals. |
| Independent simulations | Subtract the two separately estimated means | Simple to implement and analyze, but does not use a potentially helpful within-pair association. |
| Separate scenario summaries | Report each design’s operating characteristic without a paired contrast | Useful for describing trade-offs, but does not directly quantify the simulation difference between candidates. |
CRN does not remove Monte Carlo error, make an invalid simulator valid, or guarantee that one design is preferable. The variance reduction applies to a specified contrast under a specified scenario and coupling. It may vary across operating characteristics and scenarios, so a coupling that helps compare rejection probabilities may not help compare expected sample size. Nor does a more precise simulated contrast resolve scientific or practical questions about which operating characteristics should guide design choice.
Common Mistake
Wrong reasoning: “The two designs used the same seed, so the comparison is paired and must have a smaller standard error.” A seed alone neither guarantees meaningful alignment nor guarantees positive covariance.
Correction: Define how each shared input corresponds to trial features, check that both designs retain their correct marginal distributions, and calculate the standard error from the replication-level differences. Compare that paired variability with an independent benchmark. If the coupling creates negative or negligible association, it may provide no benefit or may increase noise.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- If the target is the difference in rejection probabilities for B minus A, how should the replication-level difference be defined, and what does a positive average mean?
- Why can positively correlated paired outputs improve precision? State the variance identity that explains the effect.
- A paired simulation has \(R=400\) replications and the sample standard deviation of its differences is \(0.12\). Calculate the estimated Monte Carlo standard error of the mean difference.
- Why is the independent-samples standard error formula generally inappropriate for outputs generated in pairs?
- Does using the same seed prove that a CRN comparison is valid and more precise? Explain what should be checked instead.
Solutions. (1) Define \(D_r=Y_{Br}-Y_{Ar}\); a positive average means B has the higher estimated rejection probability under that scenario. (2) \(\operatorname{Var}(Y_B-Y_A)=\operatorname{Var}(Y_B)+\operatorname{Var}(Y_A)-2\operatorname{Cov}(Y_A,Y_B)\); positive covariance subtracts from the variance. (3) \(0.12/\sqrt{400}=0.00600\), or 0.600 percentage points if the output is a probability. (4) That formula omits the covariance within pairs; the standard error should use the sample standard deviation of the paired differences divided by \(\sqrt{R}\). (5) No. Check the input mapping, the validity of each design’s marginal simulation, independence across replication pairs, and the observed variability of the paired differences against an independent benchmark.