Use Paired Replicates To Compare Designs Efficiently

Why Pair Replicates?

A simulation comparison asks how two trial designs differ in a performance measure, such as a rejection proportion or an estimator’s average error. Even when both designs are simulated under the same scenario, the Monte Carlo estimates can be noisy: each set of simulated trials is only a finite sample from the design’s operating distribution. A useful way to reduce that noise is to make each replicate under one design correspond to a replicate under the other design, with both exposed to carefully chosen common random inputs.

The aim is not to make the simulated trials identical. The designs may allocate patients differently, follow different adaptive paths, or produce different outcomes. Instead, pairing seeks to make the two replicate-level performance values move together. If a particular replicate’s shared conditions make the performance measure unusually high under both designs, their difference may still be relatively stable. The comparison then uses that difference directly.

The sign convention matters: with \(D_r=M_{Ar}-M_{Br}\), a positive \(\bar{D}\) means the simulated mean performance value is higher under A. Whether that is desirable depends on the metric. For example, a higher rejection probability can mean greater power under an alternative, but a higher rejection probability under the null can mean worse Type I error control. The contrast describes a difference; it does not by itself say which design is preferable.

What Is Shared, and What Must Remain Design-Specific?

Common random numbers (CRN) are random inputs deliberately shared, or coupled, across designs to create a useful dependence between their replicate outcomes. Depending on the simulation, shared inputs can include a replicate’s nuisance-parameter scenario, patient covariates, or latent outcome-generating values indexed by patient and treatment. Randomness that determines allocation under a particular design may need to differ or be transformed according to that design’s allocation probabilities. Each design must still have its intended marginal simulation distribution.

A sound pairing therefore has two requirements. First, each design’s replicate must be generated according to that design’s specified mechanism; sharing random inputs must not change either marginal distribution. Second, pairs from different replicate indices must be independent, or have a dependence structure that is explicitly accounted for. Within a pair, however, \(M_{Ar}\) and \(M_{Br}\) are generally dependent. That dependence is the point of the method.

For example, the same scenario value can be used for both designs in replicate \(r\), while each design applies its own allocation rule to its sequential history. If response outcomes are generated using latent uniforms, a shared set of appropriately indexed uniforms can induce related outcomes without requiring the designs to assign the same treatment. The shared inputs define a coupling: a joint way to generate two outputs that preserves each output’s correct marginal distribution.

Why Pairing Can Reduce Monte Carlo Error

The paired estimator is an average of replicate contrasts. Its variance depends on the variance of a contrast, not on the two marginal variances alone. The covariance induced within each pair can either reduce or increase that variance.

Because the \(R\) contrasts are independent across replicates, the variance of their mean is \(\operatorname{Var}(\bar{D})=\operatorname{Var}(D_r)/R\). In practice, estimate the standard deviation of the observed contrasts by \(s_D\), the sample standard deviation of \(D_1,\ldots,D_R\). Then the estimated Monte Carlo standard error is:

This standard error quantifies simulation error in estimating the difference \(\Delta\). It is not the standard error of an effect estimated from one clinical trial, and it does not include uncertainty about whether the simulation assumptions represent real trial settings.

Worked Comparisons

A Short Derivation and a Practical Comparison

The reason the paired standard error is appropriate follows directly from averaging the contrasts. Write \(\mu_D=E(D_r)\) and \(\sigma_D^2=\operatorname{Var}(D_r)\). Under independence across replicate pairs, the variance of a sum is the sum of the variances. Hence:

Replacing \(\sigma_D\) by its sample estimate \(s_D\) gives \(SE(\bar{D})=s_D/\sqrt{R}\). This derivation also shows why the pairing must be preserved in the calculation: the variability to estimate is the variability of \(D_r\), including the within-pair covariance.

An independent-replicate comparison estimates each design’s mean separately and combines their standard errors. For equal replicate counts and independent outputs across designs, its variance is approximately \(\operatorname{Var}(M_A)/R+\operatorname{Var}(M_B)/R\). A paired comparison instead has variance \(\{\operatorname{Var}(M_A)+\operatorname{Var}(M_B)-2\operatorname{Cov}(M_A,M_B)\}/R\). Positive covariance favors pairing; negative covariance favors independent simulation. Both approaches target the same difference when the marginal simulation distributions are correct.

Diagnostics, Pitfalls, and Limitations

After running a set of pairs, inspect the standard deviation of the contrasts and the association between the two designs’ replicate values. A strong positive association often accompanies a smaller paired standard error than the independent calculation, while weak or negative association signals little gain or possible loss. These are diagnostics of the realized pairing, not guarantees that a future simulation will have the same covariance. The practical question is whether the pairing is both correctly constructed and useful for the estimand.

One common mistake is to assume that using the same random-number seed for both designs automatically creates a good pair. Wrong reasoning: “The seeds match, so replicate 1 under A and replicate 1 under B must experience the same conditions.” If the designs consume random numbers in different orders or amounts, corresponding draws may represent different events. Correction: define what inputs are intended to be shared, index or generate them so their meaning is clear, and preserve each design’s own allocation and response mechanism. Then check the resulting within-pair association and contrast variability.

Another mistake is to calculate a paired standard error while discarding the pairing information. For instance, calculating separate standard errors for the two designs and combining them as if outputs were independent ignores their covariance. Conversely, treating many patients within a simulated trial as if they were independent replicate pairs confuses the trial’s sampling variability with Monte Carlo variability across simulated trials.

Pairing also has limits. A coupling that gives little covariance may not reduce computation or Monte Carlo error. A negative covariance can make precision worse. For several designs, comparisons should use a clearly defined pairing and a contrast for each prespecified comparison; the paired standard error for one contrast does not resolve uncertainty across every possible comparison. Finally, common random numbers do not correct a misspecified data-generating mechanism, a coding error, or an unsuitable performance metric. They improve the efficiency of estimating a simulation-defined comparison, not the validity of the simulation’s assumptions.