Estimating Uncertainty in Paired Design Differences

Why Uncertainty in a Design Difference Is Paired

A simulation comparison usually asks how two candidate designs differ on a specified performance metric: for example, their mean absolute estimation error or the percentage of simulated analyses that converge. A point estimate alone does not show how much that difference would vary if the simulation were repeated. A confidence interval describes the Monte Carlo uncertainty in the estimated difference, provided the simulation replicates and the comparison are handled appropriately.

The comparison should generally be made within each replicate when the two designs were simulated under matched conditions. This is the replicate-level contrast: calculate each design's metric for a replicate, subtract one from the other, and then summarize those differences. The earlier material on common random numbers explains how shared random inputs can create this useful matching. Pairing preserves that structure rather than treating the two sets of results as unrelated.

Definition: For replicate \(r\), let \(Y_{Ar}\) and \(Y_{Br}\) be the performance metrics for designs A and B, calculated using the same replicate-level conditions. Define the paired difference \(D_r=Y_{Ar}-Y_{Br}\). For \(R\) independent replicate pairs, the estimated mean difference is \(\bar{D}=\frac{1}{R}\sum_{r=1}^{R}D_r\). A negative value means design A's metric is lower on average than design B's; whether that is better depends on what the metric measures.

The target is the expected difference in the performance metric under the simulation setup being evaluated. It is not the difference between two individual simulated datasets, and it is not automatically a statement about the difference in clinical outcomes. The metric, scenario, and direction of subtraction must be stated so the estimate has a clear interpretation.

The Paired t Confidence Interval

Calculate the sample standard deviation of the paired differences, \(s_D\), and their standard error, \(SE_{\bar{D}}=s_D/\sqrt{R}\). A two-sided \(100(1-\alpha)\%\) paired t confidence interval for the mean difference is:

\bar{D}\ \pm\ t_{1-\alpha/2,\,R-1}\frac{s_D}{\sqrt{R}}

Here \(t_{1-\alpha/2,\,R-1}\) is the corresponding quantile of the t distribution with \(R-1\) degrees of freedom. This interval is exact if the \(D_r\) are independent and normally distributed with a common mean and variance. With a sufficiently large number of independent replicate pairs, it is often approximately justified by the central limit theorem, even when the differences are not exactly normal. The number of pairs \(R\), not the number of subjects within a simulated dataset, determines the degrees of freedom and standard error for this Monte Carlo interval.

The interval describes repeated-simulation uncertainty in estimating the mean paired difference. In frequentist terms, the procedure has the stated long-run coverage if its assumptions hold; it does not mean there is a \(95\%\) probability that a fixed mean difference lies in the particular interval calculated. It also does not describe the range of differences expected across individual replicates.

Why Pairing Can Improve Precision

For a paired contrast, the variance is governed by the variability of the differences, not by the variability of each design's metric in isolation. The identity below shows the role of within-pair covariance:

\operatorname{Var}(Y_A-Y_B) =\operatorname{Var}(Y_A)+\operatorname{Var}(Y_B)-2\operatorname{Cov}(Y_A,Y_B)

When matched replicate conditions make the two metrics positively correlated, the covariance term reduces the variance of their difference. Pairing does not guarantee a narrower interval: if the paired values are weakly or negatively correlated, the gain can be small or absent. Its purpose is to estimate the intended contrast while respecting how the simulation was generated.

A short derivation connects the variance result to the interval. If independent replicate pairs have a common variance \(\sigma_D^2\), then the variance of their mean is:

\operatorname{Var}(\bar{D}) =\operatorname{Var}\left(\frac{1}{R}\sum_{r=1}^{R}D_r\right) =\frac{1}{R^2}\sum_{r=1}^{R}\operatorname{Var}(D_r) =\frac{\sigma_D^2}{R}

The middle equality uses independence across replicate pairs. Replacing the unknown \(\sigma_D\) by \(s_D\) estimates the standard error as \(s_D/\sqrt{R}\). If the differences are normally distributed, the standardized statistic \((\bar{D}-\mu_D)/(s_D/\sqrt{R})\) follows a t distribution with \(R-1\) degrees of freedom, which gives the interval above.

Worked Example: Mean Difference in Estimation Error

Suppose a hypothetical simulation compares mean absolute relative error for designs A and B. The metric is reported as a percentage, and each row uses matched replicate conditions. The replicate-level differences, in percentage points, are \(-0.8,-0.4,-0.6,-0.2,-0.5\). Assume for this illustration that the five differences are independent and normally distributed.

The mean difference is \(\bar{D}=(-0.8-0.4-0.6-0.2-0.5)/5=-2.5/5=-0.500\) percentage points. The deviations from the mean are \(-0.3,0.1,-0.1,0.3,0\). Thus \(s_D=\sqrt{[0.09+0.01+0.01+0.09+0]/(5-1)}=\sqrt{0.0500}=0.2236\) percentage points, and \(SE_{\bar{D}}=0.2236/\sqrt{5}=0.1000\) percentage points.

For a 95% interval with \(df=4\), \(t_{0.975,4}=2.776\). Therefore:

-0.500\ \pm\ 2.776(0.1000) =-0.500\ \pm\ 0.2776 =[-0.7776,\,-0.2224]\text{ percentage points}

Under the simulation setup, the estimated mean absolute error for A is 0.500 percentage points lower than for B. The interval is entirely below zero, so it is consistent with a lower mean error for A. With only five pairs, the normality assumption matters and cannot be assessed reliably from the differences; this calculation illustrates the method, not a recommendation to use so few replicates.

Worked Example: Matched Replicates Versus an Unpaired Comparison

Consider six hypothetical matched replicates with metric values \(Y_A=(10,12,14,16,18,20)\) and \(Y_B=(9,12,15,15,18,21)\). Suppose lower values are preferable. The paired differences \(Y_A-Y_B\) are \(1,0,-1,1,0,-1\), with mean \(0\).

The squared deviations from the mean sum to \(1+0+1+1+0+1=4\), so \(s_D=\sqrt{4/5}=0.8944\), and \(SE_{\bar{D}}=0.8944/\sqrt{6}=0.3651\). With \(df=5\), \(t_{0.975,5}=2.571\), giving the 95% paired interval:

0\ \pm\ 2.571(0.3651) =[-0.9387,\ 0.9387]

For comparison, the sample variances of the A and B values are \(14\) and \(18\). An unpaired standard error for the difference in means would be \(\sqrt{14/6+18/6}=\sqrt{5.333}=2.309\), much larger than the paired standard error. The paired values rise together across replicates, so taking their differences removes much of that shared variation. The paired interval includes zero: these six pairs do not establish a nonzero mean difference at the 5% two-sided level. The unpaired calculation discards the matching and answers with substantially less precision; it is not a substitute for the paired analysis when the replicate pairs are the actual simulation units.

Interpreting the Interval for a Design Decision

Statistical uncertainty and practical importance are different questions. An interval that excludes zero supports a nonzero mean difference under the model and assumptions, but the size of that difference may still be operationally negligible. Conversely, an interval that includes zero does not prove the designs are equivalent. If a decision requires a meaningful improvement of at least a specified size, compare the interval with that decision threshold, not only with zero.

The sign must be interpreted in the context of the metric. For an error metric, negative \(D_r=Y_{Ar}-Y_{Br}\) favors A if smaller error is desired. For a success metric such as convergence proportion, a negative difference may instead favor B. State both the subtraction order and what a larger metric means before interpreting the result.

Worked Example: Is the Estimated Difference Practically Important?

In another hypothetical simulation, the paired differences in mean absolute estimation error, again A minus B and measured in percentage points, are \(0.4,0.2,0.5,0.1,0.3,0.0,0.6,0.3\). Assume independent replicate pairs and an approximately normal distribution of differences. Suppose a reduction of at least 0.10 percentage points is the prespecified practical target.

The mean is \(\bar{D}=2.4/8=0.300\) percentage points. The deviations from 0.300 are \(0.1,-0.1,0.2,-0.2,0,-0.3,0.3,0\); their squared deviations sum to \(0.28\). Thus \(s_D=\sqrt{0.28/7}=0.200\), and \(SE_{\bar{D}}=0.200/\sqrt{8}=0.07071\).

With \(df=7\), \(t_{0.975,7}=2.365\). The 95% interval is \(0.300\pm2.365(0.07071)=0.300\pm0.1672\), or \([0.1328,0.4672]\) percentage points. Since A minus B is positive, A has higher estimated mean absolute error. The entire interval exceeds 0.10, so this result supports the conclusion that A's error exceeds B's by more than the stated practical target under this simulation setup.

Diagnostics, Alternatives, and Limitations

Before reporting a paired interval, check how the replicates were generated and how the metrics were calculated. Every included \(D_r\) must correspond to the same defined replicate-level conditions for both designs. Replicate pairs should be independent of one another; shared random inputs within a pair are compatible with this assumption, but dependence across pairs is not. Also confirm that the metric is defined consistently and has the same units in both designs.

Inspect the paired differences. A plot or summary of \(D_r\) can reveal extreme values, skewness, or distinct clusters. These may make a small-sample t interval unreliable.
Check for missing pairs. If a metric is unavailable for one design in a replicate, silently dropping just one side changes the pairing. Complete-pair analysis estimates a contrast among pairs with both metrics observed; if failures cause missingness, that target may differ from the intended comparison.
Separate replication from subjects. Treating subjects within simulated datasets as independent replicates understates Monte Carlo uncertainty because the metric is computed at the replicate level.
Verify reproducibility and independence. Reusing identical random streams in a way that duplicates replicate pairs, or introducing dependence between nominally separate replicates, invalidates the usual standard error calculation.

An unpaired Welch interval is an alternative when the two sets of simulation replicates are genuinely independent and no matching exists. It estimates a difference in means using the separate sample variances. When meaningful replicate matching is available, that approach generally loses information by omitting the covariance. A bootstrap that resamples whole pairs can be considered when the difference distribution makes the t approximation questionable, but it still requires independent representative pairs and enough replicates for reliable resampling. Neither method fixes biased performance metrics, an unsuitable simulation scenario, or informative simulation failures.

The paired t interval is for the mean difference in the defined simulation estimand and scenario. It does not summarize the full distribution of replicate outcomes, account for uncertainty about whether the simulation model is correct, or by itself quantify uncertainty in a ranking selected from many designs. Those are distinct questions. In particular, increasing \(R\) reduces Monte Carlo uncertainty but does not remove model misspecification or make an inappropriate metric meaningful.

Common Pitfall

Common mistake: “The mean metric for A is 3.0 and for B is 3.5, so I will calculate separate standard errors and subtract the means.” This ignores the matched replicate structure and may give an unnecessarily imprecise interval. Calculate \(D_r=Y_{Ar}-Y_{Br}\) for each matched pair first, then estimate the mean and standard error from those differences. The paired interval is valid only when the pairs are correctly matched and independent across replicates.

Check Your Understanding

Use the definitions and assumptions above. The solutions follow the questions.

For each replicate, two designs produce metrics \(Y_A\) and \(Y_B\). If \(D=Y_A-Y_B\), what does a negative mean difference tell you, and what additional information is needed to decide which design is preferable?
A simulation has \(R=25\) independent replicate pairs. What are the degrees of freedom for the paired t interval, and what sample standard deviation enters its standard error?
Why can a paired interval be narrower than an unpaired interval when common random numbers are used?
A 95% paired interval for a difference in error is \([-0.4,0.1]\) percentage points. Is a nonzero difference established at the 5% two-sided level? Does this show the designs are equivalent?

Solutions. (1) A's metric is lower on average than B's, but whether lower is preferable depends on what the metric measures. (2) \(df=25-1=24\); use the sample standard deviation \(s_D\) of the 25 replicate-level differences, so \(SE_{\bar{D}}=s_D/\sqrt{25}\). (3) Positive within-pair covariance reduces \(\operatorname{Var}(Y_A-Y_B)\), so the differences can vary less than either metric separately. (4) The interval includes zero, so a nonzero mean difference is not established at that level. It does not demonstrate equivalence; that requires an appropriate equivalence margin and analysis.