Why a Difference Needs an Uncertainty Statement
A simulation benchmark may report that one multi-omics integration method has a higher average recovery score than another. The observed gap alone does not show how precisely the benchmark estimates the contrast. A small gap could be a stable feature of the simulation design, or it could be mostly Monte Carlo noise from a limited number of generated data sets. And even a precise average across a chosen set of scenarios may conceal substantial variation in which method performs better from one scenario to another.
The natural comparison is paired when both methods are evaluated on the same simulated data set. This tutorial develops uncertainty statements for those paired performance contrasts. The first essential decision is the target: an average contrast conditional on a fixed scenario, a weighted average over a specified set of scenarios, or an average intended to generalize to a broader distribution of scenarios. These targets need not have the same interval.
The sign convention matters: a positive \(d_r\) favors A under this score definition. For an error metric where smaller is better, reverse the subtraction or define the direction explicitly before interpreting signs. The target for a fixed scenario is the expected paired difference over the simulation randomness used to generate and analyze data under that scenario. The estimated Monte Carlo standard error of \(\bar{d}\) is \(SE(\bar{d})=s_d/\sqrt{R}\).
Paired Intervals and Their Justification
A conventional \(t\) confidence interval for the fixed-scenario mean contrast is
Here \(t_{1-\alpha/2,\;R-1}\) is the relevant quantile of the \(t\) distribution with \(R-1\) degrees of freedom. The interval is exact when the replication-level contrasts are independent and normally distributed with a common mean and variance. With a moderate or large number of independent replications, the central limit theorem often justifies an approximate interval for the mean, but severe skewness or influential outliers can make that approximation unreliable.
The pairing is not a cosmetic choice. Suppose the two methods' scores on one generated data set have variances \(\sigma_A^2\) and \(\sigma_B^2\), and covariance \(\operatorname{Cov}(Y_A,Y_B)\). The variance of their paired difference is
When both methods tend to do well or poorly on the same data sets, the covariance is positive and pairing can reduce the variance of the contrast. This is especially useful when data-set difficulty varies substantially. Pairing is valid only when each A score is matched to the B score from the same generated replicate, or to a genuinely defined matched unit. Arbitrarily pairing scores from unrelated simulations does not provide this benefit and can distort the estimated variance.
Worked Example: A Paired Confidence Interval Within One Scenario
Suppose a hypothetical benchmark runs methods A and B on each of \(R=39\) independently generated data sets under one fixed scenario. The score is a recovery measure from 0 to 1, with larger values better. The observed paired differences have \(\bar{d}=0.0100\) score units and \(s_d=0.0643\) score units. Assume the replicate differences are independent and reasonably compatible with a normal model, with no single outlier dominating the mean.
The estimated Monte Carlo standard error is
With \(df=R-1=38\), use the approximate 95% critical value \(t_{0.975,38}=2.024\). The margin of error is
Therefore the interval is
For this scenario, the estimated mean advantage of A is 0.0100 score units. The 95% interval includes zero, so these replications do not establish a nonzero mean difference at the two-sided 5% level. The interval also makes the uncertainty more informative than that statement alone: it is compatible with a small mean advantage for B and with a larger mean advantage for A. It describes uncertainty in the scenario-specific expected contrast, not the range of scores or contrasts on individual future data sets.
Monte Carlo Error Is Not Scenario Variation
A simulation study often includes several scenarios, such as different signal strengths, missingness patterns, or noise levels. For a fixed scenario \(s\), write the estimated paired mean difference as \(\bar{d}_s\), its replication-level standard deviation as \(s_s\), and the number of replications as \(R_s\). Its Monte Carlo standard error is \(s_s/\sqrt{R_s}\). It quantifies how much the estimated scenario mean would vary if the same scenario were simulated and analyzed again with the same replication count.
Scenario variation asks a different question: how do the expected contrasts differ across scenarios? It is not eliminated by increasing \(R_s\). A large replication count can make each scenario mean very precise while leaving genuine differences between scenarios intact. Whether that between-scenario variation belongs in the uncertainty interval depends on the estimand.
For a specified set of \(S\) scenarios with prespecified weights \(w_s\), where \(w_s\geq 0\) and \(\sum_s w_s=1\), the weighted target and estimate are
If simulation replications are independent across scenarios, the estimated Monte Carlo variance of this weighted estimate is \(\sum_s w_s^2s_s^2/R_s\). This variance concerns simulation error in estimating the fixed weighted average. It does not treat the chosen scenarios as a random sample from all possible scenarios.
Worked Example: A Weighted Average Across Fixed Scenarios
Consider three hypothetical scenarios assigned equal weight. Each has 20 independent paired replications. The scenario mean differences are 0.040, -0.010, and 0.020 score units, with within-scenario standard deviations 0.060, 0.040, and 0.050 score units, respectively. The target is explicitly the mean contrast across these three scenarios with equal weighting.
The estimated weighted contrast is
The Monte Carlo standard error, accounting for the separate replication counts, is
Using a normal approximation, a 95% Monte Carlo interval is
This interval concerns the weighted mean for exactly these three scenarios. Its assumptions include independent replications within each scenario, independence of simulation runs across scenarios, and a sufficiently accurate normal approximation for the weighted estimate. The positive interval does not mean A wins in every scenario: the second scenario's estimated contrast is negative. Nor does this interval describe how much the true method difference varies from one scenario to another.
Intervals for Generalization Across Scenarios
If scenarios are regarded as independent draws from a defined population of plausible settings, the target may instead be the mean contrast over that scenario population. Then scenario-to-scenario variation contributes to uncertainty about the population mean. A simple \(t\) interval can be formed from \(S\) independently sampled scenario contrasts, provided their distribution is suitably well behaved and the sampling design supports that interpretation. That interval is not interchangeable with a Monte Carlo interval conditional on a fixed scenario set.
Worked Example: A Mean Contrast Over Sampled Scenarios
Suppose six hypothetical scenarios are treated as independent draws from a specified population of scenarios. For each scenario, enough replication is used that its mean contrast is estimated precisely relative to differences among scenarios. The six scenario-level contrasts, in score units, are \(-0.020,\ 0.000,\ 0.010,\ 0.020,\ 0.030,\ 0.050\). Assume these scenario draws are independent and their contrast distribution is reasonably compatible with a normal model.
Their mean is \(\bar{d}_{\text{sc}}=0.090/6=0.0150\) score units. The deviations from the mean are \(-0.035,-0.015,-0.005,0.005,0.015,0.035\), so the sum of squared deviations is \(0.00295\). Thus the scenario-level sample standard deviation and standard error are
With \(df=5\), \(t_{0.975,5}=2.571\). The 95% \(t\) interval for the mean across the scenario population is
This interval is wider than the fixed-scenario Monte Carlo interval in the previous example because it reflects variation across sampled scenarios as well as uncertainty in their mean. It supports cautious conclusions about average performance over the stated scenario population, not a guarantee that A performs better in each scenario. With only six scenarios, the normality and sampling assumptions deserve particular scrutiny; simulation replication within each scenario cannot compensate for a very small or unrepresentative number of scenarios.
Diagnostics, Alternatives, and Limits
Before reporting an interval, inspect the replication-level paired differences, not only the two methods' separate score distributions. A histogram or dot plot can reveal skew, outliers, or clusters; a plot against replication order can reveal drift or dependence. Check that each pair shares the same simulated data set, that all replications use the intended scenario, and that the metric and sign convention are consistent. If a few large differences dominate the mean, report that sensitivity and consider whether the mean is still the intended summary.
When the \(t\) approximation is doubtful, a bootstrap can resample the paired replication rows \((Y_{Ar},Y_{Br})\), or equivalently the paired differences \(d_r\), within each fixed scenario. Resampling methods preserve the pairing and approximate the sampling distribution of the chosen statistic under the observed replication distribution. They do not repair dependent replications, an unrepresentative scenario set, or too few independent draws to represent heavy tails. For a weighted multi-scenario estimand, resample within scenarios and recompute the same weighted estimate in each bootstrap sample. If scenarios themselves were randomly sampled and the target generalizes across scenarios, the resampling design must also reflect that scenario sampling.
| Approach | What it estimates | Important qualification |
|---|---|---|
| Paired \(t\) interval | Mean contrast within a fixed scenario | Uses the paired differences; depends on independence and a suitable normal or large-sample approximation. |
| Unpaired comparison of method means | Difference between marginal method averages | Can discard the shared-data pairing and misstate uncertainty when outcomes are correlated within replicate. |
| Paired bootstrap interval | Uncertainty in a statistic such as a mean paired contrast | Must resample matched units; bootstrap validity still depends on independent representative replications. |
| Scenario-level interval | Mean contrast over a sampled scenario population | Requires an appropriate scenario sampling interpretation and enough scenarios to estimate heterogeneity. |
A confidence interval does not measure scientific importance by itself. For a bounded recovery score, a difference of 0.01 may be negligible or consequential depending on the application. State the score scale and units, and, when justified, compare the interval with a prespecified practically important difference. Also avoid interpreting a confidence interval as a range containing most individual-replicate outcomes: a confidence interval for a mean targets the expected contrast, whereas a prediction interval targets a future replicate or scenario and is generally wider.
Reporting a Defensible Contrast
A concise report should specify the score and its direction, the paired unit, the number of independent replications per scenario, the estimated contrast and interval in score units, and the scenario weighting or sampling target. It should identify whether the interval reflects Monte Carlo error alone or scenario variation as well. If methods share simulated inputs, say so; if simulation runs are dependent through reused seeds or other design features, the simple standard error formula may not apply. These details make a difference interval interpretable and reproducible.
Choose a fixed-scenario mean, a weighted average over a prespecified scenario set, or a mean over a defined scenario population.
For each matched replicate, subtract B's score from A's score, using one consistent direction and metric.
Confirm independent simulation units, correct matching, and no severe departures that undermine the chosen interval.
Give the estimate, uncertainty interval, units, replication count, and whether scenario variation is included.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Method A and method B are evaluated on the same generated data sets. Why should the uncertainty calculation use replicate-level differences rather than treating all A and B scores as unrelated?
- A 95% interval for a fixed-scenario mean contrast is \([-0.01,0.03]\) score units. What does it say, and what does it not say?
- A study increases replications from 20 to 200 in each of three fixed scenarios. Which kind of uncertainty is directly reduced, and which is not automatically reduced?
- When is it appropriate for an interval to include variation among scenarios, rather than only Monte Carlo error within scenarios?
Solutions. (1) Matched differences retain the within-data-set covariance. Their variance is \(\operatorname{Var}(Y_A)+\operatorname{Var}(Y_B)-2\operatorname{Cov}(Y_A,Y_B)\), which can be smaller than the variance implied by an unpaired analysis. (2) It is compatible with a small mean advantage for B or a mean advantage for A up to about 0.03 score units under the model and sampling assumptions. Because it contains zero, it does not establish a nonzero mean contrast at the corresponding two-sided 5% level. It is not an interval for most individual replicate differences. (3) More replications reduce Monte Carlo error in each fixed-scenario mean, approximately at the rate \(1/\sqrt{R}\) when the variance is stable. They do not remove true differences among the three scenarios. (4) Include scenario variation when the estimand generalizes to a defined population of scenarios sampled by a suitable design. For a prespecified fixed scenario set, report its weighted mean and distinguish its Monte Carlo error from variation among the selected scenarios.