Compare Power And Estimation Accuracy

Comparing Inferential Performance

A response-adaptive design changes treatment assignments as trial data accumulate. That change can affect not only which treatment is assigned, but also the operating characteristics of the planned analysis. A simulation comparison makes those effects measurable: under a specified data-generating mechanism, repeatedly simulate complete trials, apply the same prespecified analysis rule to each trial, and summarize the resulting decisions and estimates.

This tutorial focuses on three quantities: power, bias, and confidence-interval width. They answer different questions. Power describes how often a test detects a specified nonzero effect; bias describes systematic displacement of an estimator from its target; interval width describes the reported range of plausible effect values. None alone establishes that a design performs well. In particular, interval width must be interpreted alongside coverage, and power must be interpreted alongside Type I error control.

Simulation Setup and Assumptions

A comparison is meaningful only when the simulation represents a clearly specified target. For a simple two-treatment setting, write \(A_i=1\) for assignment to treatment and \(A_i=0\) for control. Suppose the data-generating mechanism gives the two treatment groups outcome means \(\mu_1\) and \(\mu_0\), so the target contrast is \(\tau=\mu_1-\mu_0\), measured in the outcome's units. The simulation must specify the outcome distributions, trial size, adaptive assignment rule, analysis procedure, test, and confidence-interval method. More complex settings may require a different estimand, such as a covariate-adjusted or subgroup contrast; the estimand used to generate and analyze data must still match the scientific target.

Under response-adaptive randomization, the probability of assigning the next participant to treatment may depend on the observed trial history. For example, \(P(A_i=1\mid H_i)=\pi_i(H_i)\), where \(H_i\) denotes the information available before participant \(i\) is assigned. In each simulated replicate, the assignment probabilities must be updated from that replicate's own history. Reusing one assignment sequence across replicates would not reproduce the adaptive design.

The simulation-based summaries estimate performance under the chosen mechanism, not universal properties of the design. Replicates should be generated independently under the specified mechanism; each design should use the same trial horizon, estimand, outcome scenario, significance level, and analysis target. The test and interval procedure must be prespecified and appropriate for the adaptive design. Randomization-based and model-based procedures can differ, but whichever is selected must be applied consistently. In particular, a conventional standard error cannot simply be assumed valid when assignments depend on accumulated outcomes.

When each simulated trial is analyzed at level \(\alpha\), power is evaluated at a specified alternative, such as \(\tau=4\) outcome units. The test's rejection rule is fixed before examining the simulation results. If the test is two-sided, for example, \(D_r=1\) when its p-value is less than \(\alpha\). The resulting estimate is conditional on that specified effect, sample size, assignment rule, outcome mechanism, and analysis method.

Estimating Power, Bias, and Accuracy

The power estimate is the fraction of simulated trials that reject \(H_0\). The bias estimate compares the average simulated treatment-effect estimate with the known target \(\tau\). The empirical standard deviation of the estimates describes their spread across simulated trials; root mean squared error (RMSE) combines that spread with bias. All three estimation summaries use the same target and outcome units.

Here \(\bar{\hat{\tau}}=R^{-1}\sum_{r=1}^{R}\hat{\tau}_r\). RMSE is useful because an estimator can have little bias but substantial variability, or moderate variability but substantial bias. Its relationship to variance and bias follows directly by writing \(\hat{\tau}_r-\tau=(\hat{\tau}_r-\bar{\hat{\tau}})+(\bar{\hat{\tau}}-\tau)\). The deviations from \(\bar{\hat{\tau}}\) sum to zero, so the cross-product vanishes when squared errors are averaged:

Thus squared error reflects both variability around the simulation mean and squared displacement of that mean from the target. The finite-simulation RMSE formula uses divisor \(R\); the empirical standard deviation uses \(R-1\) to estimate the underlying standard deviation.

For interval precision, calculate each interval's width in the outcome units and average across replicates:

A smaller \(\bar{W}\) means shorter intervals on average, not necessarily better-calibrated intervals. A procedure can produce narrow intervals that miss the target too often. Coverage is the proportion of intervals containing \(\tau\); it should be checked as a companion measure, rather than inferred from width. The earlier material on diagnosing Type I error and coverage failures provides the relevant calibration context.

How Allocation Can Change Precision

Allocation affects information about a treatment contrast, but the effect is not captured by the label “adaptive” alone. As a benchmark, suppose a fixed design assigns \(n_1\) participants to treatment and \(n_0\) to control, and outcomes are independent with a common variance \(\sigma^2\) in each arm. The difference in sample means has variance:

This follows because independent sample means have variances \(\sigma^2/n_1\) and \(\sigma^2/n_0\), and the variance of their difference is the sum. The expression illustrates why the amount of information in each arm matters. It is a benchmark, not an automatic variance formula for a response-adaptive trial: history-dependent assignment can change the sampling distribution and may require design-aware inference.

If adaptation tends to increase assignment to one arm, the other arm may have fewer observations, affecting uncertainty for a contrast. At the same time, how informative assignments are depends on the outcome distributions, the adaptive rule, the target estimand, and the estimator. Consequently, the direction and size of a power or precision change cannot be deduced from adaptivity alone. It should be measured using the complete simulated procedure, including its assignment rule and analysis.

Diagnostics, Comparisons, and Limitations

Before interpreting a design comparison, verify that its apparent performance is not an artifact of simulation or analysis choices. Useful checks include the following:

• Use the correct target effect. Compute bias and RMSE against the treatment effect specified by the data-generating mechanism, not against the observed mean estimate.
• Check the rejection rule. Power counts must use the prespecified test and significance level for every replicate. A changed test or threshold makes the counts incomparable.
• Inspect the estimate distribution. A mean near the target can conceal skewness, extreme estimates, or a small number of failed analyses. Summaries such as empirical standard deviation and RMSE help reveal different features.
• Account for failed fits. Report the number and reason for missing estimates or intervals. Silently dropping problematic replicates can make bias, width, and power appear better than they are.
• Pair the width summary with calibration. Investigate coverage and the distribution of interval widths. An unusually narrow average may be a warning if coverage is poor.
• Check adaptive implementation. Assignment probabilities should be based only on each replicate's available history, and the analyzed data must correspond to the assignments actually generated.

The main alternatives to these summaries are not interchangeable measures. Power is a decision-frequency measure, whereas bias and RMSE concern point estimation, and interval width concerns uncertainty summaries. Two designs can have similar power but different bias or width; they can also have similar width but different coverage. A comparison should therefore report the measures relevant to its stated inferential objective, rather than selecting a single favorable number.

Simulation results are conditional on the mechanisms and analysis methods that were actually examined. They do not guarantee performance in a real trial whose outcome process or conduct differs from the simulated conditions. Results also have Monte Carlo error: with too few replicates, differences between designs may be simulation noise. Use enough replicates for the precision needed to distinguish consequential differences, and state the number of simulated trials behind every estimate. Paired replicate comparisons, when used, can improve efficiency, but do not remove the need to check that both designs are evaluated under comparable conditions.