Select Performance Metrics For Design Comparisons

Why Metric Choice Matters

A simulation comparison does not have one universal measure of “better.” A design may allocate more participants to an arm that appears promising, yet have less power for a prespecified contrast; it may estimate an effect with little average bias but produce intervals that cover the true effect too rarely. The meaning of any comparison depends on what is measured, how it is summarized, and which simulated scenario supplies the truth.

For each design and scenario, simulate \(R\) complete trials. A replicate includes the sequential assignment process and the analysis procedure being evaluated. A performance metric is calculated from the replicate-level results and then summarized across replicates. The scenario specifies the data-generating mechanism and the parameter values that define the truth. The design comparison should use metrics that answer the scientific and operational questions motivating the comparison, rather than reporting a long list without interpretation.

Definition: Let \(r=1,\ldots,R\) index simulated trial replicates and \(a\) index treatment arms. A replicate-level result may include a rejection indicator, an estimate, a confidence interval, the number assigned to each arm, or participant outcomes. A simulation performance metric is a specified function of these results, evaluated under a specified design and data-generating scenario. Its simulation estimate is the corresponding summary calculated over the \(R\) replicates.

The notation below uses \(I_r\) for an indicator that equals 1 when a stated event occurs in replicate \(r\) and 0 otherwise; \(\hat{\theta}_r\) for an estimate of a target parameter \(\theta\); and \([L_r,U_r]\) for an interval reported by the analysis. These are replicate-level quantities, not the across-replicate performance summaries.

Testing Metrics: Type I Error and Power

Specify the hypothesis test, its significance level \(\alpha\), and the contrast or family of contrasts before calculating a testing metric. A simulated rejection means the complete analysis procedure rejects \(H_0\) according to its stated rule. For a single test, it might mean a p-value no greater than \(\alpha\). For a procedure controlling multiplicity, it must mean the rejection event corresponding to the prespecified family-level claim. Changing the rejection event changes the metric.

Definition: Under a scenario in which \(H_0\) is true, the type I error probability is the probability that the specified procedure rejects \(H_0\). Its simulation estimate is \(\widehat{\mathrm{Type\ I\ error}}=\frac{1}{R}\sum_{r=1}^{R}I_r\), where \(I_r=1\) if replicate \(r\) rejects and \(I_r=0\) otherwise. Under a scenario in which the specified alternative is true, power is the probability of rejecting \(H_0\), estimated by the same rejection proportion.

Type I error is tied to a particular null scenario: composite null hypotheses may require several parameter settings, and performance at one null setting does not establish performance at all of them. Power likewise depends on the alternative effect size, outcome distribution, sample size, allocation rule, and analysis. Report the scenario and the test procedure alongside either rate. A rejection rate is not “power” unless the scenario makes the relevant alternative true.

Worked Example: Estimating Type I Error and Power

Suppose a hypothetical comparison evaluates the same prespecified test at \(\alpha=0.05\) in two scenarios. In a null scenario, the treatment effect is zero and 27 of \(R=500\) simulated trials reject \(H_0\). In an alternative scenario with a treatment effect of 4 response-score points, 342 of 500 trials reject.

For the null scenario, the estimated type I error rate is

\widehat{\mathrm{Type\ I\ error}}=\frac{27}{500}=0.054.

Thus, the procedure rejected in 5.4% of these simulated null trials. This is an estimate under this specified null scenario, not a guarantee about every possible parameter value in the null hypothesis.

For the alternative scenario, estimated power is

\widehat{\mathrm{Power}}=\frac{342}{500}=0.684.

The test rejected in 68.4% of the simulated trials with a true treatment effect of 4 response-score points. This does not mean that an individual participant has a 68.4% chance of benefiting; it is a trial-level probability of rejection under the stated scenario and procedure.

Estimation Metrics: Bias and Interval Coverage

Estimation metrics require a clearly defined target parameter. For example, \(\theta\) might be the marginal mean difference between two treatments in the simulated population. The replicate estimate \(\hat{\theta}_r\) must target that same quantity. If the estimator targets a different population, subgroup, or effect measure, comparing it directly with \(\theta\) does not yield the intended bias.

Definition: The bias of an estimator \(\hat{\theta}\) for \(\theta\) is \(E(\hat{\theta})-\theta\) under the specified scenario and trial procedure. Its simulation estimate is \(\widehat{\mathrm{Bias}}=\bar{\hat{\theta}}-\theta\), where \(\bar{\hat{\theta}}=\frac{1}{R}\sum_{r=1}^{R}\hat{\theta}_r\). The estimated coverage probability of intervals \([L_r,U_r]\) is \(\widehat{\mathrm{Coverage}}=\frac{1}{R}\sum_{r=1}^{R}C_r\), where \(C_r=1\) if \(L_r\leq\theta\leq U_r\), and 0 otherwise.

Bias has the same units as \(\theta\), and its sign matters: positive estimated bias indicates that the average estimate exceeds the truth. A small bias does not imply that individual estimates are precise or that reported intervals are reliable. Coverage is an interval-procedure property under the scenario: for a nominal 95% interval, the target is a long-run coverage probability of 0.95 across repeated trials under the specified conditions. It is not the probability, after seeing one interval, that the fixed parameter lies inside it.

Worked Example: Bias and Coverage From Replicates

In a hypothetical scenario, the true treatment-effect parameter is \(\theta=2.0\) response-score points. Six simulated trials produce estimates \(1.8, 2.1, 2.4, 1.9, 2.2,\) and \(2.0\) points. Their reported intervals, in the same order, are \([1.4,2.2]\), \([1.8,2.4]\), \([2.1,2.7]\), \([1.5,2.3]\), \([1.9,2.5]\), and \([1.7,2.1]\).

The mean estimate is

\bar{\hat{\theta}}=\frac{1.8+2.1+2.4+1.9+2.2+2.0}{6} =\frac{12.4}{6}=2.067\text{ points}.

Therefore, estimated bias is \(2.067-2.0=0.067\) response-score points. The positive sign indicates a small upward average deviation in these replicates.

The true value 2.0 is included in intervals 1, 2, 4, 5, and 6, but not interval 3 because its lower endpoint is 2.1. Thus,

\widehat{\mathrm{Coverage}}=\frac{5}{6}=0.833.

The estimated coverage is 83.3% for this small illustrative set of replicates. It is not evidence by itself that the procedure’s exact coverage probability is 83.3%; the simulation estimate is based on only six trials.

Allocation and Participant-Outcome Metrics

Adaptive designs make allocation summaries especially relevant. Let \(N_{ra}\) be the number of participants assigned to arm \(a\) in replicate \(r\), and let \(n_r\) be that replicate’s total sample size. The replicate allocation proportion is \(P_{ra}=N_{ra}/n_r\). Its across-replicate mean describes the average share assigned to that arm. This is distinct from an arm’s final assignment probability at a particular interim history: the average share summarizes the realized allocation over whole trials.

Definition: The estimated mean allocation proportion for arm \(a\) is \(\bar{P}_a=\frac{1}{R}\sum_{r=1}^{R}N_{ra}/n_r\). When all replicates have the same total sample size, this is also the average fraction of participants assigned to arm \(a\) across trials. If sample sizes vary, averaging replicate proportions weights each trial equally; pooling all participants instead weights larger trials more heavily.

Participant-outcome metrics need an equally explicit target. One option is the mean observed outcome among participants assigned to a particular arm in each replicate, averaged across replicates. Another is a simulated participant-benefit summary, such as the average outcome under the treatments participants actually received, using the potential outcomes specified by the data-generating mechanism. These summaries answer different questions. In a response-adaptive trial, participants assigned later may differ in their assignment probabilities because earlier responses influenced allocation; an observed arm mean should not automatically be interpreted as a causal treatment contrast.

If the outcome is a response score, report it in score points; if it is a binary outcome, state whether the metric is a proportion, a difference in proportions, or another defined summary. Also state how outcomes are aggregated: a mean of replicate-specific means gives each trial equal weight, whereas pooling all participant records gives each participant equal weight. With equal trial sizes these may coincide; with unequal sizes they generally need not.

Worked Example: Summarizing Allocation and Outcomes

Consider four hypothetical adaptive trials, each enrolling 10 participants to arms A and B. The numbers assigned to A are 6, 4, 7, and 5. The corresponding replicate proportions for A are \(6/10=0.60\), \(4/10=0.40\), \(7/10=0.70\), and \(5/10=0.50\). The mean allocation proportion is

\bar{P}_A=\frac{0.60+0.40+0.70+0.50}{4} =\frac{2.20}{4}=0.550.

Because each trial has 10 participants, the overall mean proportion for B is \(1-0.550=0.450\). Across the four trials, an average of 5.5 participants per trial were assigned to A and 4.5 to B.

Suppose the mean observed response score across all 10 participants was 6.8, 6.1, 7.2, and 6.7 points in the four trials. The equally weighted mean trial outcome is

\bar{Y}_{\mathrm{trial}}=\frac{6.8+6.1+7.2+6.7}{4} =\frac{26.8}{4}=6.70\text{ points}.

This describes the average trial-level mean outcome under the simulated adaptive assignment rules. It is not, by itself, an estimate of a causal treatment effect: that would require a defined causal estimand and an analysis appropriate to it.

Choosing and Checking a Metric

Before running a comparison, write down the metric’s estimand, its replicate-level contribution, its across-replicate aggregation rule, and its units. Then check that the data-generating scenario supports the interpretation. In particular, the truth must be known for bias and coverage; a null scenario must truly satisfy the tested null for type I error; and an alternative scenario must specify the effect for power.

Name the question.
State whether the comparison concerns testing, estimation, interval performance, allocation, or participant outcomes.

Define the target and scenario.
Specify the parameter or event, the data-generating conditions, and—where relevant—the null or alternative value.

Define what one replicate contributes.
For example, record one rejection indicator, one estimate, one coverage indicator, one arm proportion, or one trial-level outcome summary.

State the aggregation and interpretation.
Specify whether trials or participants receive equal weight, report units, and describe exactly which designs and scenarios the result represents.

For a basic justification, let \(I_r\) indicate rejection in replicate \(r\) under a fixed null scenario, and let the rejection probability be \(\pi=P(I_r=1)\). Since an indicator has expectation \(E(I_r)=1\cdot P(I_r=1)+0\cdot P(I_r=0)=\pi\), the average rejection indicator satisfies

E\left(\frac{1}{R}\sum_{r=1}^{R}I_r\right) =\frac{1}{R}\sum_{r=1}^{R}E(I_r) =\frac{1}{R}\sum_{r=1}^{R}\pi =\pi.

Thus, the rejection proportion targets the scenario-specific rejection probability; under the null it estimates type I error, and under the alternative it estimates power. The same indicator argument applies to coverage indicators. For bias, linearity of expectation gives \(E(\bar{\hat{\theta}}-\theta)=E(\hat{\theta})-\theta\) when replicate estimates have the same expectation. These identities justify the summaries as targets; they do not make a small simulation precise.

Common Mistakes and Limitations

Calling every rejection rate power. A rejection proportion estimates power only under a specified alternative; under a true null, it estimates type I error.
Reporting a metric without its scenario. A power value without the effect size, outcome model, sample size, and analysis procedure is difficult to interpret or reproduce.
Confusing bias with accuracy. A near-zero average bias can coexist with considerable replicate-to-replicate variability or poor interval coverage.
Changing the weighting silently. The average of trial-specific participant means and a pooled participant mean can differ when trial sizes vary.
Dropping failed analyses without defining the rule. If an estimator or interval is unavailable in some replicates, silently excluding those trials changes the population being summarized. Report the handling rule and the frequency of unavailable results.
Overinterpreting allocation or outcome summaries. Allocation proportions describe where participants were assigned; observed outcome summaries alone do not establish causal benefit.

No single metric captures all design objectives. Testing, estimation, interval performance, allocation, and participant outcomes are complementary summaries. They should be defined before comparing designs, and their interpretation should remain tied to the scenario and analysis procedure that generated them.

Check Your Understanding

Answer each question, then compare with the solutions below.

In 400 simulated trials under a true null, 18 trials reject. What is the estimated type I error rate?
In a scenario where the true effect is 3 points, the mean estimate over 100 trials is 3.25 points. What is the estimated bias, including its units and direction?
Out of 80 intervals, 68 include the true parameter. What is the estimated coverage?
Two designs have the same average allocation to an experimental arm, but different average observed outcomes. Does this alone show which design has greater causal treatment benefit?

Solutions. (1) \(18/400=0.045\), so the estimated type I error rate is 4.5% for that null scenario and test. (2) The estimated bias is \(3.25-3.00=0.25\) points, an upward average deviation. (3) Estimated coverage is \(68/80=0.85\), or 85%. (4) No. The outcome summaries describe results under the simulated assignment rules; a causal-benefit claim requires a defined causal estimand and suitable analysis, not merely a difference in observed outcomes.