Why Plot Benefit Against Power?
A design can affect both the outcomes experienced by trial participants and the evidence the trial produces. A design that assigns more participants to a treatment with a favorable effect may increase expected participant benefit, while changing the precision and power of the treatment comparison. A benefit-power curve puts these two performance measures on the same graph. It makes a trade-off visible without implying that one measure should always take priority.
The curve is a comparison of designs under a stated set of conditions, not a universal ranking. Its meaning depends on what counts as patient benefit, the treatment-effect scenario used to evaluate power, and the trial’s analysis procedure. Designs should therefore be compared on a common population, sample size, outcome scale, alternative hypothesis, and significance level unless the purpose of the comparison explicitly changes one of these.
Define the Two Quantities Before Plotting
Suppose a trial has \(N\) participants and compares a control, coded \(A_i=0\), with a treatment, coded \(A_i=1\). Let \(Y_i(0)\) and \(Y_i(1)\) denote participant \(i\)'s potential outcomes under control and treatment. Assume the outcome is oriented so that larger values are better. The individual treatment effect is \(\tau_i=Y_i(1)-Y_i(0)\). A design \(d\) is a specified allocation policy together with the trial size and analysis procedure.
This benefit is an expected incremental outcome for participants enrolled in the trial; it is not a claim about the benefit to every individual, nor is it automatically a measure of population-wide health impact. If the outcome is a favorable continuous score, \(B_d=0.04\) means an expected gain of 0.04 score units per participant compared with the all-control benchmark. The sign and interpretation must follow the chosen outcome scale. For an outcome where smaller values are better, first define a benefit measure with its direction made explicit.
The definition also clarifies how allocation enters the benefit calculation. Conditional on the potential outcomes and the allocation history, let \(\pi_i\) be the probability of assigning participant \(i\) to treatment. Since \(A_i\) is either 0 or 1, $$ E[Y_i(A_i)-Y_i(0)]=E[A_i\{Y_i(1)-Y_i(0)\}]=E[\pi_i\tau_i]. $$ Thus, benefit depends on both treatment effects and the allocation probabilities applied to participants. It cannot generally be inferred just from the overall proportion assigned to treatment when effects differ across participants or allocation depends on trial history.
Power must refer to a particular alternative and a particular test. In a response-adaptive design, both the sequence of assignments and the outcomes can vary from trial to trial; unconditional power averages over that trial-to-trial variation. A plotted power value is not a guarantee that the trial will succeed. It is a probability calculated under the specified alternative and design assumptions.
Estimating Coordinates From Simulated Trials
When the design’s operating characteristics are evaluated by simulation, simulate \(M\) complete trials under the same alternative and target population for each design. Let \(R_m=1\) if trial \(m\)'s prespecified test rejects \(H_0\), and \(R_m=0\) otherwise. In the simulation, potential outcomes are generated or otherwise available for each participant, so the incremental outcome under the assigned treatment can be calculated.
These are simulation estimates of the power and expected benefit coordinates. The same outcome orientation, control benchmark, simulation scenario, and analysis rules should be used across designs. In an actual trial, each participant’s outcome under the treatment they did not receive is unobserved. Estimating a benefit-power curve therefore usually requires assumptions or external information about potential outcomes, rather than simply calculating both coordinates from observed outcomes.
Worked Example: Calculating Benefit and Power Coordinates
Consider three hypothetical designs for trials of \(N=100\) participants. Each design is evaluated in \(M=2{,}000\) simulated trials under the same alternative, using the same test and significance level. The outcome is a favorable score, in score units. The simulation records the total incremental score, relative to control, across all participants and trials.
| Design | Trials rejecting \(H_0\) | Total incremental score |
|---|---|---|
| Fixed allocation | 1,100 | 6,000 |
| Adaptive, moderate | 1,040 | 8,200 |
| Adaptive, stronger | 900 | 10,800 |
There are \(MN=2{,}000\times100=200{,}000\) participant-trial records for each design. For fixed allocation, $$ \widehat{P}_{\mathrm{fixed}}=\frac{1{,}100}{2{,}000}=0.550, \qquad \widehat{B}_{\mathrm{fixed}}=\frac{6{,}000}{200{,}000}=0.0300 $$ score units per participant. For moderate adaptation, $$ \widehat{P}_{\mathrm{moderate}}=\frac{1{,}040}{2{,}000}=0.520, \qquad \widehat{B}_{\mathrm{moderate}}=\frac{8{,}200}{200{,}000}=0.0410 $$ score units per participant. For stronger adaptation, $$ \widehat{P}_{\mathrm{stronger}}=\frac{900}{2{,}000}=0.450, \qquad \widehat{B}_{\mathrm{stronger}}=\frac{10{,}800}{200{,}000}=0.0540 $$ score units per participant.
With expected benefit on the horizontal axis and power on the vertical axis, the plotted points are \((0.0300,0.550)\), \((0.0410,0.520)\), and \((0.0540,0.450)\). In these hypothetical results, the designs with higher estimated participant benefit have lower estimated power under the stated alternative. The numbers describe these simulated conditions; they do not establish that stronger adaptation must have this pattern.
Read the Curve as a Set of Choices
A point farther to the right represents greater estimated expected participant benefit, and a point higher on the graph represents greater estimated treatment-effect power. The set of points displays the available trade-offs among the designs evaluated. If design options vary continuously, a curve may be drawn through them; if only a few policies were evaluated, a scatterplot of the discrete points is more honest than a smooth line that suggests untested designs have been assessed.
One useful comparison is Pareto dominance. Design \(d_1\) dominates design \(d_2\) if \(B_{d_1}\geq B_{d_2}\) and \(P_{d_1}\geq P_{d_2}\), with at least one strict inequality. A dominated design offers no advantage on either plotted criterion. If one design has more benefit but less power, neither dominates the other; the graph alone cannot decide which is preferable.
For a decision with an explicit minimum power requirement, the curve can help identify the designs that meet it and compare their benefits. Alternatively, a decision-maker may specify a minimum benefit and compare power among designs meeting that target. These are transparent decision rules, not statistical conclusions supplied automatically by the plot.
Worked Example: Comparing Designs Against a Power Requirement
Use the three estimated points from the preceding example. Suppose a hypothetical planning committee requires estimated power of at least 0.50 and, among designs meeting that requirement, prefers the one with the greatest estimated participant benefit.
Fixed allocation has power \(0.550\), so it meets the requirement, with benefit \(0.0300\) score units per participant. Moderate adaptation has power \(0.520\), so it also meets the requirement, with benefit \(0.0410\). Stronger adaptation has power \(0.450<0.50\), so it does not meet the requirement. Of the two eligible designs, moderate adaptation has the greater estimated benefit: $$ 0.0410>0.0300. $$ Under this particular decision rule, moderate adaptation is preferred among the evaluated designs. This does not make it universally best: a different power requirement or a different value placed on the two outcomes could change the choice.
Uncertainty and a Short Justification
Each plotted coordinate is estimated, so the apparent separation between points may partly reflect Monte Carlo error. If the simulated trials are independent and identically generated under the stated alternative, each \(R_m\) is a Bernoulli random variable with probability \(P_d\). Therefore \(\widehat{P}_d\) is a sample proportion, with approximate Monte Carlo standard error $$ SE(\widehat{P}_d)\approx\sqrt{\frac{\widehat{P}_d(1-\widehat{P}_d)}{M}}. $$ For the fixed-allocation result above, this is \(\sqrt{0.550(0.450)/2{,}000}=0.0111\). This quantifies simulation error in the estimated power, not uncertainty about whether the assumed alternative is clinically correct.
Similarly, let \(B_{dm}\) be the simulated average incremental outcome per participant in trial \(m\). The estimate \(\widehat{B}_d\) is the average of the \(M\) trial-level values, and its Monte Carlo standard error is the sample standard deviation of those values divided by \(\sqrt{M}\). Using trial-level values preserves the variation between simulated trials. The aggregate totals in the preceding example are not enough to calculate this standard error; the trial-level simulation results are needed.
A useful justification for these calculations is that the simulation repeats the entire trial-generating process, rather than holding one favorable allocation path fixed. For power, each complete repeat contributes a zero or one according to the prespecified rejection rule. Averaging these indicators estimates the probability of rejection under that process. For benefit, summing each participant’s assigned-treatment outcome minus their control potential outcome, then averaging over participants and repeated trials, estimates the expected incremental outcome in the definition of \(B_d\). Both calculations therefore target expectations over allocation and outcome variation, provided the simulation represents those sources of variation correctly.
Worked Example: Checking Whether a Difference Is More Than Simulation Noise
Suppose 2,000 independent simulated trials give estimated powers \(0.550\) and \(0.520\) for two designs. To obtain a rough Monte Carlo uncertainty scale for each estimate, $$ SE_1\approx\sqrt{\frac{0.550(0.450)}{2{,}000}}=0.0111, \qquad SE_2\approx\sqrt{\frac{0.520(0.480)}{2{,}000}}=0.0112. $$ The estimated difference is \(0.550-0.520=0.0300\), or 3.00 percentage points. If the two sets of simulated trials are independent, the approximate standard error of the difference is $$ SE(\widehat{P}_1-\widehat{P}_2) \approx\sqrt{0.0111^2+0.0112^2}=0.0158. $$ The difference is about \(0.0300/0.0158=1.90\) such standard errors. This is a warning against treating the ordering as precise from these simulation runs alone; it is not a formal test of clinical superiority and does not account for uncertainty in the assumed model. More simulation replicates can reduce Monte Carlo error.
Diagnostics, Alternatives, and Limitations
Before interpreting a curve, check that its points are comparable. Confirm that designs use the same trial size, target population, outcome definition and units, treatment-effect alternative, test, and significance level. Check how benefit is benchmarked and whether the outcome direction is consistent. Also verify that each simulation implements the intended allocation policy and analysis; coding errors or incorrect rejection rules can move points in misleading ways.
Look for uncertainty in both coordinates, not just power. If apparent differences are small relative to Monte Carlo standard errors, run more replicates before making a ranking claim. For comparisons based on shared simulated scenarios, account for the resulting dependence when estimating uncertainty in differences. A curve that omits uncertainty bars can make nearly indistinguishable designs look decisively separated.
A benefit-power curve is useful alongside, not instead of, a table of numerical results or separate reporting of benefit and power. A benefit-only comparison can hide an unacceptable loss of power; a power-only comparison can hide meaningful differences in participant outcomes. In contrast, a single curve may conceal important features of the distributions, such as rare harmful outcomes or variation in benefit across participant groups. It also cannot resolve how much benefit is worth a given reduction in power; that requires a stated decision context.
Finally, the curve is conditional on the assumptions used to define the population, potential outcomes, allocation policy, and test. A well-drawn plot does not validate those assumptions. Treat its coordinates as estimates for the stated evaluation scenario, label axes with units and definitions, and avoid presenting a smooth frontier as evidence about designs that were not evaluated.
Common Mistake
Wrong reasoning: “This design is farther right on the graph, so it is better.” Being farther right means higher estimated benefit, not higher performance on every criterion. The design could have substantially lower power, and a small apparent difference may be simulation noise.
Correction: Read both coordinates, check uncertainty and comparability, and state the decision rule used to choose between a gain in benefit and a loss in power. If neither design dominates the other, the graph displays a trade-off; it does not supply the value judgment needed to select one.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A design has estimated benefit \(0.06\) favorable score units per participant and power \(0.48\). What does each coordinate mean?
- Design A has benefit \(0.03\) and power \(0.60\); design B has benefit \(0.04\) and power \(0.62\). Does either dominate the other?
- In 1,000 independent simulated trials, a test rejects \(H_0\) in 700. Calculate the estimated power and its approximate Monte Carlo standard error.
- Why is a benefit-power comparison potentially misleading if one design is simulated in a larger trial?
Solutions. (1) Under the stated scenario, the design yields an estimated average incremental gain of \(0.06\) score units per enrolled participant relative to the specified control benchmark; its estimated probability of rejecting \(H_0\) under the stated alternative is \(0.48\). (2) Neither dominates: B has higher benefit, but A has higher power. (3) \(\widehat{P}=700/1{,}000=0.700\); \(SE\approx\sqrt{0.700(0.300)/1{,}000}=0.0145\). (4) Trial size can affect both expected benefit and power, so the points do not isolate the effect of the allocation design. Use a common size or make the resource difference an explicit part of the comparison.