Comparing Designs at Matched Patient Benefit

Why Match Participant Benefit Before Comparing Inference?

An adaptive trial may assign more participants to a treatment that appears to perform well as data accumulate. That can improve the outcomes experienced during the trial, but it does not by itself show whether the trial estimates the treatment effect more accurately or has greater power. A fair comparison must separate these questions: first, how much expected benefit do participants receive under each design; second, how well does each design support inference?

The comparison in this tutorial holds the participant-benefit target fixed and evaluates treatment-effect bias, standard error, and power. The designs must also use the same eligible population, outcome scale, total sample size, follow-up, treatment-effect estimand, and testing convention. Otherwise, a difference in inferential performance may reflect a change in the question or trial resources rather than adaptation itself.

Definition: Let \(Y(1)\) and \(Y(0)\) denote a participant’s potential outcomes under treatment and control, with larger values defined as better outcomes. The treatment-effect estimand is \(\Delta=E[Y(1)-Y(0)]\) in the target population. A design’s expected participant benefit is its expected average outcome under the assignments it generates, measured on the same outcome scale and for the same target population. A matched-benefit comparison chooses designs whose expected participant benefits are equal, or sufficiently close for the intended comparison.

This definition requires a meaningful common outcome or utility scale. It also assumes that each participant’s observed outcome is the potential outcome under the treatment actually assigned (consistency), that one participant’s assignment does not change another’s outcome (no interference), and that the target population is specified. When outcomes are not naturally ordered so that larger is better, benefit must be defined using a prespecified utility. A comparison that changes the utility after seeing the results is not a matched-benefit comparison.

How to Define a Match

For a fixed design with a proportion \(q\) assigned to treatment, and a constant treatment effect \(\Delta\), expected average outcome is \(E[Y(0)]+q\Delta\). Its expected benefit relative to assigning everyone to control is therefore \(q\Delta\). If an adaptive design has expected treatment fraction \(E[N_1/N]\), then under the same constant-effect conditions its expected benefit relative to all control is \(E[N_1/N]\Delta\). Matching those benefits calls for a fixed design with \(q=E[N_1/N]\), provided \(\Delta\) is positive.

That shortcut is not generally valid when effects or baseline outcomes differ across participants and assignment probabilities depend on their characteristics. In that case, calculate the expected outcome under the complete assignment rule, averaging over the target population and the design’s randomization. Matching only the overall treatment fraction can miss who receives treatment. The following example shows why.

Worked Example: Matching Benefit When Effects Differ by Subgroup

Suppose a hypothetical target population has equal numbers of participants in subgroups H and L. Control outcomes have mean 0 in both groups. Treatment increases the mean outcome by 3 units in H and 1 unit in L. An adaptive design assigns treatment with probability 0.80 in H and 0.40 in L. Find the expected participant benefit relative to assigning everyone to control, and the treatment fraction for a fixed design with the same expected benefit.

In H, the expected benefit per participant is \(0.80(3)=2.4\) units. In L, it is \(0.40(1)=0.4\) units. With equal subgroup sizes, the overall expected benefit is

\frac{1}{2}(2.4)+\frac{1}{2}(0.4)=1.4\text{ units per participant}.

The population-average treatment effect is \(\frac{1}{2}(3)+\frac{1}{2}(1)=2\) units. A fixed design assigning proportion \(q\) to treatment therefore has expected benefit \(2q\) units per participant. Equating this to 1.4 gives \(2q=1.4\), so \(q=0.70\). The matched fixed design assigns 70% to treatment.

The adaptive design’s overall expected treatment fraction is \(\frac{1}{2}(0.80)+\frac{1}{2}(0.40)=0.60\), not 0.70. The two designs nevertheless have equal expected benefit because the adaptive design more often treats the subgroup with the larger effect. This calculation matches benefit; it does not establish that either design estimates the population-average treatment effect better.

Inferential Criteria on a Common Basis

For an estimator \(\hat{\Delta}\) of the same estimand \(\Delta\), its bias is \(E[\hat{\Delta}]-\Delta\), where the expectation is over repeated trials under the specified design and outcome model. Its standard error describes the standard deviation of its sampling distribution; in a design comparison, the repeated-trial standard deviation is a natural reference for whether the reported \(SE\) is calibrated. Bias and standard error answer different questions. An estimator can have a small standard error but be systematically displaced from the estimand.

Power is the probability that the prespecified test rejects \(H_0\) when a particular alternative is true. It depends on the true effect, the test, the significance level \(\alpha\), the estimator, and the variance procedure—not just on the allocation rule. Thus, power comparisons need a specified alternative and should check that Type I error is controlled under \(H_0\). A method with inflated Type I error can appear more powerful simply because it rejects too often.

Key comparison rule: Match designs on expected participant benefit, then evaluate the same treatment-effect estimand using each design’s prespecified estimator and valid variance or testing procedure. Report bias, repeated-trial standard error, and power at stated values of \(\Delta\) and \(\alpha\). A matched-benefit comparison does not imply matched allocations or matched inferential performance.

A useful reference calculation illustrates why allocation itself matters. Suppose a fixed design assigns exactly \(n_1\) participants to treatment and \(n_0\) to control, with independent outcomes having common variance \(\sigma^2\). The difference in sample means has variance

\operatorname{Var}(\bar{Y}_1-\bar{Y}_0) =\frac{\sigma^2}{n_1}+\frac{\sigma^2}{n_0}, \qquad SE=\sigma\sqrt{\frac{1}{n_1}+\frac{1}{n_0}}.

This follows because the two sample means are independent under fixed assignment, and the variance of each mean is \(\sigma^2\) divided by its sample size. It is a benchmark under those assumptions, not a formula that can automatically be applied to an adaptive trial.

Under adaptation, treatment counts can be random, and assignments may depend on earlier outcomes. Even if a fixed-count formula looks reasonable after substituting average counts, it need not account for the full sampling variability or selection effects. The identity

\operatorname{Var}(\hat{\Delta}) =E[\operatorname{Var}(\hat{\Delta}\mid A)] +\operatorname{Var}(E[\hat{\Delta}\mid A])

holds for an assignment history \(A\), but conditioning on an outcome-adaptive history can change the conditional outcome distribution. Both terms, and any dependence induced by the allocation rule, must be handled by the design-appropriate analysis. The earlier material on conditional and unconditional performance distinguishes performance conditional on an assignment history from performance averaged over repeated trials.

Worked Example: A Fixed-Allocation Standard Error Benchmark

Consider a hypothetical fixed trial with 200 participants: 140 assigned to treatment and 60 to control. Assume independent normally distributed outcomes with common known standard deviation \(\sigma=4\) outcome units, fixed counts, and a difference-in-means estimator. Calculate its standard error.

The model assumptions give variance \(4^2=16\) square units in each arm. Substituting the arm sizes into the fixed-allocation formula,

SE=4\sqrt{\frac{1}{140}+\frac{1}{60}} =4\sqrt{0.007143+0.016667} =4\sqrt{0.023810} \approx 0.6172\text{ units}.

This value is the sampling standard deviation under the stated fixed-count model. It does not include uncertainty from estimating \(\sigma\), and it is not an appropriate automatic standard error for an adaptive design whose allocation depends on observed outcomes. It is useful as a benchmark only when those distinctions are respected.

Comparing Designs Using Repeated Trials

Design-level comparisons commonly use repeated-trial simulation. Each replicate follows the same eligibility rules, outcome model, sample size, and treatment-effect estimand. The adaptive allocation rule is run as specified; the fixed rule is chosen to deliver the target expected benefit. For each replicate, record the estimator, its reported \(SE\), and whether the prespecified test rejects. Across replicates, estimate bias from the average estimation error, the sampling standard error from the spread of estimates, and power from the rejection proportion under the stated alternative.

Matching benefit must be checked in the simulation itself. For example, if the adaptive rule’s expected treatment fraction is estimated from simulation, its Monte Carlo uncertainty should be considered when selecting the fixed comparator. If effects vary by subgroup, compare expected outcomes under the full assignment rules rather than matching only average treatment fractions. Simulation estimates also have Monte Carlo error: a reported power of 0.70 from 1,000 replicates is not an exact population value.

Worked Example: Interpreting a Matched-Benefit Simulation

Consider a hypothetical simulation with 10,000 replicates per condition, \(N=200\), normally distributed outcomes with common standard deviation 4 units, and true treatment effect \(\Delta=2\) units. The adaptive rule has expected treatment fraction 0.70, and the fixed comparator assigns 140 of 200 participants to treatment. With control mean 10 units, both designs have expected average participant outcome \(10+0.70(2)=11.4\) units. Suppose the same prespecified estimand is analyzed in both conditions using their design-appropriate inference procedures. The simulated summaries are: fixed-design bias 0.00 units, repeated-trial standard error 0.62 units, and 8,970 rejections; adaptive-design bias 0.10 units, standard error 0.78 units, and 7,310 rejections. Under \(H_0\), the respective rejection proportions are 0.050 and 0.052.

The estimated powers are \(8{,}970/10{,}000=0.897\) for the fixed design and \(7{,}310/10{,}000=0.731\) for the adaptive design. For the adaptive estimate, the Monte Carlo standard error of the power estimate is approximately

\sqrt{\frac{0.731(1-0.731)}{10{,}000}}\approx 0.00443.

Thus, at this specified effect and significance level, the simulation suggests lower power and greater sampling variability for the adaptive design despite equal expected participant benefit. The adaptive estimator’s bias is 0.10 units, meaning its average estimate exceeds the true 2-unit effect by 0.10 units in this simulation. The null rejection proportions are close to 0.05, so these summaries do not suggest a large Type I error inflation; they do not prove exact error control. These hypothetical results are specific to the stated model and procedures, not a general ranking of adaptive and fixed designs.

What to Diagnose—and What the Comparison Cannot Establish

Before interpreting a comparison, check that its conditions support the conclusions:

Benefit match: verify expected average outcome under each full assignment rule, on the same scale and in the same target population.
Estimand match: ensure both analyses target the same treatment effect. Different estimands can yield different answers even with the same data.
Bias and uncertainty: compare repeated-trial bias with the estimand and compare empirical sampling spread with reported \(SE\). A large discrepancy suggests that the variance procedure may be poorly calibrated.
Test calibration: evaluate rejection under \(H_0\) as well as power under alternatives. Power is not meaningful as a standalone advantage if Type I error is not controlled.
Simulation precision: report the number of replicates and consider Monte Carlo error for estimated bias, standard error, and power.

The main alternative comparison is a conventional fixed equal-allocation design. It is informative when it represents the practical policy of interest, but it may not deliver the same expected participant benefit as an adaptive design. A fixed unequal-allocation design matched on benefit is often the more direct comparator for this question. Conversely, a comparison at equal allocation is useful for studying the effect of the allocation rule itself, but should not be described as a matched-benefit comparison unless expected benefits actually agree.

No one criterion settles the design choice. A design can improve expected participant outcomes while increasing uncertainty about the treatment contrast; another can have better power but allocate fewer participants to the currently promising treatment. Matched expected benefit is an accounting device for clarifying that trade-off, not proof of equal experience for every participant or equal ethical acceptability. It also does not establish that the assumed treatment effects, benefit scale, or adaptive model will hold in practice.

Common mistake: “The adaptive trial assigns more participants to the apparently better treatment, so it must be the more efficient design.” This confuses expected participant benefit with inferential precision. The correction is to match expected benefit under explicit assumptions, then compare bias, standard error, and power for the same estimand using valid design-specific inference.

Check Your Understanding

Answer each question using the definitions and comparison principles above. Short solutions follow the questions.

A treatment has a constant effect of 1.5 outcome units. An adaptive design assigns an expected 60% of participants to it. What is the expected benefit relative to assigning everyone to control, and what fixed treatment fraction matches that benefit?
Why can matching the overall expected treatment fraction fail to match participant benefit when treatment effects vary across subgroups?
A study has estimated bias 0.2 units and repeated-trial standard error 0.5 units. State what each quantity describes; do they mean the same thing?
A design has estimated power 0.82 but rejects \(H_0\) in 0.12 of null simulations when \(\alpha=0.05\). Is its apparent power advantage sufficient evidence of a better design?

Solutions. (1) Expected benefit is \(0.60(1.5)=0.90\) units per participant relative to all control. With a constant effect, a fixed design assigning 60% to treatment has the same expected benefit. (2) Subgroups can have different treatment effects, and the adaptive rule can assign different treatment probabilities to each subgroup; the expected benefit depends on both quantities, not just their overall average allocation. (3) Bias is the average estimation error, so estimates average 0.2 units above the estimand; the standard error is the repeated-trial sampling standard deviation, 0.5 units. They describe distinct features of the estimator’s sampling distribution. (4) No. A 0.12 null rejection rate is substantially above the stated 0.05 level, so the apparent power may reflect inflated Type I error. Compare designs only after assessing null calibration.