Why Power Must Account for Selection
In a seamless phase 2/3 trial, the treatment tested in phase 3 may depend on the phase 2 results. Overall power therefore depends on two events: which transition outcome occurs, and whether the final test rejects its null hypothesis after that outcome. The power for one fixed treatment is not, by itself, the design's overall power when another treatment could be selected instead.
The calculation is a weighted sum. Each possible selection outcome contributes its probability multiplied by the probability of a phase 3 rejection conditional on that outcome. This is straightforward when phase 3 data are independent of the data used for selection. When data are reused in the final statistic, however, the conditional rejection probability must reflect the selection event; using an unconditional phase 3 power can give the wrong answer.
This identity is the law of total probability, not an approximation. Its usefulness depends on defining \(J\) to include every branch that can occur and calculating each conditional probability under the same assumed data-generating model. In a design with multiple confirmatory claims, \(R\) must also be specified: it could mean rejection of a particular treatment's null, or at least one rejection under the trial's prespecified decision rule. Those are different power targets.
Independent Phase 3 Data: A Direct Calculation
Suppose phase 2 selects one treatment, and new phase 3 participants provide data independent of all phase 2 data. If the final test uses only those new data and the phase 3 design for treatment \(j\) is fixed once \(j\) is selected, then the phase 3 rejection probability conditional on selecting \(j\) is the power of that branch's test. The selection probability still matters because different treatments can have different true effects and different branch-specific powers.
For example, if selection is based on two independent normally distributed phase 2 statistics, their means and variances determine the probability each is larger. The branch-specific phase 3 rejection probability can then be calculated from the phase 3 test's sampling distribution. Independence here concerns the phase 3 data and selection data; it does not mean that treatment outcomes within a stage are necessarily independent. Shared controls, for instance, can make selection statistics correlated, so their joint distribution must be used to obtain selection probabilities.
Worked Example: Two Treatments and Independent Phase 3 Tests
Consider a hypothetical design with treatments A and B. Phase 2 selects whichever has the larger statistic. Under a specified alternative, suppose the independent phase 2 statistics satisfy \(Z_{2A}\sim N(1.20,1)\) and \(Z_{2B}\sim N(0.70,1)\), independently. In phase 3, the selected treatment is tested using new, independent data with a one-sided critical value of 1.96. Suppose the resulting phase 3 statistics have distributions \(Z_{3A}\sim N(2.50,1)\) and \(Z_{3B}\sim N(1.80,1)\).
The difference \(Z_{2A}-Z_{2B}\) is normal with mean \(1.20-0.70=0.50\) and variance \(1+1=2\). Thus $$ P(J=A)=P(Z_{2A}-Z_{2B}>0) =\Phi\left(\frac{0.50}{\sqrt{2}}\right) =\Phi(0.3536)\approx 0.6382. $$ The two outcomes are exhaustive, so \(P(J=B)=1-0.6382=0.3618\).
For treatment A, the phase 3 rejection probability is $$ P(Z_{3A}>1.96)=1-\Phi(1.96-2.50)=\Phi(0.54)\approx 0.7054. $$ For treatment B, it is $$ P(Z_{3B}>1.96)=1-\Phi(1.96-1.80)=\Phi(-0.16)\approx 0.4364. $$ These probabilities are also the conditional rejection probabilities given their respective selections because the phase 3 data are independent of phase 2.
Therefore the overall power is $$ P(R)=(0.6382)(0.7054)+(0.3618)(0.4364) \approx 0.4502+0.1579=0.6081. $$ Under this specified alternative and design, the probability of rejecting the selected treatment's phase 3 null is about 60.81%. This is not the power for either treatment considered as if it were always selected.
Futility and More Than Two Outcomes
A futility outcome belongs in the sum even though it contributes no rejection if the trial stops. Omitting it from the outcome set, or renormalizing the probabilities of the treatments that continue, changes the target: it calculates power conditional on continuation rather than overall power from the start of the design. The same principle applies when the rule can select among several doses, populations, or endpoints. Each distinct branch needs its own probability and conditional rejection probability.
Worked Example: Three Transition Outcomes Including Futility
In a hypothetical three-arm design, a prespecified phase 2 rule selects A with probability 0.46, selects B with probability 0.34, or stops for futility with probability 0.20 under a particular alternative. Suppose the phase 3 rejection probabilities conditional on selection are 0.78 for A and 0.52 for B. These branch-specific probabilities account for each selected treatment's phase 3 sample size and test.
First check that the transition probabilities cover all outcomes: $$ 0.46+0.34+0.20=1.00. $$ Futility prevents a phase 3 rejection, so its conditional rejection probability is zero. The overall power is $$ P(R)=(0.46)(0.78)+(0.34)(0.52)+(0.20)(0) =0.3588+0.1768=0.5356. $$ Thus the design has 53.56% overall power under the stated alternative. Among trials that continue, the conditional power would instead be \((0.3588+0.1768)/(0.46+0.34)=0.6695\), approximately 66.95%. That answers a different question because it excludes the probability of stopping for futility.
When Selection Data Are Reused
Some seamless designs use phase 2 observations both to select a treatment and in the final test. Then selection and rejection are generally dependent. The weighting identity remains exact, but \(P(R\mid J=j)\) must be calculated from the joint distribution of the selection data and the final statistic, restricted to the event \(J=j\). It is not generally the rejection probability calculated without conditioning on selection.
One general expression makes the dependence explicit. Let \(Y\) denote all data used to determine selection and to form the final test statistic, and let \(A_j\) be the region of possible data values that leads to outcome \(j\). If \(r_j(Y)\) is 1 when the final rule rejects on branch \(j\) and 0 otherwise, then $$ P(R,J=j)=\int_{A_j}r_j(y)f(y)\,dy, \qquad P(R\mid J=j)=\frac{P(R,J=j)}{P(J=j)} $$ when \(P(J=j)>0\). Here \(f\) is the joint density under the specified treatment effects. This integral can be evaluated analytically in simple models or numerically for more complex rules. If branch-specific conditional probabilities are obtained by numerical integration, each must use the actual selection region and the full dependence structure.
The following example uses branch probabilities and conditional rejection probabilities obtained from a hypothetical joint model calculation. The purpose is to show how conditional quantities enter the overall calculation, not to prescribe values for another design.
Worked Example: Reusing Phase 2 Data in the Final Test
Suppose a hypothetical joint model gives probabilities 0.52 for selecting A, 0.33 for selecting B, and 0.15 for stopping. A joint calculation that accounts for selection and reuse of phase 2 data gives \(P(R\mid J=A)=0.74\) and \(P(R\mid J=B)=0.61\). Futility has no rejection, so \(P(R\mid J=\text{stop})=0\).
The branch probabilities sum to one: \(0.52+0.33+0.15=1.00\). Weighting the conditional rejection probabilities gives $$ P(R)=(0.52)(0.74)+(0.33)(0.61)+(0.15)(0) =0.3848+0.2013=0.5861. $$ The overall power is 58.61% under this joint model and alternative.
For contrast, suppose the unconditional rejection probabilities for tests of A and B, ignoring how selection affects the reused data, were 0.69 and 0.48. Substituting those values would give \((0.52)(0.69)+(0.33)(0.48)=0.5172\), not 0.5861. The discrepancy illustrates why marginal powers cannot simply replace conditional powers when the same data influence both selection and rejection. The correct inputs are the branch-conditional probabilities from the joint model.
Calculation Workflow and Diagnostics
A reliable power calculation begins with the decision rule, not with a single test formula. Specify the true effects, sampling distributions, transition rule, phase 3 allocation and final rejection rule. Then enumerate all outcomes and obtain their probabilities and conditional rejection probabilities. Depending on the design, these quantities can come from exact probability calculations, integration of a joint distribution, or a validated numerical evaluation of the specified model.
State which rejection event counts as success and specify the treatment effects under which power is being evaluated.
Include every treatment choice and every stopping outcome allowed by the rule. Check that the outcomes are mutually exclusive and their probabilities sum to one.
Use the joint distribution of the phase 2 outcomes, including correlations from shared controls or other shared data.
Use the test and sample size for that branch. If data are reused, condition on the selection event in the joint model.
Sum \(P(J=j)P(R\mid J=j)\). The result must lie between zero and one; check branch probabilities, units and the final decision rule if it does not.
Useful diagnostics include verifying that branch probabilities sum to one, checking that a stopping branch contributes zero when it cannot reject, and confirming that each conditional rejection probability uses the correct branch-specific test. Extreme cases also provide checks: if one selection outcome has probability one, overall power should equal that branch's rejection probability; if all branches stop, overall power should be zero. When phase 3 data are independent and the test uses only those data, conditioning on selection should not change the branch-specific phase 3 distribution. When that independence condition does not hold, the calculation should reflect the dependence rather than assume it away.
Common Pitfall and Limits
Wrong reasoning: “The selected treatment has a phase 3 power of 70%, so the whole design has 70% power.” This ignores the chance of selecting a different treatment, stopping for futility, and any difference in power across branches. Another version multiplies selection probabilities by unconditional rejection probabilities even though the selection data are reused.
Correction: Define the overall rejection event, enumerate the mutually exclusive transition outcomes, and sum their probabilities times the appropriate conditional rejection probabilities. If the final statistic reuses phase 2 evidence, calculate those conditional probabilities under the joint distribution of selection and final testing.
The calculation is only as credible as its assumptions. Misspecified treatment effects, variances, correlations, missing-data behavior or transition rules can change both selection probabilities and conditional rejection probabilities. A single power value is therefore tied to a specified alternative and model; it does not summarize performance across every plausible treatment-effect configuration. Comparison with a design using a fixed treatment can be informative, but the comparison must use the same target rejection event and clearly stated assumptions.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A design selects A with probability 0.30, B with probability 0.50, and stops with probability 0.20. Conditional rejection probabilities are 0.80 for A and 0.40 for B. What is overall power?
- Why is power conditional on continuing not the same as overall power when a futility stop is possible?
- When may a branch-specific phase 3 power be used as \(P(R\mid J=j)\) without adjusting for selection?
- What must be changed in the calculation when phase 2 data are reused in the final statistic?
Solutions. (1) \(0.30(0.80)+0.50(0.40)+0.20(0)=0.24+0.20=0.44\), so overall power is 44%. (2) Conditional power among continuers excludes the probability of stopping; overall power includes that branch with zero rejection probability. (3) This is appropriate when the phase 3 data used for the test are independent of the selection data and the branch's test and allocation are fixed after selection. (4) Calculate the rejection probability conditional on the selection outcome using the joint distribution of selection data and the final statistic; an unconditional branch power may not be valid.