Estimating Probability of Success for a Confirmatory Trial

Why Probability of Success Is a Predictive Quantity

A confirmatory-trial design can have high power at a chosen treatment effect yet still have a more modest chance of success when current evidence leaves substantial uncertainty about what the true effect is. Power answers a conditional question: if the true effect were a specified value, how often would the trial meet its success criterion? Probability of success (PoS) averages that chance over uncertainty in the effect informed by the evidence available before the trial.

This distinction matters in development decisions. A point estimate from an earlier study is not the true effect, and planning as though it were known can make a design look more reliable than it is. The calculation below combines uncertainty in the treatment effect with the sampling variability of the future trial, then compares the resulting PoS with a go-or-no-go threshold specified before examining the result.

Definition: The probability of success for a planned trial is the probability, conditional on the evidence and assumptions used for planning, that the future trial will meet a prespecified success criterion. When uncertainty about the true treatment effect is represented by a probability distribution, PoS averages the future trial’s success probability over that distribution. It is not the probability that the treatment is effective, nor is it the p-value of the future trial.

A Normal Approximation for a Confirmatory Trial

Consider a continuous endpoint where larger values favor treatment. Let \(\theta\) be the true treatment-minus-control mean difference, measured in endpoint units. Suppose the current evidence represents uncertainty about \(\theta\) as a normal distribution with mean \(\mu_\theta\) and variance \(\tau^2\). For a future trial with \(n\) participants per arm, assume equal allocation and a common, known within-arm standard deviation \(\sigma\). The future estimate \(\hat{\theta}\), conditional on the true effect, is then normally distributed with mean \(\theta\) and standard error \(\sigma\sqrt{2/n}\).

Suppose success means that the future trial rejects a two-sided null hypothesis at the 5% level in the favorable direction. Under the normal approximation with known standard error, this means the estimated effect exceeds the positive critical value \(c_n=1.96\,SE_n\), where \(SE_n=\sigma\sqrt{2/n}\). In practice, a trial may use a different test or success criterion; the boundary must match the actual protocol.

Model and assumptions: For this calculation, \(\theta\sim N(\mu_\theta,\tau^2)\), and \(\hat{\theta}\mid\theta\sim N(\theta,SE_n^2)\), with \(SE_n=\sigma\sqrt{2/n}\). The normal distribution for \(\theta\) summarizes uncertainty from the current evidence; the conditional sampling model describes the future estimate if \(\theta\) were fixed. The calculation assumes those distributions are appropriately calibrated, the standard error represents the planned analysis, and success is exactly the event \(\hat{\theta}>c_n\). It does not automatically account for uncertainty in \(\sigma\), missing data, protocol changes, or model misspecification.

Because the future estimate contains both uncertainty about \(\theta\) and new sampling error, its predictive variance is the sum of those variances. The two components are not interchangeable: increasing the sample size reduces future sampling error, but it does not erase uncertainty in the treatment effect from the evidence available before the trial.

\hat{\theta}\sim N\left(\mu_\theta,\tau^2+SE_n^2\right), \qquad \operatorname{PoS}(n) =P(\hat{\theta}>c_n) =\Phi\left(\frac{\mu_\theta-c_n}{\sqrt{\tau^2+SE_n^2}}\right).

Why the Predictive Variance Adds

The variance result follows from the law of total variance. Conditional on \(\theta\), the future estimate has mean \(\theta\) and variance \(SE_n^2\). Its unconditional mean is therefore \(E(\hat{\theta})=E(\theta)=\mu_\theta\). Its variance is

\begin{aligned} \operatorname{Var}(\hat{\theta}) &=E\{\operatorname{Var}(\hat{\theta}\mid\theta)\} +\operatorname{Var}\{E(\hat{\theta}\mid\theta)\}\\ &=E(SE_n^2)+\operatorname{Var}(\theta)\\ &=SE_n^2+\tau^2. \end{aligned}

Since a normal variable with mean \(\mu_\theta\) and standard deviation \(\sqrt{\tau^2+SE_n^2}\) exceeds \(c_n\) with probability equal to the standard normal cumulative distribution function evaluated at \((\mu_\theta-c_n)/\sqrt{\tau^2+SE_n^2}\), the stated PoS formula follows. This calculation is a predictive probability: it averages over both the uncertain effect and the future trial result.

Worked Example: Calculating PoS for 80 Participants per Arm

Suppose, hypothetically, that current evidence gives \(\theta\sim N(5,4)\) endpoint units squared, so \(\mu_\theta=5\) units and \(\tau=2\) units. Plan a two-arm trial with 80 participants per arm and assume a common within-arm standard deviation of 10 units. Success requires a positive result meeting the two-sided 5% significance criterion described above.

First calculate the standard error of the estimated treatment difference:

SE_{80}=10\sqrt{\frac{2}{80}} =\sqrt{2.5} =1.5811\text{ units}.

The success boundary is \(1.96SE_{80}=1.96(1.5811)=3.0990\) units. The predictive standard deviation includes both the 2-unit uncertainty in \(\theta\) and the future sampling error:

\sqrt{\tau^2+SE_{80}^2} =\sqrt{4+2.5} =\sqrt{6.5} =2.5495\text{ units}.

Thus,

\operatorname{PoS}(80) =\Phi\left(\frac{5.000-3.0990}{2.5495}\right) =\Phi(0.7456) \approx 0.7721.

Under these assumptions, the chance that this trial meets the specified success criterion is about 77.21%. This is not a 77.21% probability that the treatment has a positive effect; it is the predictive probability of passing this particular statistical criterion.

Comparing Sample-Size Scenarios

The formula can be recalculated for each planned sample size. As \(n\) increases, \(SE_n\) and the critical boundary \(c_n\) both decrease. The predictive standard deviation also decreases, but it approaches \(\tau\), not zero: uncertainty in the treatment effect remains even if sampling error becomes very small. Consequently, more participants do not guarantee success, and the gain in PoS need not be large when uncertainty in the effect dominates.

Worked Example: Comparing 80, 100, and 120 per Arm

Keep the hypothetical effect distribution and standard deviation from the preceding example: \(\theta\sim N(5,4)\) and \(\sigma=10\) endpoint units. For 100 participants per arm, \(SE_{100}=10\sqrt{2/100}=1.4142\) units, the success boundary is \(1.96(1.4142)=2.7719\) units, and the predictive standard deviation is \(\sqrt{4+2}=2.4495\) units. Therefore,

\operatorname{PoS}(100) =\Phi\left(\frac{5.000-2.7719}{2.4495}\right) =\Phi(0.9096) \approx 0.8185.

For 120 participants per arm, \(SE_{120}=10\sqrt{2/120}=1.2910\) units. The critical boundary is \(1.96(1.2910)=2.5303\) units, and the predictive standard deviation is \(\sqrt{4+1.6667}=2.3805\) units. Hence,

\operatorname{PoS}(120) =\Phi\left(\frac{5.000-2.5303}{2.3805}\right) =\Phi(1.0375) \approx 0.8502.

The calculated PoS values are approximately 0.7721, 0.8185, and 0.8502 for 80, 100, and 120 per arm, respectively. These are conditional on the same effect distribution, endpoint variability, analysis, and success rule. They are not universal operating characteristics for those sample sizes.

Combining Design Scenarios and Applying a Threshold

A single planned design may not capture uncertainty about the design that will actually be implemented. For example, feasibility or operational constraints may leave several sample sizes under consideration. If scenario \(j\) has probability weight \(w_j\), where weights are nonnegative and sum to one, the overall predictive probability is the weighted average of the scenario-specific probabilities. The weights represent uncertainty about which scenario will apply; they should not be confused with the evidence-based distribution for \(\theta\).

Worked Example: Applying a Prespecified Go Threshold

Suppose a development team prespecifies a go decision when the weighted PoS is at least 0.80. Use the three hypothetical sample-size scenarios above. If the scenarios are assigned weights of 0.25, 0.50, and 0.25 to 80, 100, and 120 participants per arm, respectively, the overall PoS is

\begin{aligned} \operatorname{PoS}_{\text{weighted}} &=0.25(0.7721)+0.50(0.8185)+0.25(0.8502)\\ &=0.193025+0.409250+0.212550\\ &=0.814825\approx 0.8148. \end{aligned}

This exceeds the prespecified 0.80 threshold, so the rule gives a go decision. If instead the scenario weights are 0.50, 0.30, and 0.20, the calculation is

0.50(0.7721)+0.30(0.8185)+0.20(0.8502) =0.38605+0.24555+0.17004 =0.80164\approx 0.8016.

This also exceeds 0.80, but only narrowly. The conclusion is therefore sensitive to inputs and assumptions near the threshold. The threshold is a decision rule, not a statistical theorem: it should be chosen in advance and justified in the development context. A different threshold or different scenario weights could lead to a different decision.

Interpretation, Diagnostics, and Alternatives

The calculation is only as useful as its success rule and uncertainty model. Before using a PoS for a decision, check that the planned endpoint, analysis population, missing-data strategy, test direction, significance level, and success boundary match the intended confirmatory trial. Confirm that the effect distribution reflects the uncertainty in current evidence rather than treating a noisy estimate as known. Also check that the assumed endpoint variability and sample size correspond to the planned design.

A useful diagnostic is to inspect the components separately. If \(\tau\) is large relative to \(SE_n\), uncertainty in the effect dominates and increasing sample size may have limited impact on PoS. If \(SE_n\) is large, sampling variability may be a major contributor. Recalculate PoS under defensible alternative effect distributions, variability assumptions, and scenario weights. A single normal approximation can hide skewness, multiple plausible effect patterns, or substantial uncertainty in nuisance parameters. In such cases, simulation from a richer evidence and trial model may be preferable.

Approach	What it calculates	Key limitation
Plug-in power	Success probability at one fixed treatment effect, often the current point estimate	Does not average over uncertainty in the treatment effect
Conditional power	Success probability conditional on a specified effect or interim result	Its answer depends on the chosen condition; it is not automatically an unconditional predictive probability
Predictive PoS	Success probability averaged over an effect distribution and future sampling variation	Depends on the calibration of the effect distribution, trial model, and success rule
Simulation-based PoS	Estimated success frequency across simulated effects and trial outcomes	Can represent complex designs, but requires validated simulation assumptions and adequate simulation precision

The normal calculation is transparent and efficient when its assumptions are reasonable. For a complex endpoint, adaptive design, non-normal estimator, uncertain dropout, or a success rule involving several criteria, simulate the entire planned analysis and count the fraction of simulated trials meeting that rule. Simulation does not remove assumptions; it makes a more detailed set of assumptions computable.

Common Mistakes and Practical Limits

Calling PoS power. Power conditions on one fixed true effect. PoS integrates over the uncertainty distribution for the effect as well as future sampling variation.
Using the effect’s point estimate as if it were certain. This omits \(\tau^2\) from the predictive variance and can overstate the chance of success.
Changing the success criterion mid-calculation. The critical boundary must match the protocol’s endpoint, test, direction, and significance level. A boundary for a positive two-sided-test result is not the same as a one-sided test boundary.
Treating a threshold crossing as certainty. A PoS just above a prespecified cutoff still carries substantial risk of failure. The threshold determines the action under the rule; it does not guarantee the outcome.
Ignoring model uncertainty. The normal model and known-variability assumption may be inadequate. Alternative assumptions and, where needed, simulation should be used to assess robustness.

Key takeaway: A confirmatory-trial PoS combines uncertainty in the treatment effect with future trial variability under a prespecified success rule. Report the model, assumptions, scenario weights, and threshold alongside the number; the number alone is not a complete development rationale.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In the normal predictive model, which two variance components determine the variance of the future treatment-effect estimate?
If uncertainty about the treatment effect is ignored, which quantity does the calculation resemble, and what uncertainty has been left out?
For a design with scenario-specific PoS values 0.70 and 0.90 and weights 0.40 and 0.60, calculate the weighted PoS.
Why does a PoS of 0.81 not mean there is an 81% probability that the treatment has a positive effect?

Solutions. (1) The variance is \(\tau^2+SE_n^2\): uncertainty in the treatment effect plus future sampling variance. (2) It resembles plug-in power at the fixed effect used, and omits uncertainty in the effect distribution. (3) \(0.40(0.70)+0.60(0.90)=0.28+0.54=0.82\). (4) PoS is the probability of meeting a specified trial success criterion under the predictive model; it is not the probability that the true effect is positive. The latter is a different probability derived from the effect distribution alone.