Assurance Across Clinically Plausible Effect Sizes

Why Examine a Grid of Treatment Effects?

A design’s conditional power is not one universal property of the design. It depends on the treatment effect assumed to generate the data. A trial may have very little chance of meeting its Bayesian success rule if the true benefit is small, but a substantially greater chance if the true benefit is larger. Reporting conditional power across a prespecified grid makes that dependence visible.

The grid should represent effects that matter clinically, not merely convenient numerical values. It can include no benefit, effects below a clinically important threshold, the threshold itself, and larger plausible effects. The resulting values describe how the proposed design performs under those distinct fixed-effect scenarios. They do not assign probabilities to the scenarios and do not replace a prior-averaged assurance calculation.

Definition: For a specified design, analysis model, prior, and Bayesian success rule, the conditional Bayesian power at treatment effect \(\delta\) is the probability, under repeated data generation with the true effect fixed at \(\delta\), that the trial’s posterior-based success rule will be met. A conditional-power grid is a prespecified set of clinically relevant values \(\delta_1,\ldots,\delta_J\), together with the conditional power calculated at each value.

A Normal-Endpoint Model and Success Rule

Consider a continuous outcome for which larger values are better. Let \(\delta\) be the treatment-minus-control mean difference, measured in outcome points, and let \(Y\) be the observed estimate of that difference. For this illustration, assume \(Y\mid\delta\sim N(\delta,r^2)\), where the sampling variance \(r^2\) is known and fixed by the design. Use the analysis prior \(\delta\sim N(0,\tau^2)\), with known \(\tau^2\). Assume observations follow the stated model, the endpoint and treatment contrast are defined consistently, and there are no missing-data or nuisance-parameter complications affecting the calculation.

A one-sided Bayesian success rule might declare success when the posterior probability of benefit exceeds 0.95: \(P(\delta>0\mid Y)>0.95\). The rule is part of the design and must be specified before computing conditional power. Here, “benefit” means a treatment effect greater than zero on the stated outcome scale.

Key result: Under the normal sampling model and normal prior above, the posterior is normal with variance \(v=(1/\tau^2+1/r^2)^{-1}\) and mean \(wY\), where \(w=v/r^2\). For a success rule \(P(\delta>0\mid Y)>q\), where \(q>0.5\), success is equivalent to \(Y>c\), with \(c=z_q\sqrt{v}/w\). Thus, at a fixed true effect \(\delta\), conditional power is \(1-\Phi((c-\delta)/r)\).

Why the Conditional-Power Formula Holds

Given the data estimate \(Y\), normal-normal conjugacy gives posterior precision equal to prior precision plus sampling precision. The posterior mean is the precision-weighted combination of the prior mean, zero, and the estimate. In this zero-centred prior case, these statements give:

v=\left(\frac{1}{\tau^2}+\frac{1}{r^2}\right)^{-1}, \qquad E(\delta\mid Y)=\frac{v}{r^2}Y=wY.

The posterior probability of benefit is \(P(\delta>0\mid Y)=\Phi(wY/\sqrt{v})\). Since \(\Phi\) is strictly increasing, the rule that this probability exceeds \(q\) is equivalent to \(wY/\sqrt{v}>z_q\). With \(w>0\), rearranging gives the data threshold \(Y>z_q\sqrt{v}/w=c\). Under repeated data generation at a fixed true \(\delta\), \(Y\sim N(\delta,r^2)\), so the probability of crossing that threshold is:

P_\delta(Y>c) =1-\Phi\left(\frac{c-\delta}{r}\right) =\Phi\left(\frac{\delta-c}{r}\right).

This derivation separates two probability statements. The posterior probability in the success rule is calculated conditional on the observed data. Conditional power is instead a repeated-sampling probability with the true effect held fixed. The same trial design and success rule define both, but they answer different questions.

Worked Example: Calculating Power Across a Prespecified Grid

Suppose a hypothetical trial measures a continuous outcome in points. Its estimate has known sampling variance \(r^2=2\) points squared. The analysis prior is \(\delta\sim N(0,0.6965)\), where the prior variance is in points squared. Success requires \(P(\delta>0\mid Y)>0.95\). Consider the prespecified grid \(\delta=0,2,4,6\) points.

First calculate the posterior variance and the coefficient multiplying \(Y\) in the posterior mean:

v=\left(\frac{1}{0.6965}+\frac{1}{2}\right)^{-1} =\left(1.4358+0.5000\right)^{-1} \approx 0.5166, \qquad w=\frac{v}{r^2}=\frac{0.5166}{2}\approx 0.2583.

For \(q=0.95\), \(z_q\approx1.6449\). The success rule is therefore equivalent to exceeding this estimate threshold:

c=\frac{1.6449\sqrt{0.5166}}{0.2583} \approx 4.577\text{ points}.

At each grid value, use \(1-\Phi((c-\delta)/\sqrt{2})\). For example, at \(\delta=4\) points, the standardized threshold is \((4.577-4)/1.4142\approx0.408\), giving conditional power \(1-\Phi(0.408)\approx0.342\). The same calculation at the other fixed effects gives:

True effect \(\delta\) (points)	Standardized threshold \((c-\delta)/r\)	Conditional power
0	3.236	0.0006
2	1.822	0.034
4	0.408	0.342
6	-1.006	0.843

For instance, at a true effect of 6 points, the estimated effect exceeds the success threshold with probability \(1-\Phi(-1.006)\approx0.843\). Thus this design is much more likely to meet its rule at 6 points than at 4 points. These are hypothetical operating characteristics under the stated model; they are not posterior probabilities that any particular effect is true.

Reading the Grid as a Design Assessment

A grid is most useful when its values are selected to answer a clinical question. If 4 points is considered the minimum worthwhile benefit, include effects below 4, at 4, and above 4. The table then shows whether the design is likely to meet its success rule when the true effect is clinically unimportant, just worthwhile, or more favorable. State the units and the clinical meaning of the grid values alongside the computed probabilities.

For the model above, conditional power increases with \(\delta\). Differentiating \(\Phi((\delta-c)/r)\) with respect to \(\delta\) gives \(\phi((\delta-c)/r)/r\), which is positive because the normal density \(\phi\) and \(r\) are positive. This monotonicity is a useful check for this particular model and rule: a computed grid that decreases as the true effect rises suggests an arithmetic, coding, or interpretation error. It is not a license to assume monotonicity for every possible model and decision rule.

The grid can also reveal where a design’s performance changes from unlikely to likely under an explicitly chosen benchmark. Such a benchmark is a communication aid, not a universal definition of success. The clinical importance of effects and the consequences of a failed trial still matter. A single high conditional-power value at a very large effect cannot establish that the design performs well across effects that clinicians consider plausible.

Worked Example: Locating a Chosen Power Benchmark

Use the hypothetical design and success rule in the previous example. Suppose the planning team wants to know which true effects give at least 80% conditional power. This 80% benchmark is a chosen assessment threshold, not part of the Bayesian success rule itself.

Starting from \(1-\Phi((c-\delta)/r)\geq0.80\), rearrange to obtain \((c-\delta)/r\leq z_{0.20}\), where \(z_{0.20}\approx-0.8416\). Therefore:

\delta\geq c+0.8416r =4.577+0.8416\sqrt{2} \approx 5.767\text{ points}.

So, under this model and rule, effects of about 5.767 points or larger have at least 80% conditional power. The grid value of 5 points would not reach the benchmark: its power is \(1-\Phi((4.577-5)/1.4142)=\Phi(0.299)\approx0.618\). At 6 points, power is about 0.843, as calculated earlier. The conclusion is specific: the design is likely to meet the Bayesian rule at 6 points by this chosen benchmark, but not at 5 points.

Comparison, Diagnostics, and Limitations

Conditional power at fixed effects and assurance answer related but distinct planning questions. A grid reports success probabilities separately for specified effects. Assurance averages conditional success probabilities over a design distribution for the treatment effect. That average depends on the design distribution as well as the operating characteristics. A grid is preferable when stakeholders want to inspect particular clinical scenarios; an average alone can conceal whether its value is driven by a narrow range of effects.

The analysis prior still matters even though the grid fixes the true effect in each calculation. It affects the posterior success rule and therefore the estimate threshold \(c\). Do not confuse the analysis prior, which informs the posterior used to decide success, with a distribution over true effects used to average operating characteristics. The grid values themselves have no implied probability weights unless weights are separately specified for an averaging calculation.

Before relying on the results, check that the effect direction and units match the success rule, that the sampling variance matches the proposed design, and that the grid covers the clinically relevant range. Recalculate at least one row directly from the normal tail probability. In simulation implementations, compare simulated success frequencies at selected grid points with the analytic calculation when the normal model applies. Unexpected nonmonotonicity in this setting, incorrect handling of the one-sided threshold, or a mismatch between effect units and standard error are warning signs.

The calculation is conditional on the stated sampling variance and model. It does not account for uncertainty in nuisance parameters, and it assumes the normal sampling distribution and prior are suitable. The grid is also only as informative as its chosen values: a coarse grid can obscure a steep change in performance, while an implausible range can distract from the clinical decision. These limitations should be stated rather than hidden by a single summary number.

Common Mistakes

Calling conditional power the probability that the effect is true. It is a repeated-data probability of satisfying a rule with the true effect fixed; it is not a posterior probability about the effect.
Treating a grid as an assurance calculation. Each row is conditional on one effect. Averaging across rows requires explicit weights or a design distribution.
Using a clinical threshold as if it were the success rule. A minimum worthwhile effect helps choose and interpret grid values; it does not automatically replace the prespecified posterior decision criterion.
Reporting a probability without its scenario or units. “Power is 0.34” is incomplete. State that it is approximately 0.342 at a true treatment benefit of 4 outcome points under the specified design and rule.
Mixing up variance and standard error. In the formula, \(r\) is the standard deviation of \(Y\), while \(r^2\) is its variance. Entering one where the other belongs changes both the posterior and the power.

Key takeaway: A clinically chosen grid makes explicit how conditional Bayesian power changes with the fixed treatment effect. Interpret every value under the stated model and success rule, and do not mistake the grid for a probability distribution over effects.

Check Your Understanding

Answer each question using the definitions and calculations above.

In the normal model, what does conditional power at \(\delta=3\) points mean?
For the worked design, why is the threshold calculated using \(w=v/r^2\), rather than \(w=v/\tau^2\)?
At \(\delta=5\) points in the worked design, calculate the conditional power to three significant figures.
Does a grid with power values at 2, 4, and 6 points assign probabilities to those effects? Explain.
Under the worked model, should conditional power rise or fall as the true effect increases, and what calculation justifies the answer?

Solutions. (1) It is the repeated-trial probability of satisfying the specified posterior success rule when the true treatment benefit is fixed at 3 points. (2) The posterior mean is \(vY/r^2\), so its coefficient on the estimate is \(w=v/r^2\); \(\tau^2\) is the prior variance, not the sampling variance. (3) The standardized threshold is \((4.577-5)/\sqrt{2}\approx-0.299\), so power is \(1-\Phi(-0.299)=\Phi(0.299)\approx0.618\). (4) No. Each value is conditional on its stated effect; weights are needed to average over effects. (5) It should rise: the derivative of \(\Phi((\delta-c)/r)\) is \(\phi((\delta-c)/r)/r>0\).