Design-Effect Distributions for Assurance

Why the Design Distribution Matters

A clinical trial’s assurance depends not only on its decision rule and sampling variability, but also on which treatment effects are considered plausible before the trial. A design distribution represents that uncertainty for planning. Different distributions can assign different probabilities to effects near a success boundary or in the tails, and those differences can change assurance even when the distributions have the same mean and variance.

The design distribution is not the analysis prior used to update beliefs after observing trial data. The analysis prior and the success rule determine whether a particular data set meets the prespecified Bayesian criterion. The design distribution instead describes plausible values of the true effect for evaluating the trial’s operating characteristics. This distinction, developed in the earlier material on analysis priors and design priors in assurance, is essential: assurance averages the chance of success over the design distribution.

Definition: Let $\delta$ be the true treatment effect, and let $G$ be a design distribution for $\delta$. Let $q(\delta)$ be the probability, under the trial’s data-generating model when the true effect is $\delta$, that the prespecified success criterion is met. The assurance is $$ A=\int q(\delta)\,dG(\delta)=E_G\{q(\delta)\}. $$ For a discrete design distribution with possible effects $\delta_1,\ldots,\delta_K$ and probabilities $w_1,\ldots,w_K$, where $w_k\geq0$ and $\sum_{k=1}^K w_k=1$, this becomes $A=\sum_{k=1}^K w_kq(\delta_k)$.

The conditional probability $q(\delta)$ is sometimes called the operating characteristic for success at effect $\delta$. Assurance is its average under $G$. It is a planning probability that accounts for uncertainty about the true effect as well as sampling variation. It is not the posterior probability that the treatment is effective, and it is not generally a frequentist power calculation at one fixed effect.

A Working Success-Probability Model

To make the comparisons concrete, suppose the trial produces an estimate $Y$ satisfying $Y\mid\delta\sim N(\delta,1)$, where effects and standard errors are measured in the same clinical units. Assume the prespecified Bayesian analysis rule declares success when $Y>2$. For this illustration, the analysis prior and posterior criterion are treated as already incorporated into that rule; the threshold is illustrative, not a universal Bayesian cutoff.

Under these assumptions, the conditional success probability is the probability that a normal estimate with mean $\delta$ and standard deviation 1 exceeds 2:

q(\delta)=P(Y>2\mid\delta) =1-\Phi(2-\delta) =\Phi(\delta-2).

Here $\Phi$ is the standard normal cumulative distribution function. The model assumes the stated normal sampling distribution, a known standard error of 1, and a fixed success boundary. In a real design, $q(\delta)$ should come from the actual trial data model and decision rule; it may be evaluated analytically or by simulation. The design distribution is then used to average that function.

Worked Example: Three Clinically Interpretable Scenarios

Suppose clinicians and investigators judge three effects to be plausible: no benefit, a modest benefit of 2 units, and a larger benefit of 4 units. For planning, assign probabilities $0.20$, $0.50$, and $0.30$, respectively. These are hypothetical effects and weights, stated in the clinical outcome’s units. Use the success-probability model $q(\delta)=\Phi(\delta-2)$.

First check the weights: $0.20+0.50+0.30=1$, so they define a probability distribution. The conditional success probabilities are

q(0)=\Phi(-2)=0.0228,\qquad q(2)=\Phi(0)=0.5000,\qquad q(4)=\Phi(2)=0.9772.

Weight each probability by the design probability of its scenario:

$$ A=0.20(0.0228)+0.50(0.5000)+0.30(0.9772) =0.00456+0.2500+0.29316 =0.54772. $$

Thus the estimated assurance under this design distribution is about $0.548$, or 54.8%. The distribution’s mean effect is $0.20(0)+0.50(2)+0.30(4)=2.20$ units. The assurance is not $q(2.20)$: averaging a nonlinear function is generally different from evaluating that function at the average effect.

Continuous Specifications and Their Calculations

A discrete distribution is useful when a small number of scenarios have clear clinical interpretations. A continuous design distribution can be more appropriate when uncertainty is represented as a range of effects rather than a short scenario list. Two common choices are a uniform distribution over a plausible interval and a normal distribution centered on a planning value. Neither choice is automatically justified by convenience: their support and tail behavior must make clinical sense.

Worked Example: Uniform Effects Over a Plausible Range

Suppose the true effect is modeled as uniformly distributed from $3-\sqrt{3}$ to $3+\sqrt{3}$ units. This distribution has mean 3 and variance 1, because a uniform distribution on $[a,b]$ has variance $(b-a)^2/12$. With the same conditional success function $q(\delta)=\Phi(\delta-2)$, assurance is the average of that function over the interval:

A=\frac{1}{2\sqrt{3}}\int_{3-\sqrt{3}}^{3+\sqrt{3}}\Phi(\delta-2)\,d\delta.

Set $z=\delta-2$. The integration endpoints become $1-\sqrt{3}\approx-0.7321$ and $1+\sqrt{3}\approx2.7321$. An antiderivative of $\Phi(z)$ is $F(z)=z\Phi(z)+\phi(z)$, where $\phi$ is the standard normal density; differentiating gives $F'(z)=\Phi(z)+z\phi(z)-z\phi(z)=\Phi(z)$. Therefore,

A=\frac{F(1+\sqrt{3})-F(1-\sqrt{3})}{2\sqrt{3}} \approx\frac{2.7330-0.1353}{3.4641} \approx0.7499.

The assurance is approximately 75.0%. The calculation uses the uniform density $1/(2\sqrt{3})$ and integrates over the entire assumed support. Using only the midpoint effect would give $q(3)=\Phi(1)\approx0.8413$, which is not the assurance because effects below 3 have lower conditional success probabilities.

Worked Example: Normal Effects and a Matched-Moment Comparison

Now let the design distribution be $\delta\sim N(3,1)$, so the mean effect is 3 units and the standard deviation is 1 unit. Keep $Y\mid\delta\sim N(\delta,1)$ and the success rule $Y>2$. We can evaluate assurance without numerical integration. Write $Y=\delta+\varepsilon$, where $\varepsilon\sim N(0,1)$ is independent of $\delta$. Then $Y\sim N(3,2)$, since its mean is 3 and its variance is $1+1=2$. Hence

A=P(Y>2)=1-\Phi\left(\frac{2-3}{\sqrt{2}}\right) =\Phi\left(\frac{1}{\sqrt{2}}\right) \approx\Phi(0.7071) \approx0.7602.

Compare this result with a discrete distribution that assigns probabilities $0.25,0.50,0.25$ to $3-\sqrt{2},3,3+\sqrt{2}$. Its mean is 3 and its variance is $0.25(2)+0.25(2)=1$; the center contributes zero to the variance. Its assurance is

A=0.25\Phi(1-\sqrt{2})+0.50\Phi(1)+0.25\Phi(1+\sqrt{2}) \approx0.25(0.3396)+0.50(0.8413)+0.25(0.9921) \approx0.7536.

The uniform distribution in the previous example also has mean 3 and variance 1. The three assurances are therefore approximately $0.7499$ for the uniform specification, $0.7536$ for the discrete specification, and $0.7602$ for the normal specification. These are distributions with exactly matched means and variances, but not identical shapes. Their assurance values differ because averaging the nonlinear function $q(\delta)$ depends on the distribution’s full shape, not just its first two moments.

Choosing and Checking the Distribution

A design distribution should be built from clinically meaningful uncertainty, not selected solely because it leads to a convenient calculation. For discrete scenarios, state each effect and its probability, and explain how the probabilities reflect the planning assumptions. For a continuous distribution, justify its location, spread, and support. The design distribution may be informed by external evidence, elicitation, or a structured range of plausible outcomes; it is a planning assumption and should be examined as such.

Several checks help reveal when a specification is not doing the intended job:

Check the weights and support. Discrete weights must be nonnegative and sum to 1. Continuous bounds and tail probabilities should include plausible effects without assigning substantial mass to clinically impossible values.
Check units and direction. Effects, the success boundary, and the sampling standard error must use the same scale and sign convention. If a larger value indicates worse outcome, the success rule and $q(\delta)$ must reflect that direction.
Inspect $q(\delta)$. Under the illustrative model, $q(\delta)=\Phi(\delta-2)$ increases with $\delta$. If a computed operating characteristic does not increase under a rule where larger effects should help, check the event direction, model, and implementation.
Check integration or simulation accuracy. For continuous specifications, verify numerical integration over the full distribution. For simulation, report Monte Carlo uncertainty separately from uncertainty about the assumed design distribution.
Vary the shape assumption. Recalculate assurance under defensible alternatives, such as discrete scenarios versus a continuous distribution with similar moments. A material change signals sensitivity to the design distribution.

These checks diagnose different problems. A calculation can be numerically exact but based on implausible tails; conversely, a clinically reasonable distribution can be integrated inaccurately. Assurance is conditional on both the trial model and the specified $G$, so the assumptions should be reported alongside the result.

Comparison, Limitations, and a Common Pitfall

Specification	Useful when	Key consideration
Discrete scenarios	There are a few interpretable clinical cases with elicitable weights.	Results can be sensitive to which scenarios are included and how weights are assigned.
Uniform distribution	A bounded interval is meaningful and no value within it is deliberately favored.	Equal density across the interval is an assumption, not a neutral default.
Normal distribution	A central effect and spread provide a reasonable continuous approximation.	Its unbounded tails may assign probability to implausible effects.

A common mistake is to match the mean and variance of two design distributions and then assume their assurances must match. That reasoning would be valid only for certain special forms of $q$, such as a function whose expectation is determined by those moments. Here $q(\delta)=\Phi(\delta-2)$ is nonlinear, and its average depends on more than the mean and variance. The matched-moment comparison above shows this directly.

Another limitation is that a single design distribution can conceal disagreement about plausible effects. A reported assurance should therefore be read as conditional on the stated design assumptions, not as a guarantee that the trial will succeed or as a probability that the treatment truly works. When uncertainty about the distribution itself is important, a useful response is to present assurance across several defensible specifications rather than to imply that one chosen distribution is uniquely correct.

Key takeaway: Assurance averages the trial’s conditional success probability over a design distribution for the true effect. Clinically justified support, weights, and tail behavior matter; matching moments alone does not ensure matching assurance.

Check Your Understanding

Try each question, then compare your reasoning with the solutions.

For a discrete design distribution with effects 1 and 3 units, each assigned probability 0.5, and $q(1)=0.20$, $q(3)=0.80$, calculate assurance.
Why is assurance not generally equal to $q(E_G[\delta])$?
Under the illustrative model, what is $q(2)$, and what does that value mean?
What is one important limitation of using a normal design distribution for a clinically bounded effect?

Solutions. (1) $A=0.5(0.20)+0.5(0.80)=0.50$. (2) In general, $q$ is nonlinear, so the average of $q(\delta)$ over $G$ need not equal $q$ evaluated at the mean effect; the scenario example gave assurance $0.5477$, not $q(2.20)$. (3) $q(2)=\Phi(0)=0.5000$; at a true effect of 2 units, repeated data sets have a 50.0% probability of meeting the illustrative success rule. (4) A normal distribution has unbounded support and may assign nonzero probability to clinically impossible effects, so its tails and resulting assurance need scrutiny.