Sensitivity of Assurance to Prior Spread

Why Prior Spread Matters

Assurance averages a trial’s probability of meeting its success rule over uncertainty about the true treatment effect. That averaging distribution is the design prior. Its spread determines how much weight the calculation gives to effects far from its center, including effects that may be too small—or even in the opposite direction—to support success. A change in assurance after widening this prior is therefore a change in the assumed distribution of possible true effects, not a change in the trial data or decision rule.

A separate distribution, the analysis prior, is used to calculate the posterior after data are observed. If the analysis prior changes, the posterior and potentially the success rule’s threshold change. A useful sensitivity analysis varies one prior at a time: first recalculate assurance under alternative design-prior spreads while holding the analysis prior fixed, then, if needed, examine analysis-prior changes separately.

Definition: Let $\delta$ be the true treatment effect, and let $S$ denote the event that a trial meets its prespecified Bayesian success rule. For a design prior $\pi_D(\delta)$, assurance is $$ \mathcal{A}=\int P(S\mid \delta)\,\pi_D(\delta)\,d\delta. $$ The probability $P(S\mid\delta)$ is the conditional probability of trial success if the true effect is $\delta$. The design prior averages that operating probability over possible true effects. The analysis prior is the prior distribution used to obtain the posterior distribution that defines $S$; it need not equal the design prior.

A sensitivity analysis of prior spread changes the width of $\pi_D$, usually while keeping its center and the trial design fixed. The calculation must still integrate the operating probability over the entire revised distribution. It is not enough to report the operating probability at the prior mean: that would be a point-effect calculation, not assurance.

A Normal-Outcome Model and Success Rule

For a transparent calculation, suppose the endpoint is continuous and the treatment effect $\delta$ is the treatment-minus-control difference in population means, measured in endpoint units. Assume independent normal outcomes in the two arms, a known common standard deviation $\sigma$, equal allocation with $n$ participants per arm, and no missing outcomes. The observed difference in sample means, written $Y$, then has sampling distribution

Y\mid\delta\sim N(\delta,s^2), \qquad s^2=\frac{2\sigma^2}{n}.

For the analysis, suppose the prior is $\delta\sim N(0,\tau_A^2)$, where $\tau_A$ is the analysis-prior standard deviation. Define success as a posterior probability greater than $q$ that the treatment effect is positive: $P(\delta>0\mid Y)>q$, with $q=0.975$. This rule, the likelihood, and the analysis prior together determine the observed-data threshold for success.

The normal likelihood and normal prior give posterior variance $V=(s^{-2}+\tau_A^{-2})^{-1}$ and posterior mean $VY/s^2$. Because the posterior is normal, $P(\delta>0\mid Y)=\Phi((VY/s^2)/\sqrt V)$, where $\Phi$ is the standard normal cumulative distribution function. Writing $z_q=\Phi^{-1}(q)$, the success rule is equivalent to $Y>c$, where

c=\frac{z_qs^2}{\sqrt V}.

Thus, holding the analysis prior fixed holds $c$ fixed. Given a true effect $\delta$, the probability of success is $P(Y>c\mid\delta)=\Phi((\delta-c)/s)$. If the design prior is $N(m_D,\tau_D^2)$, averaging over it makes $Y$ marginally normal with mean $m_D$ and variance $s^2+\tau_D^2$. The resulting exact assurance is

\mathcal{A}(\tau_D) = \Phi\!\left( \frac{m_D-c}{\sqrt{s^2+\tau_D^2}} \right).

This expression isolates the effect of design-prior spread: $m_D$, the analysis rule and prior, and the trial’s sampling variance stay fixed while $\tau_D$ changes. It also shows why the direction of change is not universal. If $m_D>c$, increasing $\tau_D$ lowers assurance in this model; if $m_D<c$, it raises assurance; if $m_D=c$, assurance remains $0.5$. A larger assurance in the second case reflects more averaging over a broad distribution, not a more favorable clinical effect.

Worked Example: Recalculating Assurance at Three Design-Prior Widths

Suppose the endpoint standard deviation is $\sigma=10$ units and there are $n=100$ participants per arm. Use the analysis prior $N(0,5^2)$ and the success rule $P(\delta>0\mid Y)>0.975$. Compare design priors $N(4,\tau_D^2)$ with standard deviations 1, 2, and 3 units. All quantities describing effects are in endpoint units.

First verify the sampling variance and decision threshold. The standard error is $s=\sqrt{2(10^2)/100}=\sqrt{2}=1.414$ units, so $s^2=2$. The posterior variance is $V=(1/2+1/25)^{-1}=1/0.54=1.852$ square units, and $\sqrt V=1.361$ units. With $z_{0.975}=1.960$, the threshold is $c=1.960(2)/1.361=2.880$ units.

For $\tau_D=1$, the total marginal standard deviation of $Y$ is $\sqrt{2+1^2}=\sqrt3=1.732$. Therefore $\mathcal{A}=\Phi((4-2.880)/1.732)=\Phi(0.646)=0.741$, approximately.

For $\tau_D=2$, it is $\sqrt{2+2^2}=\sqrt6=2.449$, giving $\mathcal{A}=\Phi(1.120/2.449)=\Phi(0.457)=0.676$. For $\tau_D=3$, it is $\sqrt{2+3^2}=\sqrt{11}=3.317$, giving $\mathcal{A}=\Phi(1.120/3.317)=\Phi(0.338)=0.632$.

The design-prior mean and analysis prior did not change, but assurance fell from about 0.741 to 0.632 as the design prior widened. Here the mean effect is above the success threshold. Greater spread places more design-prior weight on smaller effects, for which the trial is less likely to meet the rule.

Justifying the Direction of Sensitivity

The normal-model formula also provides a short justification for the direction of this effect. Define $a=(m_D-c)/\sqrt{s^2+\tau_D^2}$, so that $\mathcal{A}=\Phi(a)$. Differentiating with respect to the design-prior standard deviation gives

\frac{d\mathcal{A}}{d\tau_D} = \phi(a)\, \frac{-(m_D-c)\tau_D}{(s^2+\tau_D^2)^{3/2}},

where $\phi$ is the standard normal density and is positive for every finite $a$. For $\tau_D>0$, the derivative’s sign is therefore the opposite of the sign of $m_D-c$. This proves the direction statements for this model and rule. It does not establish a universal direction for assurance under other likelihoods, priors, or decision rules.

Worked Example: A Wider Design Prior Can Raise Assurance

Keep the same trial and analysis prior, so $s^2=2$ and $c=2.880$ units. Now set the design-prior mean to $m_D=2.5$ units, below the success threshold. Compare design-prior standard deviations of 0.5 and 2.5 units.

At $\tau_D=0.5$, the marginal standard deviation is $\sqrt{2+0.5^2}=\sqrt{2.25}=1.500$. Assurance is $\Phi((2.5-2.880)/1.500)=\Phi(-0.253)=0.400$, approximately.

At $\tau_D=2.5$, the marginal standard deviation is $\sqrt{2+2.5^2}=\sqrt{8.25}=2.872$. Assurance is $\Phi(-0.380/2.872)=\Phi(-0.132)=0.447$, approximately.

In this case, widening the design prior raises the calculated assurance. That does not mean the broader prior describes a better treatment effect: it increases weight on effects above the threshold even as it also increases weight on effects below zero. For example, under $N(2.5,2.5^2)$, the probability assigned to $\delta<0$ is $\Phi(-1)=0.159$, whereas under $N(2.5,0.5^2)$ it is $\Phi(-5)$, approximately $2.87\times10^{-7}$. Assurance is an average of trial success probabilities, not a summary of whether every part of the prior is clinically desirable.

Keep Design-Prior and Analysis-Prior Changes Separate

An analysis-prior change affects the posterior success rule, whereas a design-prior change affects the averaging distribution. The distinction can be seen by holding the design prior at $N(4,1^2)$ in the first example and changing only the analysis-prior standard deviation.

Worked Example: Changing the Analysis Prior Instead

The trial still has $s^2=2$, the design prior has mean 4 and standard deviation 1 unit, and success still requires posterior probability above 0.975. With an analysis prior $N(0,2^2)$, the posterior variance is $V=(1/2+1/4)^{-1}=1.333$ square units. Thus $c=1.960(2)/\sqrt{1.333}=3.395$ units, and assurance is $\Phi((4-3.395)/\sqrt3)=\Phi(0.349)=0.637$, approximately.

With the broader analysis prior $N(0,10^2)$, $V=(1/2+1/100)^{-1}=1.961$ square units and $c=1.960(2)/\sqrt{1.961}=2.799$ units. Assurance becomes $\Phi((4-2.799)/\sqrt3)=\Phi(0.693)=0.756$, approximately. In both calculations the design prior is unchanged. The assurance difference arises because the analysis prior changes the posterior and hence the observed-data threshold.

These calculations should not be combined into a single unexplained change in “the prior.” To attribute sensitivity correctly, report the baseline design prior, each alternative design-prior spread, the fixed analysis prior and success rule, and the resulting assurance values. If analysis-prior sensitivity is also of interest, report those calculations separately and identify the posterior rule or threshold that changed.

Assumptions, Diagnostics, and Limitations

The closed-form calculations rely on the specified normal sampling model, known common outcome variance, fixed sample size, independent observations, and a normal design prior. They also assume the success criterion is exactly the stated posterior-probability rule and that the treatment effect is measured on the same scale throughout. In a real design, uncertain nuisance parameters, unequal allocation, dropout, or a nonnormal likelihood may require numerical integration or simulation rather than this formula. The assumptions are part of the calculation: a precise answer under a misspecified model is not automatically a reliable design assessment.

When assurance is estimated by simulation, inspect the spread of generated design-prior effects and the conditional success estimates across that range. Check that the simulated prior has the intended center and standard deviation, that the success rule is applied identically at every generated effect, and that Monte Carlo uncertainty is small enough to distinguish the alternatives being compared. If a narrow difference between assurance estimates is important, common random numbers can make the comparison more precise. For rare success events, the rare-event methods used for assurance estimation may be relevant, but changing prior width does not remove the need to check simulation error.

A common mistake is to widen a design prior and then also change the analysis prior, while describing the resulting assurance difference as sensitivity to design-prior spread. That comparison confounds two mechanisms: the design prior changes the distribution over which operating probability is averaged, and the analysis prior changes how the observed data are interpreted. The correction is to vary one distribution at a time, hold all other design and decision inputs fixed, and state explicitly which distribution changed.

Another limitation is interpretive. Assurance is conditional on the chosen design prior, model, and decision rule; it is not the probability that the trial will succeed under every plausible true effect, nor a guarantee of clinical benefit. A single assurance value can conceal sensitivity to the prior’s tail behavior or to its center. Report a range or curve of results when plausible widths lead to materially different conclusions, and explain the clinical basis for the prior scenarios rather than choosing a width solely because it yields a preferred assurance.

Key takeaway: To measure sensitivity to design-prior spread, change the design-prior width and recalculate the full assurance while holding the analysis prior, trial, and success rule fixed. A separate analysis-prior sensitivity calculation answers a different question.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In the normal model above, what does the design prior change, and what does the analysis prior change?
If $m_D>c$, what is the direction of the assurance change as $\tau_D$ increases under the stated normal model? Give the reason.
Suppose $m_D=c$. What is assurance for any finite $\tau_D$ in this model?
A calculation changes $\tau_D$ and $\tau_A$ simultaneously. Can the resulting assurance difference be attributed to design-prior spread alone? Explain.

Solutions. (1) The design prior changes the distribution used to average conditional success probabilities; the analysis prior changes the posterior and can change the observed-data success threshold. (2) Assurance decreases: the derivative has sign opposite to $m_D-c$, so it is negative when $m_D>c$. (3) The numerator in the assurance formula is zero, so assurance is $\Phi(0)=0.5$. (4) No. The change combines the effect of different averaging weights with the effect of a different posterior decision threshold; vary the priors separately to identify each contribution.