Why Examine Skeptical Analysis Priors?
A trial’s assurance depends not only on its sample size and the range of effects considered plausible at the design stage, but also on the analysis rule that will declare success. In particular, a skeptical analysis prior centered at no treatment effect can require stronger data before the posterior probability of benefit crosses its success threshold. This can reduce conditional power at a fixed true effect and, in turn, reduce assurance.
The analysis prior and the design prior have different jobs. The analysis prior is used to update beliefs after observing trial data and is part of the prespecified decision rule. The design prior represents uncertainty about the true effect before data are observed and is used to average over possible trial outcomes when calculating assurance. A sensitivity analysis changes the analysis prior while holding the design and success criterion fixed, so that the effect of prior skepticism is visible.
The calculations below use a normal endpoint. Let \(\theta\) be the treatment-minus-control difference in population means, measured in the endpoint’s units, and let \(Y\) be the trial estimate of that difference. Assume a planned standard error \(s_e\) and sampling model \(Y\mid\theta\sim N(\theta,s_e^2)\). The analysis declares success when the posterior probability that \(\theta\) is positive exceeds a specified threshold \(\gamma\).
Success Cutoff Under a Skeptical Prior
For the normal sampling model, suppose the planned standard error is known and constant, the treatment effect is represented by the same \(\theta\) in the prior and likelihood, and the analysis prior is \(\theta\sim N(0,\tau_a^2)\), with \(\tau_a>0\). These are the assumptions for the calculations that follow. They make the posterior normal and give a single data threshold for success. In practice, the normal model and planned standard error are approximations when the endpoint or analysis is more complicated.
Write \(w=\tau_a^2/(\tau_a^2+s_e^2)\). Normal-normal conjugacy gives posterior mean \(wY\) and posterior variance \(V=s_e^2\tau_a^2/(s_e^2+\tau_a^2)\). Let \(z_\gamma\) be the \(\gamma\) quantile of the standard normal distribution. Since a normal posterior with mean \(wY\) and variance \(V\) assigns probability \(\Phi(wY/\sqrt V)\) to \(\theta>0\), success requires that probability to exceed \(\gamma\). Thus the rule can be written as a cutoff on the observed estimate.
This cutoff is the key to the scenario calculations. When \(\gamma>0.5\), \(z_\gamma>0\), so the cutoff increases as \(\tau_a\) decreases: a more concentrated skeptical prior requires a larger observed treatment effect to satisfy the same posterior-probability rule. When \(\gamma=0.5\), \(z_\gamma=0\) and the cutoff is zero for every \(\tau_a\). When \(\gamma<0.5\), \(z_\gamma<0\), so the cutoff moves downward as \(\tau_a\) decreases. Therefore, the claim that increasing skepticism raises the cutoff requires \(\gamma>0.5\).
Worked Example: Deriving the Cutoff and Conditional Power
Consider a hypothetical normal-endpoint trial with \(s_e=0.25\) units, analysis prior \(N(0,0.25^2)\), and success threshold \(\gamma=0.95\). Use \(z_\gamma=1.645\). Find the cutoff and the conditional power if the true effect is \(0.50\) units.
Here \(\tau_a=s_e=0.25\), so \(s_e^2/\tau_a^2=1\). The cutoff is
The assumptions are the normal sampling model, fixed planned standard error, and the stated normal analysis prior. At a true effect of \(0.50\) units, \(Y\sim N(0.50,0.25^2)\). Therefore,
The conditional probability of meeting the rule is about 37.2% when the true treatment benefit is \(0.50\) units. The cutoff exceeds that effect, so success is not more likely than failure under this scenario.
From Conditional Power to Assurance
At a fixed true effect, the success rule is \(Y>c\), and the sampling model gives \(Y\mid\theta\sim N(\theta,s_e^2)\). Consequently, conditional power has a direct expression. To obtain assurance, specify a design prior and average this conditional probability over it. For a normal design prior, this integration also has a closed form.
Suppose the design prior is \(\theta\sim N(\mu_d,\tau_d^2)\), with \(\mu_d\) and \(\tau_d\) distinct from the analysis-prior parameters. Because \(Y=\theta+\varepsilon\), where \(\varepsilon\sim N(0,s_e^2)\) is independent of \(\theta\), the marginal distribution is \(Y\sim N(\mu_d,\tau_d^2+s_e^2)\). Assurance is therefore the probability that this marginal estimate exceeds the analysis-rule cutoff.
This result shows precisely how a skeptical analysis prior affects assurance under a fixed design prior: for \(\gamma>0.5\), a smaller \(\tau_a\) raises \(c\), reducing \(P(Y>c)\). It does not change the distribution of true effects specified by the design prior. The result assumes a normal design prior and a constant standard error; with other design distributions or data-dependent standard errors, assurance can instead be evaluated by integration or simulation under the specified model.
Worked Example: Comparing Two Analysis-Prior Scenarios
Suppose a hypothetical trial has \(s_e=0.20\) units and \(\gamma=0.95\), using \(z_\gamma=1.645\). Its design prior is \(\theta\sim N(0.35,0.20^2)\), with effects measured in endpoint units. Compare an analysis prior \(N(0,0.15^2)\) with a less concentrated prior \(N(0,1.00^2)\). The normal model and the same planned standard error are assumed in both scenarios.
For \(\tau_a=0.15\), the cutoff is
The marginal standard deviation of \(Y\) under the design prior is \(\sqrt{0.20^2+0.20^2}\approx0.2828\) units. Using the unrounded cutoff in the standardization gives approximately \(0.7012\), so assurance is
For \(\tau_a=1.00\), the cutoff is
Using the unrounded cutoff and marginal standard deviation gives a standardized cutoff of approximately \(-0.0512\). Thus,
Under these hypothetical assumptions, assurance is about 24.2% with the more concentrated prior and 52.0% with the less concentrated prior. The difference arises from the analysis cutoff, not from a change in the design prior or sampling variability.
How to Assess a Skeptical-Prior Scenario
A scenario comparison is informative only if it changes what it is intended to examine. Keep the endpoint scale, sample size and allocation (as reflected in \(s_e\)), success threshold, and design prior fixed when comparing analysis priors. State each analysis-prior scale in the endpoint’s units: the same numerical \(\tau_a\) can represent very different degrees of skepticism on different scales. If a treatment effect is standardized or transformed, the prior and cutoff must be expressed on that same scale.
Several checks help identify a calculation or interpretation that may be going wrong:
- Check the direction against \(\gamma\). A smaller \(\tau_a\) raises the cutoff only when \(\gamma>0.5\). If \(\gamma=0.5\), the cutoff is zero; if \(\gamma<0.5\), it decreases as the prior concentrates.
- Check the probability being calculated. Conditional power fixes \(\theta\); assurance averages over the design prior. Do not average over the analysis prior when the question is how the specified design prior affects assurance.
- Check units and standardization. The cutoff \(c\) has the same units as the treatment effect, while \((c-\mu_d)/\sqrt{\tau_d^2+s_e^2}\) is dimensionless. A standard error or prior spread on a different scale cannot be substituted without conversion.
- Check the success rule. The cutoff above follows from \(P(\theta>0\mid Y)>\gamma\) and a zero-centered normal prior. A different posterior event, threshold, likelihood, or prior may give a different rule.
The main alternative to an assurance calculation is to report conditional power at one or more specified true effects. That answers a clear scenario question but does not average over uncertainty about which effect will occur. Assurance does that averaging, conditional on the chosen design prior. A useful sensitivity presentation can show both: conditional power across clinically interpretable effects and assurance across plausible analysis-prior scenarios. Neither approach eliminates the need to justify the assumptions used to specify its effects or priors.
Common Pitfalls and Limitations
A common mistake is to say, “A more skeptical prior always raises the cutoff.” That statement omits the condition on \(\gamma\). The cutoff is \(z_\gamma s_e\sqrt{1+s_e^2/\tau_a^2}\): its dependence on \(\tau_a\) is multiplied by \(z_\gamma\). It rises as \(\tau_a\) falls when \(\gamma>0.5\), is unchanged when \(\gamma=0.5\), and falls when \(\gamma<0.5\).
Another mistake is to treat the design prior as if it were the analysis prior. They may both be distributions for \(\theta\), but they serve different roles in this calculation. Changing \(\tau_a\) changes the decision cutoff; changing \(\mu_d\) or \(\tau_d\) changes the distribution over which success is averaged. Mixing these roles makes it impossible to interpret a scenario comparison.
Finally, a closed-form normal calculation is not a guarantee that the design is realistic. The result depends on a normal approximation for the estimate, the assumed standard error, the selected design prior, and the particular success rule. A skeptical prior’s concentration should be justified on the clinical effect scale, and the sensitivity scenarios should be presented as assumptions rather than facts about the eventual treatment effect. For binary outcomes, nonnormal likelihoods, or complex decision rules, the same distinction between analysis and design assumptions remains useful, but the normal cutoff formula need not apply.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- For \(s_e=0.25\), \(\tau_a=0.25\), and \(\gamma=0.95\) with \(z_\gamma=1.645\), calculate the success cutoff.
- Using that cutoff, find conditional power when the true effect is \(0.50\) units.
- In the comparison example, what happens to assurance when the analysis prior changes from \(N(0,0.15^2)\) to \(N(0,1.00^2)\), and why?
- What is the cutoff when \(\gamma=0.5\), and how does it depend on \(\tau_a\)?
- Explain the distinct roles of the analysis prior and design prior in assurance.
Solutions. (1) \(c=1.645(0.25)\sqrt{2}\approx0.5816\) units. (2) \(1-\Phi((0.5816-0.50)/0.25)=1-\Phi(0.3264)\approx0.3721\), or 37.2%. (3) Assurance rises from about 0.2416 to 0.5204 because, with \(\gamma=0.95>0.5\), the less concentrated analysis prior gives a lower success cutoff; the design prior and sampling model are unchanged. (4) Since \(z_{0.5}=0\), \(c=0\) for every \(\tau_a>0\). (5) The analysis prior determines the posterior success rule and its cutoff; the design prior describes uncertainty about the true effect and supplies the distribution over which conditional power is averaged.