A Skeptical Prior in a Small Phase 2 Trial

Why Prior Scale Matters in a Small Trial

A small phase 2 trial can produce an observed treatment effect that is clinically promising but imprecise. In this setting, the prior distribution may have a visible influence on the posterior: a skeptical prior pulls estimates toward no average effect, while a broader prior allows the observed data to move the estimate more. Comparing these priors makes the influence explicit. It does not determine which prior is correct; that choice requires clinical justification.

This tutorial considers a randomized trial with a continuous endpoint. The endpoint is improvement on a 0-to-100 symptom score, where a higher score means greater improvement. Let \(\theta\) be the treatment-group mean improvement minus the control-group mean improvement, in score points. Thus positive values favor treatment. Let \(\delta=1\) point be the prespecified minimum clinically meaningful benefit. The decision probability of interest is \(P(\theta>\delta\mid D)\), where \(D\) denotes the observed trial data.

The distinction between a continuous prior centered at zero and a point mass at zero matters. The normal priors used here allow positive and negative effects, including effects close to zero; they do not assert that treatment has exactly no effect. The analysis reports posterior probabilities for effect thresholds, not the probability that a null hypothesis is true.

Model and Assumptions

Let \(D=\bar{x}_T-\bar{x}_C\) be the observed difference in mean improvement, treatment minus control. Assume the conditional sampling model and prior are

The second argument of \(N\) is the variance. The prior mean is zero, and \(\tau\) controls its spread. Smaller \(\tau\) expresses stronger skepticism about effects far from zero. Conditional on the model, \(v\) is the sampling variance of the observed contrast.

This simple model is appropriate only under stated conditions. The treatment contrast must correspond to the trial’s prespecified estimand, and randomization, outcome measurement, and analysis must support a valid comparison of arms. The normal sampling model should be a reasonable description of the contrast, with no important unmodelled features such as severe skewness or influential observations. The example treats the residual standard deviation as known, so that \(v\) is fixed. In an actual small trial, uncertainty in the residual variance may need to be modeled rather than ignored. The prior scale should be expressed in the endpoint’s units and checked against clinical knowledge.

A prior centered at zero is a modeling choice, not a consequence of randomization. It can be reasonable when there is a substantive reason to temper large effects, but it should not be selected after looking at the trial result. The endpoint direction, clinical threshold, prior scales, and decision rule should be specified or justified transparently.

Deriving the Posterior

The likelihood for \(\theta\), apart from a factor that does not depend on \(\theta\), is proportional to \(\exp\{-(D-\theta)^2/(2v)\}\). Multiplying it by the normal prior gives the posterior kernel:

Complete the square in \(\theta\). The coefficient of \(\theta^2\) is \(1/v+1/\tau^2\), so the posterior variance is its reciprocal. The posterior mean is that variance multiplied by \(D/v\). Therefore,

The posterior mean is a weighted version of the observed contrast. The factor \(\tau^2/(v+\tau^2)\) is nearer zero for a more concentrated prior, so that prior produces more shrinkage toward zero. For any threshold \(\delta\), the posterior probability of exceeding it is

where \(\Phi\) is the standard normal cumulative distribution function. This probability is conditional on the stated model, prior, and observed data.

Reading the Comparison and Checking Its Assumptions

The comparison does not show that the broader prior is preferable simply because its posterior probability crosses a decision cutoff. It shows that, with these data, the conclusion is sensitive to a defensible modeling choice that should be visible to readers. Reporting both analyses can be useful when reasonable prior scales lead to different decisions. The result should include the effect definition and units, the prior distributions, the posterior summaries, and the decision threshold.

Before interpreting a result, check that the assumed model fits the question and the data. Confirm which direction represents benefit and that \(\delta\) is in the same units as \(\theta\). Check whether the uncertainty assigned to the contrast includes all relevant sources, including estimation of the outcome variance. Review the observed data for influential values or patterns inconsistent with the assumed sampling model. Finally, check the priors in the endpoint’s clinical context: a prior can be mathematically convenient while allowing effects that experts consider implausible.

A decision probability is sensitive not only to the prior but also to the clinical threshold. The next example keeps the data and priors fixed and varies the threshold. This is not an invitation to choose the threshold after seeing the results; it illustrates why the threshold must be clinically motivated.

When More Data Change the Picture

A concentrated prior has greater influence when the likelihood is imprecise. As the sampling variance \(v\) falls, the likelihood precision \(1/v\) rises relative to the prior precision \(1/\tau^2\), and the posterior mean becomes less dominated by the prior. This is one reason a skeptical prior can matter especially in a small trial. The next example illustrates that the prior does not prevent a sufficiently strong observed effect from supporting a clinically meaningful benefit.

Alternatives, Limitations, and Common Pitfalls

A broader normal prior is a direct comparator because it keeps the model and prior center fixed while changing the prior scale. A frequentist analysis of the same contrast can provide a useful comparison of estimates and uncertainty, but its testing framework does not yield \(P(\theta>\delta\mid D)\). A point-null approach is also different: assigning positive prior probability to exactly \(\theta=0\) changes the prior model and the questions answered. None of these alternatives removes the need to justify the endpoint, assumptions, and clinical threshold.

The normal-normal calculation is deliberately transparent, but it has limitations. Treating the standard deviation as known can understate uncertainty when it is estimated from only a few participants. A normal model for a contrast may be inadequate for highly skewed, bounded, or otherwise non-normal outcomes. A normal prior may also put too much probability on effects that are not clinically plausible. In such situations, use a likelihood and prior that reflect the outcome and design, and propagate relevant uncertainty rather than relying on this conjugate calculation unchanged.

Common mistake: “The skeptical analysis gives only a 0.632 probability, so the probability that the treatment works is 0.632, and the null is probably true.” This is incorrect. The calculated quantity is the probability, under a particular model and prior and conditional on these data, that the mean effect exceeds the specified 1-point threshold. It is not the probability that treatment has any positive effect, and it is not the probability that a null hypothesis is true. State the threshold, prior, and conditioning explicitly.

A second pitfall is calling one prior “objective” because it is broader. A weakly informative prior still makes assumptions, including a center and scale. Show the prior’s implications in the clinical units, explain why its scale is reasonable, and report whether the decision changes under alternative plausible scales. This is a focused sensitivity comparison, not a substitute for checking the model or for planning a trial capable of answering the clinical question.