Two probabilities with different questions
At an interim analysis, a trial team may ask whether an experimental arm has a sufficiently large treatment effect to continue. A Bayesian decision rule can answer this by calculating the posterior probability that the effect exceeds a prespecified threshold. A frequentist analysis might instead report a p-value for a null hypothesis about the effect. Both calculations use the observed data, but they refer to different probability statements.
This distinction matters in a multi-arm multi-stage (MAMS) trial because a probability used to guide an arm-level decision is not automatically an error rate for the trial, and a p-value is not the probability that an arm is effective. The interpretation depends on the model, the event or null hypothesis, and the decision context.
The posterior probability refers to uncertainty about the parameter after combining the likelihood with the prior. The p-value refers to the distribution of data or a test statistic under a null hypothesis. Neither number is a general-purpose measure of “how convincing” the evidence is: each has a defined question and assumptions.
Assumptions and a normal-model calculation
For the examples, suppose a hypothetical arm-versus-control contrast is summarized by an estimate \(y\) of \(\delta\), with known standard error \(s_y\). Assume the sampling model \(y\mid\delta\sim N(\delta,s_y^2)\), and specify a normal prior \(\delta\sim N(m_0,v_0)\). The normal model assumes the contrast estimate is unbiased for \(\delta\), has the stated standard error, and is adequately described by a normal sampling distribution. The prior must represent the analysis plan's prespecified beliefs about plausible effects; it is not inferred from the current contrast alone.
Under these assumptions, the posterior distribution is normal. Multiplying the normal likelihood and prior gives posterior precision equal to the sum of their precisions. Completing the square in the exponent gives the posterior variance and mean:
Thus, for a threshold \(c\), \(P(\delta>c\mid y)=1-\Phi((c-m_1)/\sqrt{v_1})=\Phi((m_1-c)/\sqrt{v_1})\), where \(\Phi\) is the standard normal cumulative distribution function. This is a posterior probability only under the stated likelihood and prior.
For comparison, a frequentist one-sided test of \(H_0:\delta\leq c\) against \(H_a:\delta>c\) uses the boundary value \(c\) in this normal model. Its p-value is \(1-\Phi((y-c)/s_y)\). It describes the upper-tail probability under the null boundary, not the posterior probability that \(\delta>c\). The p-value uses the sampling standard error and the null hypothesis; the posterior probability also uses the prior and posterior uncertainty.
Worked Example: Positive Benefit and a Null of No Benefit
Suppose, hypothetically, an experimental arm is estimated to reduce systolic blood pressure by \(4.00\) mmHg relative to control, with standard error \(2.00\) mmHg. Define \(\delta\) as the reduction in mmHg, so positive values favor the experimental arm. Use the normal likelihood above and a prior \(\delta\sim N(0,10^2)\) mmHg. The model assumes the contrast estimate is normal with the stated known standard error, and the prior is centered at no effect with a standard deviation of \(10\) mmHg.
The posterior variance and mean are
The posterior standard deviation is \(\sqrt{3.846}=1.961\) mmHg. The posterior probability of any benefit is therefore
Under this model and prior, the posterior probability that the reduction is greater than zero is about \(97.51\%\). This is not a probability that the null hypothesis is true or false; it is a probability assigned to the parameter event \(\delta>0\) conditional on the data and model.
For the frequentist one-sided test \(H_0:\delta\leq0\) versus \(H_a:\delta>0\), the test statistic at the null boundary is \(z=(4.00-0)/2.00=2.00\). The p-value is \(1-\Phi(2.00)\approx0.0228\). It is the probability, under the boundary null and the assumed sampling model, of obtaining a statistic at least this large. The two reported values differ because they answer different questions; their numerical sizes are not directly comparable.
Clinical thresholds are not null hypotheses by default
A treatment decision often depends on whether an effect exceeds a clinically meaningful threshold, not merely whether it differs from zero. For instance, a reduction greater than \(1.50\) mmHg may be the criterion for a hypothetical decision. The Bayesian quantity \(P(\delta>1.50\mid y)\) directly describes uncertainty about that parameter event. A frequentist p-value can also be computed for \(H_0:\delta\leq1.50\), but it still answers a sampling-probability question under that null.
Worked Example: Exceeding a Prespecified Benefit Threshold
Suppose a hypothetical arm has an estimated reduction of \(y=2.00\) mmHg with known standard error \(s_y=0.500\) mmHg. Retain the prior \(\delta\sim N(0,10^2)\), and set the clinical criterion at \(c=1.50\) mmHg. The normal sampling model and prior are the assumptions for the posterior calculation; for the p-value, use the corresponding normal sampling model and test the null boundary \(1.50\) mmHg.
The posterior variance is \(v_1=(1/100+1/0.25)^{-1}=0.2494\) mmHg squared, and the posterior mean is \(m_1=0.2494(0+2.00/0.25)=1.995\) mmHg. Hence the posterior standard deviation is \(0.4994\) mmHg, and
Under the specified model and prior, the posterior probability that the mean reduction exceeds \(1.50\) mmHg is about \(83.92\%\). This does not by itself establish that the arm should continue: the decision rule may also depend on harms, costs, or other prespecified considerations.
For \(H_0:\delta\leq1.50\) against \(H_a:\delta>1.50\), the boundary statistic is \(z=(2.00-1.50)/0.500=1.00\). The one-sided p-value is \(1-\Phi(1.00)\approx0.1587\). It is not the probability that \(\delta\) exceeds \(1.50\); that event has the posterior probability calculated above.
Why the prior matters to one quantity but not the other
For the same likelihood, changing the prior can change the posterior probability because the prior contributes information to the posterior. It does not change the frequentist p-value for a fixed data summary, standard error, null hypothesis, and test procedure. This is a difference in construction, not a contradiction. A prior-sensitive posterior probability can be appropriate for a Bayesian decision, while a p-value can be appropriate for assessing data compatibility with a specified null under repeated sampling.
Worked Example: Same Data, Different Prior Information
Use the same hypothetical estimate \(y=2.00\) mmHg, standard error \(0.500\) mmHg, and clinical threshold \(c=1.50\) mmHg. Now suppose the analysis instead uses a more concentrated, skeptical prior \(\delta\sim N(0,0.5^2)\) mmHg. This prior and the normal likelihood are explicit assumptions; they imply more prior concentration near zero than the prior in the preceding example.
The posterior variance is \(v_1=(1/0.25+1/0.25)^{-1}=0.1250\) mmHg squared, and the posterior mean is \(m_1=0.1250(0+2.00/0.25)=1.000\) mmHg. The posterior standard deviation is \(\sqrt{0.1250}=0.3536\) mmHg, so
With this prior, the posterior probability of exceeding the threshold is about \(7.87\%\), rather than \(83.92\%\) under the less concentrated prior. The data and the frequentist test have not changed: for \(H_0:\delta\leq1.50\), the one-sided p-value remains \(1-\Phi((2.00-1.50)/0.500)=0.1587\). The contrast illustrates why prior choice and its clinical justification should be examined when a posterior probability drives an interim decision.
Interpretation, diagnostics, and alternatives
The introductory material on interim efficacy decisions from posterior probabilities frames posterior thresholds as components of decision rules. In using such a rule, distinguish the probability calculation from the action it triggers. A rule might continue an arm when \(P(\delta>c\mid y)\) exceeds a prespecified cutoff, but that cutoff is a decision criterion; it is not automatically a frequentist significance level or a guarantee of a particular false-positive rate.
Frequentist p-values also require context. The null hypothesis, direction of the alternative, test statistic, and one-sided or two-sided convention must be stated. A p-value for \(H_0:\delta=0\) does not answer whether \(\delta\) exceeds a clinically meaningful positive threshold. If a threshold is the actual scientific question, the null and alternative need to reflect that threshold.
In a MAMS trial, arm-specific posterior probabilities and arm-specific p-values do not, by themselves, account for all consequences of examining several arms across multiple stages. Posterior probabilities do not automatically control frequentist type I error, and a collection of unadjusted p-values does not automatically provide familywise error control. The trial's prespecified decision and error-control procedures determine how the quantities are used.
| Quantity | Question answered | Key inputs | Common misreading |
|---|---|---|---|
| \(P(\delta>c\mid y)\) | Given the data, model, and prior, what posterior probability is assigned to \(\delta>c\)? | Likelihood, prior, observed data, and threshold \(c\) | “The probability that the null is true” |
| Frequentist p-value | Under the specified null, how often would a statistic at least this incompatible with it occur? | Null hypothesis, sampling model, statistic, and tail convention | “The probability that the observed result is due to chance” |
| Confidence interval | Which parameter values are compatible with a procedure having stated long-run coverage? | Sampling model and interval procedure | “The posterior probability that the fixed parameter lies in this realized interval” |
Useful checks include confirming that the effect direction and units are consistent across the estimate and threshold; checking that the prior has the intended scale and meaning; and verifying the sampling standard error and normal approximation. For Bayesian calculations near a decision cutoff, numerical accuracy also matters: a reported probability should be checked against adequate computation, especially if Monte Carlo methods are used. For frequentist calculations, verify that the p-value's tail matches the alternative and that the null boundary corresponds to the question of interest.
A common mistake is to say, “The p-value is 0.16, so there is a 16% chance the arm fails to exceed the clinical threshold.” That statement is wrong: the p-value is calculated under the null, not as a probability of the parameter event. The correction is to report the posterior probability of exceeding the threshold if that is the desired probability statement, or to report the p-value with its null hypothesis and sampling interpretation. Another mistake is to call a posterior probability of \(0.84\) “statistically significant.” It is a posterior probability; whether it leads to an action depends on the prespecified decision rule and the broader trial objectives.
The central limitation is that neither quantity can answer every decision question. A posterior probability is conditional on the likelihood and prior, while a p-value depends on the null and repeated-sampling model. Both can be misleading if the model is unsuitable, the threshold is chosen after seeing the data, or the multiplicity and stage structure of the MAMS design are ignored. Reporting the target event or null, assumptions, units, and decision context makes the interpretation auditable.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In a normal model, an analyst reports \(P(\delta>0\mid y)=0.95\). What event does this describe, and what does it not describe?
- An estimate is \(1.20\) units with standard error \(0.40\). For the one-sided test \(H_0:\delta\leq0\) versus \(H_a:\delta>0\), calculate the boundary z-statistic and approximate p-value.
- For a fixed estimate, standard error, null hypothesis, and frequentist test, what happens to the p-value when the Bayesian prior changes? Explain briefly.
- Why is a p-value for \(H_0:\delta=0\) not a direct test of whether \(\delta\) exceeds a clinically meaningful threshold \(c>0\)?
Solutions. (1) It is the posterior probability, conditional on the stated model and prior and the observed data, that the parameter is positive. It is not the probability that the null is true, nor a frequentist p-value. (2) \(z=(1.20-0)/0.40=3.00\), so the one-sided p-value is \(1-\Phi(3.00)\approx0.00135\), under the assumed normal sampling model. (3) It does not change: the frequentist p-value is determined by the data summary, null, sampling model, and test. The posterior probability may change because the prior contributes to the posterior. (4) The zero-effect null and the clinical-threshold question specify different parameter boundaries. To test whether the effect exceeds \(c\), the relevant one-sided null is \(H_0:\delta\leq c\), with the p-value calculated under that null.