Interim Efficacy Decisions From Posterior Probabilities

Why Make an Interim Efficacy Decision?

A multi-arm trial may collect enough information to identify a promising treatment before its planned final analysis. An interim efficacy rule states in advance what posterior evidence is sufficient to make a decision at a scheduled look. When the rule is crossed, the protocol might stop assigning participants to that arm, stop the trial, or take another specified action. The statistical rule alone does not determine which action follows.

For efficacy, the important question is not simply whether an arm’s effect is positive. A small positive effect may be too small to matter clinically. Instead, define a clinically meaningful benefit margin and ask how much posterior probability the treatment effect exceeds that margin. The resulting rule is transparent, but its interpretation depends on the model, prior, clinical margin, and threshold having been specified appropriately.

Definition: Let $\delta_j$ be the treatment effect for arm $j$, oriented so that larger values indicate greater benefit, and let $\Delta>0$ be the prespecified minimum clinically meaningful benefit. At interim look $k$, an arm-specific posterior efficacy rule declares arm $j$ to have met the efficacy criterion when $$ P(\delta_j>\Delta\mid D_k)\ge\gamma, $$ where $D_k$ is all relevant data available at that look and $\gamma$ is a prespecified posterior probability threshold.

The rule asks whether the posterior probability of exceeding the clinical margin reaches the decision threshold. These are different quantities: $\Delta$ is on the treatment-effect scale, while $\gamma$ is a probability. For example, a risk difference of $0.10$ is a 10-percentage-point benefit margin; a threshold of $0.99$ requires at least 99% posterior probability that the true risk difference exceeds 10 percentage points.

The direction and units must be clear. If an outcome is time to an adverse event, a smaller value may be better, so the effect should either be defined in the beneficial direction or the inequality reversed consistently. The efficacy rule should also state which looks are eligible, whether it applies separately to each arm, and what action follows a crossing.

A Normal-Posterior Boundary

A useful derivation comes from a normal model for an estimated treatment effect. Suppose the cumulative data at look $k$ provide an estimate $D_{jk}$ satisfying $D_{jk}\mid\delta_j\sim N(\delta_j,v_{jk})$, where $v_{jk}$ is known. Suppose also that the prior is $\delta_j\sim N(\mu_{0j},\tau_{0j}^{2})$, with known prior mean and variance. These assumptions give a normal posterior. They are an illustrative model, not a requirement for defining the decision rule.

\delta_j\mid D_k\sim N(m_{jk},s_{jk}^{2}),\qquad s_{jk}^{2}=\left(\frac{1}{\tau_{0j}^{2}}+\frac{1}{v_{jk}}\right)^{-1},\qquad m_{jk}=s_{jk}^{2}\left(\frac{\mu_{0j}}{\tau_{0j}^{2}}+\frac{D_{jk}}{v_{jk}}\right).

The posterior variance follows by multiplying the normal prior density by the normal likelihood: the coefficients of $\delta_j^2$ in the log densities add, giving posterior precision $1/\tau_{0j}^{2}+1/v_{jk}$. The corresponding linear coefficients add as well, producing the stated posterior mean. Since a normal distribution is symmetric, its probability above $\Delta$ is the standard normal cumulative probability at the posterior mean’s distance above $\Delta$, measured in posterior standard deviations.

P(\delta_j>\Delta\mid D_k) =1-\Phi\left(\frac{\Delta-m_{jk}}{s_{jk}}\right) =\Phi\left(\frac{m_{jk}-\Delta}{s_{jk}}\right).

Here $\Phi$ is the standard normal cumulative distribution function. Because $\Phi$ is increasing, the efficacy rule is equivalent to a boundary on the posterior mean:

P(\delta_j>\Delta\mid D_k)\ge\gamma \quad\Longleftrightarrow\quad \frac{m_{jk}-\Delta}{s_{jk}}\ge z_\gamma \quad\Longleftrightarrow\quad m_{jk}\ge\Delta+z_\gamma s_{jk},

where $z_\gamma$ is the $\gamma$-quantile of the standard normal distribution. Thus, when posterior uncertainty is larger, the posterior mean must be farther above the clinical margin to cross the same probability threshold. This is why a positive posterior mean, by itself, does not establish interim efficacy.

Interpretation: Under the specified posterior model, $P(\delta_j>\Delta\mid D_k)$ is the posterior probability, conditional on the observed data and model, that arm $j$'s effect exceeds the clinical margin. It is not the probability that the null hypothesis is true, nor is it a frequentist type I error rate.

Applying the Rule at an Interim Look

The following examples use hypothetical posterior summaries. Effects are risk differences in percentage points expressed as proportions: for example, $0.10$ is a 10-percentage-point increase in response probability. Each example assumes that the treatment effect is oriented so larger values mean greater benefit, that the posterior summaries arise from the prespecified analysis model, and that the look is one of the protocol’s eligible interim looks.

Worked Example: Positive Posterior Mean, but No Efficacy Crossing

Suppose the clinically meaningful benefit margin is $\Delta=0.10$, the efficacy threshold is $\gamma=0.975$, and the interim posterior for an arm’s risk difference is $N(0.14,0.045^2)$. Calculate the probability and apply the rule.

The posterior mean is above zero, but the decision concerns whether the effect exceeds $0.10$. The standardized distance above the margin is

\frac{m-\Delta}{s} =\frac{0.14-0.10}{0.045} =\frac{0.04}{0.045} \approx 0.8889.

Therefore,

P(\delta>0.10\mid D_k)=\Phi(0.8889)\approx 0.8130.

Since $0.8130<0.975$, this arm does not meet the interim efficacy criterion. It would be incorrect to declare efficacy merely because the posterior mean risk difference, $0.14$, is positive or exceeds the clinical margin. The posterior still assigns too much probability to effects at or below the margin to satisfy the prespecified rule.

Worked Example: Crossing a Stringent Threshold

At another eligible look, suppose arm A has posterior risk difference $N(0.1850,0.0330555^2)$, with $\Delta=0.10$ and $\gamma=0.99$. Determine whether it crosses the threshold.

The posterior standard deviation is $s_A=0.0330555$. The standardized distance from the clinical margin is

\frac{m_A-0.10}{s_A} =\frac{0.1850-0.10}{0.0330555} =\frac{0.0850}{0.0330555} \approx 2.571.

Consequently,

P(\delta_A>0.10\mid D_k)=\Phi(2.571)\approx 0.9949.

Because $0.9949\ge0.99$, arm A crosses the prespecified efficacy threshold at this look. Under the protocol, the corresponding action for arm A should now be taken. The result is conditional on the posterior model and observed interim data; it does not guarantee a particular effect in future patients.

Worked Example: Comparing Two Arms at the Same Look

Suppose two arms have posterior risk differences with the same posterior standard deviation, $s=0.040$. Use the clinical margin $\Delta=0.10$ and threshold $\gamma=0.975$. Arm C has posterior mean $m_C=0.175$, and arm D has posterior mean $m_D=0.180$. Which arm crosses?

The 0.975 standard normal quantile is approximately $z_{0.975}=1.960$. The posterior-mean boundary for either arm is

\Delta+z_{0.975}s =0.10+(1.960)(0.040) =0.1784.

Arm C’s mean, $0.175$, is below $0.1784$. Directly calculating its probability gives

P(\delta_C>0.10\mid D_k) =\Phi\left(\frac{0.175-0.10}{0.040}\right) =\Phi(1.875) \approx 0.9696.

Thus arm C does not cross. For arm D,

P(\delta_D>0.10\mid D_k) =\Phi\left(\frac{0.180-0.10}{0.040}\right) =\Phi(2.000) \approx 0.9772.

Arm D crosses because $0.9772\ge0.975$. Both posterior means exceed the clinical margin, but only arm D has sufficient posterior probability above it. This comparison also shows why rounding a probability near the threshold too aggressively can obscure the decision: retain adequate precision for the comparison and report the decision rule explicitly.

Prespecification, Diagnostics, and Limitations

A defensible rule requires more than choosing a probability cutoff after seeing the interim results. Before the trial begins, specify the clinical margin, posterior probability threshold, eligible looks, analysis model and prior, treatment-effect direction, and action after crossing. The earlier material on selecting posterior probability thresholds and on sensitivity analysis for posterior arm selection provides relevant context for choosing and checking such inputs. Elicited utilities can also clarify the consequences of acting too early or waiting for more evidence.

At each look, use all data accumulated to that point exactly once in the posterior analysis. Reusing cumulative data as if each look were a new independent data set would produce incorrect posterior uncertainty. Check that the outcome definition, control comparison, allocation, missing-data handling, and likelihood match the protocol and analysis. If the actual posterior is not normal, calculate the posterior probability from the specified posterior distribution rather than applying the normal formula mechanically.

In a multi-arm trial, an arm-specific rule can be evaluated using its marginal posterior probability. If the decision instead concerns a joint event, such as multiple arms meeting criteria or at least one arm succeeding, dependence between arms matters. Shared-control data can induce posterior dependence, so separate marginal probabilities do not generally determine the probability of a joint event. The decision event must match the protocol’s intended claim.

Repeated interim opportunities do not change the conditional meaning of a posterior probability at a given look. However, the chance that a rule will cross at some point during a trial depends on the design, prior, data-generating conditions, and timing of looks. Assess operating characteristics under plausible scenarios before implementation; a posterior threshold should not be treated as a universal guarantee of frequentist error control. Also examine sensitivity to reasonable alternative priors and model choices, especially when decisions are close to the boundary.

A posterior-mean-only rule is a tempting alternative, but it ignores uncertainty: two arms with the same mean and different posterior standard deviations can have very different probabilities of exceeding $\Delta$. A final-analysis-only rule avoids interim decisions but gives up the possibility of acting early. Frequentist group-sequential rules are another approach, with their own calibration and interpretation. These approaches answer related but distinct questions; they should not be blended without a clear decision rationale.

Common Mistakes

Testing for any benefit instead of meaningful benefit. Replacing $P(\delta_j>\Delta\mid D_k)$ with $P(\delta_j>0\mid D_k)$ changes the scientific question whenever $\Delta>0$.
Treating a posterior probability as a frequentist error rate. A 0.99 posterior probability is conditional on the model, prior, and observed data; it does not mean a 1% long-run false-positive rate.
Using an unspecified action. A statistical crossing is a trigger. The protocol must say whether it stops an arm, the whole trial, or initiates another specified action.
Ignoring uncertainty or the clinical scale. A posterior mean above the margin is not enough; compare the posterior probability with $\gamma$, and state the effect’s units and direction.
Changing the rule after seeing interim results. Choosing the margin, threshold, or eligible look in response to the observed data undermines the prespecified decision process.

Key takeaway: An interim efficacy decision is a prespecified comparison of $P(\delta_j>\Delta\mid D_k)$ with $\gamma$. The clinical margin defines meaningful benefit; the probability threshold defines how much posterior support is required. Both, together with the model and action following a crossing, belong in the trial design.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

An arm has posterior risk difference $N(0.16,0.03^2)$, a margin of $0.10$, and threshold $0.95$. Does it cross?
For a normal posterior with $\Delta=0.08$, $s=0.05$, and $\gamma=0.975$, what posterior mean is required to cross?
Why can two arms with the same posterior mean have different efficacy decisions?
Does a posterior efficacy threshold of $0.99$ by itself guarantee a 1% frequentist type I error rate? Explain.
If a protocol’s decision is that at least two arms must exceed their clinical margins, are the two marginal arm probabilities always enough to calculate the relevant posterior probability?

Solutions. (1) The standardized distance is $(0.16-0.10)/0.03=2.000$, so the posterior probability is $\Phi(2.000)\approx0.9772$. Since $0.9772\ge0.95$, the arm crosses. (2) The boundary is $\Delta+z_{0.975}s=0.08+(1.960)(0.05)=0.1780$, so the posterior mean must be at least $0.1780$. (3) Their posterior standard deviations may differ, changing the probability mass above the clinical margin even when the means match. (4) No. The 0.99 is a posterior probability conditional on the data and specified Bayesian model, not a frequentist long-run error-rate guarantee. (5) No. If the arm effects are dependent, their joint posterior distribution is needed to calculate the probability that at least two exceed their margins.