Interpreting Low Predictive Success Despite Favorable Interim Results

Why Favorable Interim Evidence May Not Predict Final Success

An interim analysis can support a favorable conclusion about a treatment while the probability of meeting the trial’s final success rule remains low. These statements answer different questions. The current posterior summarizes uncertainty about the treatment effect using the data observed so far. Predictive success asks how likely future data are, under the current posterior and the planned remaining design, to make the final posterior satisfy a specified rule.

The difference matters when little information remains: future results may not shift the posterior much, even if the current estimate is favorable. It also matters when the final rule is demanding: a treatment can look promising now yet still be unlikely to produce the strong final evidence required. Neither situation is a mathematical contradiction; each is a consequence of the remaining information and the decision threshold.

A Normal-Model Setup

To make the distinction precise, consider a hypothetical trial with a continuous endpoint. Let \(\theta\) be the treatment benefit in millimetres of mercury (mmHg), with positive values representing a reduction in systolic blood pressure relative to control. Let \(\delta\) be the minimum benefit considered clinically meaningful. At the interim analysis, suppose the posterior distribution is \(N(m,s^2)\). The notation \(N(\mu,\sigma^2)\) uses the second argument for the variance.

Assume the remaining trial data can be summarized by \(Y\), an estimate of the treatment benefit from the future observations alone. Conditional on \(\theta\), suppose \(Y\sim N(\theta,v)\), where the known variance \(v\) reflects the remaining sample size and design. Assume the future observations are independent of the observed interim data conditional on \(\theta\), and that the likelihood and prior are correctly specified. With the current normal posterior as the distribution to update, the final posterior is normal.

The posterior update uses the precision-weighted combination of the interim information and the future estimate. Writing \(m_F\) for the final posterior mean and \(s_F^2\) for its variance gives:

The final posterior success rule therefore becomes \(m_F\geq\delta+z_\gamma s_F\). Predictive success is the probability, conditional on the interim data, that the future estimate \(Y\) makes this inequality hold. This formulation separates the two ingredients: how much the posterior can move with future data, and how far the final posterior must move to cross the threshold.

Why the Predictive Probability Can Be Low

The calculation follows from the predictive distribution of \(Y\). Given the interim data, uncertainty about \(\theta\) remains, so future data vary both because of that parameter uncertainty and because of sampling variation. Under the assumptions above, \(Y\mid\text{interim data}\sim N(m,s^2+v)\). Since \(m_F\) is a linear function of \(Y\), it is also normally distributed under this predictive distribution.

The equality uses \(s_F^2(1/s^2+1/v)=1\). The predictive variance of the final posterior mean is:

Thus \(m_F\mid\text{interim data}\sim N(m,s^2-s_F^2)\), and predictive success can be calculated directly:

where \(\Phi\) is the standard normal cumulative distribution function. This expression assumes \(s^2>s_F^2\), as holds when there is positive future information. If no information remains, \(s_F=s\) and the final posterior does not change; success is then determined by whether the current posterior already meets the criterion.

Separating the Effects of the Threshold and Remaining Information

The same interim posterior can yield quite different predictive success probabilities if the final criterion changes. That is why a predictive-success estimate should always be reported alongside its success rule. A probability without the criterion it refers to is incomplete.

Remaining sample size affects a different part of the calculation: how much the final posterior can change. With the same interim posterior and success rule, more future information generally permits greater movement in the final posterior mean. This does not guarantee success; it can increase the chance of either crossing or missing the threshold, depending on where the threshold lies relative to the current mean.

Diagnostics, Interpretation, and Common Mistakes

Before interpreting a low estimate, check that it reflects the intended model and design rather than an input or implementation error. Confirm the direction and units of \(\theta\), the definition of the clinically meaningful threshold \(\delta\), the final posterior probability level \(\gamma\), and the remaining information represented by \(v\). A sign reversal in the treatment effect or a mismatch between a variance and a standard deviation can change the result substantially. The computed final mean threshold should also be compared with the current mean: when it lies well above \(m\) and the predictive standard deviation is small, low predictive success is plausible.

The normal calculation relies on a normal interim posterior, a normal future-data summary with known variance, and a future design whose variance is correctly specified. If the remaining sample size, allocation, or data collection changes in response to interim results, the predictive distribution must reflect that rule. A fixed-\(v\) calculation is not appropriate if it ignores such adaptation. For skewed or discrete posteriors, a direct posterior predictive calculation or simulation may be more suitable than this normal formula. These are model and design checks, not reasons to reinterpret predictive success as a probability about the truth of the null hypothesis.

A common mistake is to say, “The current posterior probability of a clinically meaningful benefit is 90%, so there is a 90% chance the trial will succeed.” The correction is that 90% refers to \(P(\theta>\delta\mid\text{interim data})\), whereas predictive success is the probability that future data will make the final posterior satisfy the specified final rule. The two probabilities have different events. Another mistake is to report predictive success without stating the rule: a final 80% posterior criterion and a final 97.5% criterion can produce very different answers for the same interim data.

Predictive success also differs from calculations that assume a fixed true effect and ask how often a trial would succeed under repeated sampling at that value. Predictive success averages over uncertainty about the effect represented by the current posterior, as well as over future sampling. It is conditional on the observed interim data and specified model; it is not, by itself, a long-run operating characteristic of the trial procedure. Its value for interim interpretation is precisely that it combines current uncertainty, the planned remaining information, and the actual final decision rule.