Predictive Success When Control Response Is Uncertain

Why Control-Rate Uncertainty Matters

In a binary treatment-control trial, predictive success is the probability, given the data observed so far, that the final analysis will meet a prespecified success rule. The future control response rate is often not known exactly. Treating it as a fixed number, such as its current posterior mean, can understate the variation in future control outcomes and can misrepresent how those outcomes will update the final comparison.

The key is to average over the current posterior distribution of the control response rate as well as over future control outcomes. This is not the same as setting the rate equal to its posterior mean and simulating a binomial count. The difference is especially important when the posterior is diffuse or when several control participants remain.

Model and Predictive-Success Calculation

Suppose the treatment arm has observed \(x_T\) responses among \(n_T\) participants, and the control arm has observed \(x_C\) responses among \(n_C\) participants. Use independent beta priors, \(p_T\sim\operatorname{Beta}(a_{T0},b_{T0})\) and \(p_C\sim\operatorname{Beta}(a_{C0},b_{C0})\). Conditional on each response probability, outcomes are Bernoulli and independent within and between arms. These assumptions give independent current posteriors: \(p_j\mid D\sim\operatorname{Beta}(a_j,b_j)\), where \(a_j=a_{j0}+x_j\) and \(b_j=b_{j0}+n_j-x_j\), for \(j=T,C\). Here \(D\) denotes the current data.

Let \(m_T\) and \(m_C\) be the numbers of remaining participants. Conditional on the response probabilities, future response counts \(Y_T\) and \(Y_C\) follow independent binomial distributions. A final decision rule might declare success when the final posterior probability of superiority reaches a threshold \(\gamma\). For each possible pair of future counts, the final posteriors are \(\operatorname{Beta}(a_T+Y_T,b_T+m_T-Y_T)\) and \(\operatorname{Beta}(a_C+Y_C,b_C+m_C-Y_C)\).

For a current \(\operatorname{Beta}(a,b)\) posterior and \(m\) remaining participants, integrating the binomial likelihood over the posterior gives

Here \(B(\cdot,\cdot)\) is the beta function. This beta-binomial distribution is the predictive distribution for the remaining response count. Substituting the arm-specific beta-binomial probabilities into the sum gives an exact finite calculation of predictive success. The final success rule can also be another prespecified rule; the predictive averaging is the same, but the indicator \(G\) must represent that rule exactly.

Why the Beta-Binomial Is More Variable Than a Plug-In Binomial

Write \(s=a+b\) and \(\mu=a/(a+b)\) for the mean of the current beta posterior. Conditional on \(p\), \(Y\sim\operatorname{Binomial}(m,p)\). The law of total variance gives

For the last equality, a beta distribution has variance \(\mu(1-\mu)/(s+1)\), so \(E\{p(1-p)\}=\mu(1-\mu)-\operatorname{Var}(p)\). Compared with the plug-in binomial variance \(m\mu(1-\mu)\), the beta-binomial variance is larger when \(m\geq 2\) and the posterior has nonzero variance. For \(m=1\), the count is Bernoulli with probability \(\mu\), so the two predictive distributions agree. If \(m=0\), the future count is identically zero and its variance is zero; the variance ratio is not defined. In particular, the factor \((s+m)/(s+1)\) is at least one when \(m\geq1\), but it is less than one when \(m=0\), when it is not a variance multiplier for a nonzero binomial variance.

Worked Calculations With a Posterior-Mean Rule

The examples below use a different, deliberately simple final rule: declare success when the final posterior mean of \(p_T\) exceeds the final posterior mean of \(p_C\). This is a specified decision rule, not a synonym for requiring a particular posterior probability of superiority. It makes the predictive averaging transparent. In an actual trial analysis, use the rule that was selected for that trial.

Practical Checks, Alternatives, and Limitations

For modest remaining sample sizes, the beta-binomial probabilities can be enumerated and the final rule evaluated at every possible pair of counts. For larger problems, simulation draws response probabilities from their current posteriors, draws future counts conditional on those probabilities, applies the final analysis, and estimates predictive success by the fraction of simulated trials meeting the rule. Simulation approximates the same integral; it does not justify replacing an uncertain response rate by a fixed value.

Several checks help reveal errors or model mismatch:

• For each arm, verify that the predictive probabilities over \(y=0,\ldots,m\) sum to one and that the predictive mean is \(m a/(a+b)\).
• When \(m=1\), confirm that the beta-binomial prediction reduces to a Bernoulli distribution with response probability \(a/(a+b)\). When \(m=0\), the count is zero with probability one.
• Check that the final success indicator uses the stated rule, including whether its inequality is strict and which threshold applies.
• Compare predictions across plausible priors or posterior concentrations. A diffuse posterior can produce a substantially more variable future count than a concentrated posterior with the same mean.
• Check whether the assumed arm independence is defensible. If a shared parameter or hierarchical model induces dependence between \(p_T\) and \(p_C\), the joint predictive distribution does not factor into two beta-binomial probabilities; integrate over the joint posterior instead.

A plug-in binomial calculation is a convenient approximation, but it discards posterior uncertainty and may change the probability of crossing a final decision boundary. Direct enumeration is exact under the stated model, while simulation is flexible for complicated rules and models but has Monte Carlo error. Both approaches still depend on the likelihood, prior, and assumptions being reasonable. Neither predictive success nor a high probability of meeting the rule guarantees that the treatment effect is clinically important, that the model is correct, or that the trial procedure has acceptable performance across repeated trials.