Worked Decision Analysis for a Binary Superiority Trial

From Response Rates to a Decision

A binary superiority trial records whether each participant experiences a defined response. The statistical question is not only whether the treatment response rate exceeds the control rate, but whether the difference is large enough to matter clinically. A decision rule makes that distinction explicit: it names a minimum worthwhile increase and states how much posterior evidence is required before recommending a go decision.

The worked analyses below use the absolute response-rate difference, measured in percentage points. The method depends on defining the response consistently, selecting a clinically justified margin before examining trial outcomes, and specifying the probability threshold in advance. These choices are part of the decision policy, not quantities to tune after seeing which choice produces a preferred answer.

Definition: Let \(p_T\) and \(p_C\) be the response probabilities for participants assigned to treatment and control. For a prespecified clinically meaningful margin \(\delta\), the treatment has a clinically meaningful response-rate advantage when \(p_T-p_C>\delta\). The posterior success probability is \(P(p_T-p_C>\delta\mid\text{observed data})\). A go/no-go rule compares this probability with a prespecified cutoff \(q\): go if it is at least \(q\), and no-go otherwise.

For example, \(\delta=0.05\) means that the treatment response rate must exceed the control response rate by more than five percentage points. It does not mean a five percent relative increase. If the control response rate is 30%, a five-percentage-point advantage corresponds to a treatment response rate above 35%, whereas a five percent relative increase would correspond to 31.5%.

The margin should reflect clinical benefit, burden, safety, and the consequences of adopting an ineffective treatment. The probability cutoff \(q\) is a separate policy choice. Neither is automatically determined by the observed response counts.

Binary Response Model and Posterior Probability

Suppose there are \(n_T\) treatment participants, of whom \(x_T\) respond, and \(n_C\) control participants, of whom \(x_C\) respond. Use a separate binomial model for each randomized arm: conditional on its response probability, each participant’s response is a Bernoulli outcome, and responses are independent within and between arms. In this analysis, the response probabilities are treated as constant within each arm; the model does not adjust for covariates or account for clustering.

For a transparent worked analysis, assign independent uniform priors, \(p_T\sim\operatorname{Beta}(1,1)\) and \(p_C\sim\operatorname{Beta}(1,1)\). The uniform prior gives equal density to response probabilities between zero and one. It is proper, but it is not a guarantee of negligible prior influence in every trial; prior sensitivity is important when data are limited.

The binomial likelihood for one arm is proportional to \(p^x(1-p)^{n-x}\). Multiplication by a \(\operatorname{Beta}(a,b)\) prior gives a posterior with updated shape parameters \(a+x\) and \(b+n-x\). Thus, under the uniform priors used here:

p_T\mid\text{data}\sim\operatorname{Beta}(1+x_T,1+n_T-x_T), \qquad p_C\mid\text{data}\sim\operatorname{Beta}(1+x_C,1+n_C-x_C).

Because the priors and arm-specific likelihoods factor into separate treatment and control components, the two posterior response probabilities are independent under this model. The probability of exceeding a nonzero margin can therefore be obtained by integrating one posterior distribution against the other:

P(p_T-p_C>\delta\mid\text{data}) = \int_{\delta}^{1} f_T(t\mid\text{data})F_C(t-\delta\mid\text{data})\,dt,

where \(f_T\) is the treatment posterior density and \(F_C\) is the control posterior cumulative distribution function. For \(t\leq\delta\), the treatment probability cannot exceed the control probability by more than \(\delta\), so that region contributes zero. The integral is usually evaluated by numerical integration or posterior simulation. For the examples, a normal approximation to each beta posterior provides a hand-checkable calculation; the reported probabilities are therefore approximate, not exact beta-integral values.

Approximation used in the examples: A \(\operatorname{Beta}(a,b)\) posterior has mean \(a/(a+b)\) and variance \(ab/[(a+b)^2(a+b+1)]\). When the posterior distributions are sufficiently concentrated and not strongly affected by the boundaries at zero and one, approximate the difference by a normal distribution with mean equal to the difference of posterior means and variance equal to the sum of the posterior variances. Then \(P(p_T-p_C>\delta\mid\text{data})\) is approximated by \(1-\Phi\big((\delta-m_D)/s_D\big)\), where \(m_D\) and \(s_D\) are the approximate mean and standard deviation of the difference.

Worked Decision Analyses

Worked Example: A Go Decision With Strong Evidence

Suppose the trial team has justified a five-percentage-point minimum worthwhile improvement and prespecified \(q=0.95\). In hypothetical data, 63 of 100 treatment participants and 38 of 100 control participants respond. The model assumes independent binary responses within each arm and independent uniform priors as specified above.

The posteriors are \(p_T\mid\text{data}\sim\operatorname{Beta}(64,38)\) and \(p_C\mid\text{data}\sim\operatorname{Beta}(39,63)\). Their posterior means are \(64/102=0.6275\) and \(39/102=0.3824\), giving a mean difference of \(0.2451\), or 24.51 percentage points.

For the treatment posterior, the variance is \(64(38)/(102^2\cdot103)=0.002269\). For control, it is \(39(63)/(102^2\cdot103)=0.002293\). The approximate standard deviation of the difference is therefore \(\sqrt{0.002269+0.002293}=0.06754\). The standardized distance above the margin is \((0.2451-0.05)/0.06754=2.888\). Thus the approximate posterior success probability is \(\Phi(2.888)=0.9981\).

Since \(0.9981\geq0.95\), the prespecified rule gives a go decision. Under this model and approximation, there is about a 99.81% posterior probability that the treatment response rate exceeds the control rate by more than five percentage points. This probability is conditional on the chosen model, priors, margin, and observed data.

Worked Example: Favorable Difference, but No-Go

Keep the same prespecified margin of \(0.05\), cutoff \(q=0.95\), model, and priors. Consider different hypothetical data: 54 responses among 100 treatment participants and 45 among 100 control participants. Both arms have the same sample size, and the response definition and independence assumptions are unchanged.

The posteriors are \(\operatorname{Beta}(55,47)\) and \(\operatorname{Beta}(46,56)\). Their means are \(55/102=0.5392\) and \(46/102=0.4510\); the posterior mean difference is \(0.08824\). The posterior variances are \(55(47)/(102^2\cdot103)=0.002412\) and \(46(56)/(102^2\cdot103)=0.002404\). The difference has approximate standard deviation \(\sqrt{0.002412+0.002404}=0.06940\).

The distance from its mean to the success boundary is \(0.08824-0.05=0.03824\), or \(0.03824/0.06940=0.551\) standard deviations. The approximate probability of exceeding the margin is \(\Phi(0.551)=0.709\). The observed response rate is higher in the treatment arm, but \(0.709<0.95\), so the rule gives a no-go decision.

This is not a conclusion that the treatments are equal. It says that these data do not meet the chosen standard for proceeding on the basis of a clinically meaningful advantage. The probability of any positive difference and the probability of exceeding the specified margin are distinct quantities.

Worked Example: Evidence Against the Targeted Advantage

Again use \(\delta=0.05\), \(q=0.95\), independent uniform priors, and the same binomial assumptions. In hypothetical data, 45 of 100 treatment participants and 47 of 100 control participants respond.

The posteriors are \(\operatorname{Beta}(46,56)\) for treatment and \(\operatorname{Beta}(48,54)\) for control. Their means are \(46/102=0.4510\) and \(48/102=0.4706\), so the mean difference is \(-0.01961\). The posterior variances are \(46(56)/(102^2\cdot103)=0.002404\) and \(48(54)/(102^2\cdot103)=0.002419\). The approximate standard deviation of the difference is \(\sqrt{0.002404+0.002419}=0.06945\).

The standardized distance from the mean to the margin is \((0.05-(-0.01961))/0.06945=1.002\). The approximate posterior success probability is \(1-\Phi(1.002)=0.158\). Since \(0.158<0.95\), the decision is no-go. The data put the posterior mean difference below zero, but the decision follows from the probability of exceeding the prespecified five-percentage-point margin, not from the sign of the sample difference alone.

Why the Decision Rule Matters

A margin prevents “superiority” from being treated as a single undifferentiated claim. If the clinical question is whether any positive difference exists, the event of interest is \(p_T-p_C>0\). If the decision requires a worthwhile difference, the event is \(p_T-p_C>\delta\) for a positive margin. A posterior probability for the first event cannot be substituted for the second.

The posterior success probability is also different from predictive success: it evaluates the target event given the data observed so far, whereas predictive success concerns the chance that a final decision criterion will be met after future observations. Here the decision is explicitly based on the posterior probability at the data being analyzed.

A probability cutoff such as 0.95 is not a frequentist significance level \(\alpha\), and it does not mean there is a 5% probability that the null hypothesis is true. It is a threshold in a posterior decision rule. The long-run false-go and false-no-go rates depend on the complete trial procedure and the true response rates; the posterior cutoff alone does not specify those rates.

The integral for a nonzero margin is straightforward to evaluate numerically. A practical analysis should check the calculation using a second method, such as posterior simulation, and verify that the numerical result is stable. If using the normal approximation, inspect whether the beta posteriors are concentrated away from zero and one. Very small samples, extreme response counts, or substantial posterior skew can make the approximation unreliable; use the beta integral or posterior simulation instead.

Assumptions, Alternatives, and Common Pitfalls

The model assumes a clearly defined binary endpoint, comparable assessment in both arms, and independent participant outcomes conditional on each arm’s response probability. It also assumes a single response probability per arm. Clustering by site, repeated observations, important covariates, or informative missing outcomes may require a different model. The uniform priors and independence of the arm-specific priors must also be appropriate for the intended analysis. With sparse data, reasonable alternative priors can materially change the posterior probability, so examine that sensitivity rather than treating the prior as irrelevant.

The margin and cutoff should be set before observing outcome data. Choosing a smaller margin after seeing results makes success easier by changing the clinical question; selecting a probability cutoff to force a desired decision undermines the rule. Record the endpoint definition, margin, prior, probability cutoff, and analysis population so the decision can be reproduced.

A frequentist confidence interval or hypothesis test can address a related treatment comparison, but it does not itself give the posterior probability that \(p_T-p_C\) exceeds the clinical margin. Conversely, a Bayesian go decision under this rule should not be described as proof that the treatment works or as a guarantee of benefit for an individual patient. The decision is conditional on a model and chosen policy, and clinical safety and other evidence remain relevant.

Mistake: “The treatment rate is higher, so the trial succeeds.” A positive observed difference does not show that the posterior probability of exceeding the clinical margin meets the cutoff. Calculate the probability for the specified event.
Mistake: “The posterior probability is 0.95, so the null has a 5% probability of being true.” The probability is about the parameter event \(p_T-p_C>\delta\), conditional on the model and data; it is not a probability assigned to a null hypothesis by this rule.
Mistake: “The margin is just a statistical tuning parameter.” The margin defines what counts as a worthwhile benefit and needs clinical justification independent of the observed result.

Key takeaway: A binary-trial go/no-go analysis requires three explicit ingredients: a clinically justified response-rate margin, a model that yields the posterior distribution of the arm-specific response probabilities, and a prespecified posterior probability cutoff. The decision concerns the probability of exceeding the margin, not merely which observed response rate is larger.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If the control response rate is 30% and the absolute margin is 0.05, what treatment response rate is the boundary for exceeding the margin? Is this a five percent relative increase?
With a posterior success probability of 0.93 and a prespecified cutoff of 0.95, what decision follows? What does that decision not establish?
Why can a positive observed treatment-control difference fail to produce a go decision?
Is a posterior probability cutoff of 0.95 the same as a frequentist significance level of 0.05? Explain.

Solutions. (1) The boundary is \(0.30+0.05=0.35\), or 35%; exceeding the margin requires a rate above 35%. It is not a five percent relative increase: a five percent relative increase from 30% would be \(0.30(1.05)=0.315\), or 31.5%. (2) Since \(0.93<0.95\), the rule gives no-go. This does not establish equal response rates or prove that the treatment has no benefit. (3) A positive observed difference may still leave substantial posterior probability below the clinically meaningful margin, so the probability of exceeding that margin can remain below the cutoff. (4) No. The posterior cutoff is a decision threshold for a conditional probability; a frequentist significance level defines a long-run Type I error probability under a null hypothesis.