Reporting Posterior Probability of Treatment Benefit

What Does “Probability of Treatment Benefit” Mean?

A report that a treatment “probably works” is incomplete unless it states what effect is being considered and which direction counts as benefit. For a continuous outcome, a larger value may be favorable, as with a function score, or unfavorable, as with symptom severity. The sign of a treatment contrast therefore has no interpretation until its definition and the outcome’s direction are stated.

In Bayesian analysis, the probability of benefit is calculated from the posterior distribution of a specified treatment effect, given the observed trial data and the statistical model. It is not a probability that the model is true, nor a probability that the null hypothesis is true. It summarizes uncertainty about the effect under the stated prior and likelihood.

Definition: Let \(\theta\) be a treatment-effect contrast, defined so that \(\theta>0\) represents benefit for the experimental arm. The posterior probability of benefit is \(P(\theta>0\mid y)\), where \(y\) denotes the observed trial data. The probability is conditional on those data and on the specified Bayesian model, including its prior and likelihood.

A useful report names the experimental and control arms, defines the contrast, gives the favorable direction, identifies the data conditioning, and reports the probability in a readable form. For example: “Under the specified model and prior, the posterior probability that the experimental-minus-control difference in the 12-week function score is greater than zero is 85.7%, given the observed trial data.” The units belong to the effect contrast—in this example, score points—not to the probability.

The estimand also needs to be unambiguous: specify the outcome, time point, population, and treatment comparison relevant to the analysis. A reader should not have to infer whether the statement concerns an endpoint at week 12 or week 24, or whether the reported contrast is experimental minus control or the reverse.

Choose the Contrast Before Reporting Its Probability

For an outcome where higher values are better, a common contrast is \(\theta=\mu_E-\mu_C\), where \(\mu_E\) and \(\mu_C\) are the experimental- and control-arm means. In that case, positive values favor the experimental arm, and the probability of benefit is \(P(\mu_E-\mu_C>0\mid y)\).

For an outcome where lower values are better, define the contrast in the favorable direction—for example, \(\theta=\mu_C-\mu_E\) for symptom severity. Then positive values still favor the experimental arm. Alternatively, one can retain the contrast \(\mu_E-\mu_C\) and calculate the probability that it is below zero. Either choice is valid, but the direction and sign must be explicit. Reversing the contrast without reversing the inequality reverses the meaning of the probability.

The following examples use hypothetical continuous-outcome trial summaries. Their normal likelihoods and priors are assumptions for illustration, not claims about any real study. Each example defines the effect so that a positive value favors the experimental arm.

Worked Example: Higher Scores Favor the Experimental Arm

Suppose a hypothetical trial compares an experimental treatment with control on a function score at week 12, for which higher values are better. Let \(\theta=\mu_E-\mu_C\), measured in score points. The estimated contrast is 2.0 points with a standard error of 1.5 points. Assume the summary estimate has likelihood \(d\mid\theta\sim N(\theta,1.5^2)\), and use the prior \(\theta\sim N(0,2^2)\). The prior is centered at no difference and has standard deviation 2 points.

These assumptions give a normal posterior. Its variance \(V\) and mean \(m\) are

V=\left(\frac{1}{4}+\frac{1}{2.25}\right)^{-1}=1.44,\qquad m=V\left(\frac{0}{4}+\frac{2.0}{2.25}\right)=1.28.

Thus \(\theta\mid y\sim N(1.28,1.44)\), with posterior standard deviation \(\sqrt{1.44}=1.20\) points. Since positive values favor the experimental arm, standardizing the threshold zero gives

P(\theta>0\mid y) =1-\Phi\left(\frac{0-1.28}{1.20}\right) =\Phi(1.0667) \approx 0.8569.

A contextual report is: “For the week-12 function-score difference, defined as experimental minus control, the posterior probability that the experimental arm has a higher mean score is 85.7%, given the observed trial data and the stated model and prior.” This is a probability about the contrast under the posterior; it does not establish that the treatment caused an improvement.

Derivation for a Normal Posterior

If the posterior for the benefit-oriented contrast is \(\theta\mid y\sim N(m,V)\), the probability of benefit follows by standardizing \(\theta\). This calculation requires a properly specified posterior distribution; the normal formula is not appropriate merely because a point estimate and standard error are available.

State the favorable direction.
For this derivation, assume larger \(\theta\) means greater benefit, so benefit corresponds to \(\theta>0\).

Standardize the posterior variable.
If \(\theta\mid y\sim N(m,V)\), then \(Z=(\theta-m)/\sqrt{V}\) has a standard normal posterior distribution.

Translate the event and evaluate it.
The event \(\theta>0\) is equivalent to \(Z>-m/\sqrt{V}\). Therefore \(P(\theta>0\mid y)=1-\Phi(-m/\sqrt{V})=\Phi(m/\sqrt{V})\), where \(\Phi\) is the standard normal cumulative distribution function.

This derivation also shows why the posterior mean alone is not the probability of benefit: the result depends on the posterior spread as well as the location. Two posteriors with the same positive mean can assign different probabilities to \(\theta>0\) if their variances differ.

Worked Example: Lower Symptom Scores Favor the Experimental Arm

In a hypothetical trial, symptom severity is measured in points, with lower scores better. Define \(\theta=\mu_C-\mu_E\), so positive values represent lower mean severity in the experimental arm. Suppose the observed contrast estimate is \(d=1.1\) points with standard error \(0.7\) points. Assume \(d\mid\theta\sim N(\theta,0.49)\) and prior \(\theta\sim N(0,2.25)\).

The posterior variance is \(V=(1/2.25+1/0.49)^{-1}=441/1096\approx0.402372\) points squared. The posterior mean, using the unrounded variance, is

m=\frac{441}{1096}\left(\frac{1.1}{0.49}\right) =\frac{990}{1096} \approx 0.9033\text{ points}.

The posterior standard deviation is \(\sqrt{0.402372}\approx0.6343\) points. Because \(\theta>0\) favors the experimental arm, the posterior probability of benefit is

P(\theta>0\mid y) =\Phi\left(\frac{0.9033}{0.6343}\right) =\Phi(1.424) \approx0.9227.

A suitable report is: “For mean symptom severity, the contrast was defined as control minus experimental; the posterior probability that the experimental arm has lower severity is 92.3%, given the observed trial data and the stated model and prior.” Stating “experimental minus control is positive” here would be wrong: that contrast has the opposite favorable sign.

Reporting a Probability Estimated from Posterior Draws

Not every posterior has a convenient closed-form distribution. With posterior simulation, the probability can be estimated by counting draws that satisfy the benefit condition. If \(\theta^{(1)},\ldots,\theta^{(M)}\) are independent draws from the posterior, then

\widehat{P}(\theta>0\mid y) =\frac{1}{M}\sum_{j=1}^{M} I(\theta^{(j)}>0),

where \(I(\theta^{(j)}>0)\) equals 1 when draw \(j\) is positive and 0 otherwise. This estimator is the proportion of posterior draws in the favorable region. It estimates the posterior probability; it is not itself the probability that the effect is positive.

Worked Example: Counting Favorable Posterior Draws

Suppose a hypothetical analysis defines \(\theta\) as the experimental-minus-control difference in a higher-is-better outcome. From 10,000 independent draws from the posterior conditional on the trial data, 8,420 draws have \(\theta>0\). The estimated posterior probability of benefit is

\widehat{P}(\theta>0\mid y)=\frac{8420}{10000}=0.842=84.2\%.

For independent draws, the Monte Carlo standard error of this estimated proportion is approximately

\sqrt{\frac{0.842(1-0.842)}{10000}} =\sqrt{0.0000133036} \approx0.00365.

That is about 0.365 percentage points of simulation error. A report might say: “The estimated posterior probability that the experimental-minus-control difference is positive is 84.2%, conditional on the observed trial data and the stated model and prior.” The Monte Carlo standard error describes numerical precision in estimating that posterior probability; it is not the posterior uncertainty about the treatment effect itself.

Checks, Alternatives, and Limitations

Before reporting the result, check that the contrast, sign, outcome direction, population, and time point match the analysis. For simulation-based estimates, confirm that draws adequately represent the posterior. If draws are correlated, the independent-draw standard-error formula above is not generally appropriate; use a Monte Carlo error assessment that accounts for dependence. A probability near 0 or 1 deserves particular attention to the direction of the inequality, computational accuracy, and whether the model supports the apparent concentration.

A posterior probability of benefit answers a directional question, while a credible interval describes a range of effect values under the posterior. These summaries are related but not interchangeable: the probability that an effect is positive does not provide the size or clinical meaning of the effect. A posterior mean or median summarizes location, but alone does not state the probability that the effect favors either arm. A frequentist p-value is a different quantity and should not be substituted for \(P(\theta>0\mid y)\).

The probability also depends on the chosen estimand, likelihood, prior, and data. It does not, by itself, establish that a treatment is clinically worthwhile, safe, or appropriate for every patient. Nor does a high probability resolve concerns about data quality, missing-data assumptions, or whether the trial population supports the intended claim. Those considerations remain part of interpretation.

Common pitfall: “There is an 85.7% probability the experimental treatment is better” is ambiguous if the outcome, contrast, direction, or conditioning is unstated. It may also be read as an unconditional claim about treatment performance. Instead, define the effect and say, for example, “Given the observed trial data and the stated model and prior, the posterior probability that the week-12 experimental-minus-control function-score difference is greater than zero is 85.7%.”

Key takeaway: A posterior probability of benefit is interpretable only after the effect contrast and favorable direction are explicit. Report it as conditional on the observed data and the specified Bayesian model, and do not treat it as a summary of effect magnitude or clinical importance.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A higher-is-better outcome is analyzed using \(\theta=\mu_E-\mu_C\). Which event represents benefit?
A lower-is-better outcome is analyzed with \(\theta=\mu_E-\mu_C\). Which event represents benefit?
If \(\theta\mid y\sim N(0.8,0.64)\) and positive values favor the experimental arm, calculate \(P(\theta>0\mid y)\).
In 5,000 independent posterior draws, 4,000 values of a benefit-oriented contrast are positive. What probability estimate should be reported?
Why is “the probability that the null hypothesis is true is 10%” not an appropriate restatement of a posterior probability of benefit equal to 90%?

Solutions. (1) Benefit is \(\theta>0\), or \(P(\theta>0\mid y)\). (2) Lower experimental values mean \(\mu_E-\mu_C<0\), so benefit is \(P(\theta<0\mid y)\); alternatively, reverse the contrast and use a positive threshold. (3) The posterior standard deviation is \(\sqrt{0.64}=0.8\), so \(P(\theta>0\mid y)=\Phi(0.8/0.8)=\Phi(1)\approx0.8413\), or 84.1%. (4) The estimate is \(4000/5000=0.800\), or 80.0%, conditional on the observed data and model. (5) A 90% probability that the specified contrast is positive is a posterior probability about the effect’s direction; it is not a probability assigned to a null hypothesis being true. The contrast and conditioning must still be stated.