Communicating Probability of Clinically Important Effects

From Favorable Effects to Clinically Important Effects

A treatment effect can favor the new treatment without being large enough to matter to patients or clinicians. For that reason, a report that gives only the posterior probability of any benefit may leave an important question unanswered: how probable is an effect large enough to be clinically important? The answer depends on a clinically justified threshold specified on the same scale as the treatment effect.

Let \(\theta\) denote a treatment effect oriented so that larger values favor the treatment. For example, if \(\theta\) is the mean improvement in a symptom score, positive values indicate greater improvement with treatment. Let \(\delta_{\mathrm{CI}}>0\) be a prespecified threshold for a clinically important effect. Both \(\theta\) and \(\delta_{\mathrm{CI}}\) must refer to the same estimand, population, time point, and measurement scale.

Definition: The posterior probability of any benefit is \(P(\theta>0\mid D)\). The posterior probability of a clinically important benefit is \(P(\theta>\delta_{\mathrm{CI}}\mid D)\), where \(D\) denotes the trial data and \(\delta_{\mathrm{CI}}>0\) is a specified clinical-importance threshold. These are probabilities under the stated statistical model and prior, conditional on the observed data.

The threshold is not estimated by the posterior probability calculation. It is a clinical interpretation boundary, supported by relevant clinical knowledge and stated in the protocol or analysis plan when possible. Its justification may depend on patient preferences, expected burdens and benefits, the endpoint’s measurement properties, and the decision being considered. A threshold should not be selected after examining the posterior merely because it produces a more favorable probability.

If a reported treatment contrast is instead defined so that negative values favor treatment, first state that convention and express the clinically important event in the corresponding direction. For instance, the probability of important benefit might be \(P(\theta<-\delta_{\mathrm{CI}}\mid D)\). A clear direction prevents the reader from having to infer whether a positive estimate is favorable.

Why the Two Probabilities Differ

For \(\delta_{\mathrm{CI}}>0\), the event \(\theta>\delta_{\mathrm{CI}}\) is contained within the event \(\theta>0\). Any clinically important benefit is a benefit, but a benefit need not be clinically important. The gap between the probabilities measures posterior probability assigned to positive effects that fall short of the specified threshold.

P(\theta>0\mid D) = P(0<\theta\leq\delta_{\mathrm{CI}}\mid D) + P(\theta>\delta_{\mathrm{CI}}\mid D).

This identity follows because the two disjoint events \(0<\theta\leq\delta_{\mathrm{CI}}\) and \(\theta>\delta_{\mathrm{CI}}\) together make up \(\theta>0\). Therefore, subtracting the clinically important probability from the any-benefit probability gives \(P(0<\theta\leq\delta_{\mathrm{CI}}\mid D)\). This is a useful check on calculations and a direct way to explain why the results differ.

The probabilities depend on the posterior distribution, which in turn depends on the likelihood, prior, and analysis assumptions. They do not by themselves establish that a particular treatment decision is appropriate. A decision also involves the consequences of treatment and no treatment, including harms, costs, and uncertainty in the clinical threshold.

Worked Example: A Normal Posterior and a Prespecified Threshold

Suppose a hypothetical randomized trial estimates a benefit-oriented symptom improvement, in points, with posterior distribution \(\theta\mid D\sim N(0.30,0.20^2)\). Assume the clinically important threshold was set in advance at \(\delta_{\mathrm{CI}}=0.25\) points. The normal approximation is assumed to describe the posterior adequately, and the threshold and posterior refer to the same population and follow-up time.

For a normal posterior with mean \(m\) and standard deviation \(s\), standardizing gives \(P(\theta>a\mid D)=1-\Phi((a-m)/s)\), where \(\Phi\) is the standard normal cumulative distribution function. Equivalently, this is \(\Phi((m-a)/s)\).

For any benefit, use \(a=0\), \(m=0.30\), and \(s=0.20\):

P(\theta>0\mid D) = \Phi\!\left(\frac{0.30-0}{0.20}\right) = \Phi(1.50) \approx 0.9332.

For clinically important benefit, use \(a=0.25\):

P(\theta>0.25\mid D) = \Phi\!\left(\frac{0.30-0.25}{0.20}\right) = \Phi(0.25) \approx 0.5987.

Thus, under this posterior, there is about a 93.3% probability of some symptom improvement and about a 59.9% probability that improvement exceeds 0.25 points. The difference, \(0.9332-0.5987=0.3345\), is the posterior probability of a positive improvement no greater than 0.25 points. The first result should not be presented as if it implied a similarly high probability of clinically important improvement.

Calculating and Reporting the Probability

When the posterior distribution has a convenient analytic form, calculate the probability by integrating its density beyond the threshold. For a posterior density \(p(\theta\mid D)\), the general expression is:

P(\theta>\delta_{\mathrm{CI}}\mid D) = \int_{\delta_{\mathrm{CI}}}^{\infty}p(\theta\mid D)\,d\theta.

With simulation-based inference, the same probability can be estimated from posterior draws. If \(\theta^{(1)},\ldots,\theta^{(M)}\) are draws from the posterior, count the draws above the threshold and divide by the total number of draws. The draws must adequately represent the target posterior; dependence between Markov chain Monte Carlo draws affects the precision of this estimate.

Worked Example: A Clinically Important Threshold Above Zero

In a hypothetical trial, define \(\theta\) as the reduction in average weekly symptom burden, measured in points, with positive values favoring treatment. Suppose the posterior is \(N(1.8,1.0^2)\) points and the prespecified clinically important reduction is 2.5 points. Assume the normal posterior is an adequate summary for this estimand.

The probability of any benefit is:

P(\theta>0\mid D) = \Phi\!\left(\frac{1.8-0}{1.0}\right) = \Phi(1.8) \approx 0.9641.

The probability of a clinically important benefit is:

P(\theta>2.5\mid D) = \Phi\!\left(\frac{1.8-2.5}{1.0}\right) = \Phi(-0.70) \approx 0.2420.

The posterior assigns a high probability, about 96.4%, to some reduction in symptom burden, but only about 24.2% to a reduction exceeding 2.5 points. This does not mean that treatment has a 24.2% probability of being effective in every sense; it is the posterior probability of this particular threshold event. A concise report would name the endpoint, units, time point, threshold, and both probabilities if the distinction is clinically relevant.

Worked Example: Estimating a Threshold Probability From Posterior Draws

Consider a hypothetical analysis of a benefit-oriented functional score, measured in points, with a prespecified clinically important threshold of 4 points. Suppose 10,000 adequately mixed posterior draws are available. Of these, 9,400 are greater than zero and 6,100 are greater than 4 points. Assume these draws represent the posterior for the same estimand and time point.

The Monte Carlo estimates are the respective fractions of draws meeting each condition:

\widehat{P}(\theta>0\mid D)=\frac{9400}{10000}=0.940, \qquad \widehat{P}(\theta>4\mid D)=\frac{6100}{10000}=0.610.

The estimated posterior probability of a positive effect that does not exceed 4 points is \(0.940-0.610=0.330\). If the draws were independent, the approximate Monte Carlo standard error for the second estimated probability would be:

\sqrt{\frac{0.610(1-0.610)}{10000}} \approx 0.00488.

This standard error describes simulation precision, not uncertainty about the treatment effect itself. With correlated draws, an effective sample size rather than the raw draw count is relevant to Monte Carlo precision. A report should not imply that many stored draws guarantee an accurate tail estimate: mixing and effective sample size in the threshold region also matter.

Assumptions, Diagnostics, and Common Pitfalls

A threshold probability is meaningful only when the underlying quantity is well defined. The treatment effect must identify the comparison, endpoint, population, and assessment time; the clinical threshold must be expressed on that same scale. The posterior calculation also inherits the model and prior assumptions used to obtain the posterior. If these choices are materially uncertain, examine whether the threshold probability changes under reasonable alternative specifications.

For analytic calculations, check that the posterior distribution and its scale parameter have been identified correctly. In particular, \(N(m,s^2)\) uses \(s\) as the standard deviation, not the variance. For simulated calculations, verify chain mixing, convergence, effective sample size, and stability of the estimated probability. A tail probability may be less stable than a posterior mean when few effective draws fall beyond the threshold.

A further diagnostic is coherence: for a benefit-oriented effect and a positive threshold, the calculated probability of clinically important benefit cannot exceed the probability of any benefit. If it does, check the sign convention, threshold, event definition, or arithmetic. When the threshold is close to zero, the two probabilities may be similar; when it is farther into the favorable tail, the clinically important probability will generally be smaller.

Common pitfall: Reporting “the probability of benefit is 93%” and treating that as “a 93% probability of clinically important benefit.” The first statement concerns \(\theta>0\); the second concerns \(\theta>\delta_{\mathrm{CI}}\). Unless the threshold is zero, these are different events and can have substantially different posterior probabilities.

The probability also changes when the threshold changes. That is not a defect in the calculation: it reflects that “clinically important” is defined relative to a chosen boundary. Report the threshold and its units rather than giving a percentage without its event definition. If there is genuine uncertainty about the appropriate threshold, show how the probability varies across clinically defensible values, while clearly distinguishing that sensitivity analysis from a single prespecified primary threshold.

How This Summary Complements Other Results

A posterior mean or median describes the center of the posterior; it does not show how much posterior mass lies beyond a clinical threshold. A credible interval describes a range of values supported by the posterior at a stated level; it may help readers see the location and spread of plausible effects, but it does not replace a direct probability for the threshold event. Conversely, a single threshold probability does not describe the full posterior shape.

The probability of any benefit answers whether the effect is favorable under the chosen sign convention. The clinically important probability answers whether it exceeds a particular boundary. These are complementary summaries, not competing estimates. Reporting both can be especially informative when the posterior favors some benefit strongly but assigns much less probability to effects large enough to matter.

A practical report should state the effect direction, clinical threshold and units, posterior probability beyond that threshold, and the endpoint and time point to which the estimand refers. If the probability is estimated by simulation, assess whether computation is sufficiently precise for the reported digits. Avoid presenting a probability as a stand-alone verdict: it is conditional on the data, model, prior, and specified threshold, and it does not alone encode the consequences of alternative treatment decisions.

Check Your Understanding

For each question, identify the event being summarized and explain its clinical interpretation.

A benefit-oriented effect has \(P(\theta>0\mid D)=0.88\) and \(P(\theta>3\mid D)=0.42\), where 3 points is the prespecified threshold. What is the posterior probability of a positive effect no greater than 3 points?
For a posterior \(N(2,1^2)\), calculate the probability of exceeding a clinically important threshold of 1 point.
A report says, “There is a 90% probability of benefit,” but the clinical threshold is 2 units. What information is missing to establish the probability of clinically important benefit?
For the same benefit-oriented posterior and a positive threshold, could the probability of clinically important benefit be larger than the probability of any benefit? Explain.

Solutions. (1) The events partition as in the identity above, so the probability is \(0.88-0.42=0.46\), or 46%, for \(0<\theta\leq3\). (2) Standardize: \(P(\theta>1\mid D)=\Phi((2-1)/1)=\Phi(1)\approx0.8413\), or about 84.1%. (3) The report must give \(P(\theta>2\mid D)\), with the effect direction, units, and relevant endpoint and time point stated; the probability of \(\theta>0\) does not determine it. (4) No. Since \(\{\theta>\delta_{\mathrm{CI}}\}\) is contained in \(\{\theta>0\}\) for \(\delta_{\mathrm{CI}}>0\), its posterior probability cannot be larger.