Posterior Probability of Clinically Important Harm

Why Define Harm on an Explicit Scale?

A treatment can have an estimated effect that is unfavorable without that effect being large enough to matter clinically. Conversely, an estimate close to zero may still leave meaningful posterior probability that the true effect exceeds a harm threshold. A posterior probability of clinically important harm addresses this directly: given the model, prior, and observed data, how much posterior probability lies in the region designated as important harm?

The direction of the effect must be specified before calculating that probability. In this tutorial, \(\theta_j\) is the effect for treatment arm \(j\) relative to the shared control, measured on a clinical scale oriented so that larger values indicate a worse outcome. Thus positive values are unfavorable, while negative values are favorable. For example, if the outcome is a symptom-burden score, \(\theta_j\) might be the treatment-minus-control difference in score points.

Definition: A clinically important harm threshold, denoted \(\delta_H\), is a prespecified effect value on the chosen scale such that effects at or above \(\delta_H\) count as clinically important harm. For arm \(j\), the posterior probability of clinically important harm is \(P(\theta_j \geq \delta_H\mid D)\), where \(D\) denotes the observed data. Under a continuous posterior distribution, using \(>\) rather than \(\geq\) gives the same probability.

A threshold is not automatically zero. Zero separates effects with opposite directions on this scale, but it does not distinguish small from clinically important worsening. The threshold should be chosen using clinical reasoning and stated in the units of the effect. In a multi-arm trial, a common threshold can be used when effects have the same meaning and scale across arms. If outcomes or scales differ, their thresholds need not be the same.

Calculating a Posterior Harm Probability

Suppose the marginal posterior distribution for arm \(j\)'s effect is normal, \(\theta_j\mid D\sim N(m_j,v_j)\), where \(m_j\) is the posterior mean and \(v_j\) the posterior variance. Write \(s_j=\sqrt{v_j}\) for the posterior standard deviation. Standardizing the threshold gives a normal-tail calculation:

P(\theta_j\geq\delta_H\mid D) =1-\Phi\left(\frac{\delta_H-m_j}{s_j}\right),

where \(\Phi\) is the cumulative distribution function of a standard normal random variable. The standardized threshold \((\delta_H-m_j)/s_j\) measures how many posterior standard deviations the harm boundary lies above the posterior mean. If the boundary is far above the mean, the harm probability is small; if it lies below the mean, the probability can be large.

This calculation uses the marginal posterior distribution for each arm. With a shared control, arm-specific contrasts can be posterior-correlated, but their marginal harm probabilities still follow from their marginal distributions. The correlations matter for probabilities involving several arms simultaneously; separate marginal probabilities must not be presented as such a joint probability.

Worked Example: Harm and Benefit Thresholds on One Scale

Suppose a hypothetical arm has posterior effect \(\theta\mid D\sim N(-0.5,0.7^2)\), measured in symptom-score points, with larger values worse. The prespecified harm threshold is \(\delta_H=0.6\) points. A clinically meaningful benefit threshold is \(\delta_B=-1.2\) points. Calculate the posterior probabilities of harm, benefit, and neither threshold region.

Clinically important harm. The posterior standard deviation is \(0.7\) points, so

P(\theta\geq 0.6\mid D) =1-\Phi\left(\frac{0.6-(-0.5)}{0.7}\right) =1-\Phi(1.5714) \approx 0.05804.

Thus the posterior probability of an effect at least \(0.6\) symptom-score points worse than control is about \(5.804\%\).

Clinically meaningful benefit. Benefit is the lower-tail event \(\theta\leq-1.2\):

P(\theta\leq-1.2\mid D) =\Phi\left(\frac{-1.2-(-0.5)}{0.7}\right) =\Phi(-1) \approx 0.15866.

Between the thresholds. Since the benefit and harm regions do not overlap, the probability of neither clinically meaningful benefit nor clinically important harm is

P(-1.2<\theta<0.6\mid D) =\Phi(1.5714)-\Phi(-1) \approx 0.94196-0.15866 =0.78330.

The three probabilities sum to approximately \(0.15866+0.78330+0.05804=1.00000\). This example also shows why harm and benefit thresholds should not be assumed to be equal in magnitude: clinical judgments may define different amounts of worsening and improvement as important.

Multiple Arms and Arm-Specific Results

For \(K\) treatment arms, define a harm event for each arm using its own threshold, \(H_j=\{\theta_j\geq\delta_{H,j}\}\). The corresponding posterior probability is calculated from that arm's marginal posterior. A table of these probabilities describes the posterior evidence about harm in each arm separately; it does not by itself describe the probability that at least one arm is harmful or that all arms are harmful.

Worked Example: Comparing Harm Probabilities Across Three Arms

In a hypothetical trial, three treatments are compared with a shared control on an adverse symptom-burden scale, in which larger treatment-minus-control differences are worse. For illustration, suppose the fitted marginal posteriors and common clinically important harm threshold are:

Arm	Marginal posterior	Harm threshold
A	\(N(0.10,0.25^2)\)	0.50 points
B	\(N(0.80,0.40^2)\)	0.50 points
C	\(N(-0.20,0.30^2)\)	0.50 points

The posterior summaries are assumed to come from an analysis appropriate to the trial's outcome and design. The same prespecified threshold of \(0.50\) points is used for all three arms because the effect scale and clinical interpretation are common.

For arm A,

P(\theta_A\geq0.50\mid D) =1-\Phi\left(\frac{0.50-0.10}{0.25}\right) =1-\Phi(1.60) \approx0.0548.

For arm B,

P(\theta_B\geq0.50\mid D) =1-\Phi\left(\frac{0.50-0.80}{0.40}\right) =1-\Phi(-0.75) =\Phi(0.75) \approx0.7734.

For arm C,

P(\theta_C\geq0.50\mid D) =1-\Phi\left(\frac{0.50-(-0.20)}{0.30}\right) =1-\Phi(2.3333) \approx0.0098.

The posterior probability of clinically important harm is largest for arm B, about \(77.34\%\), and smallest for arm C, about \(0.98\%\). These are arm-specific posterior probabilities, not error rates and not probabilities that an arm is harmful in a frequentist sense. Because the arms share a control, their posteriors may be correlated; one cannot add these three values to obtain the probability that at least one arm causes important harm.

Justification From a Normal-Normal Model

The tail calculation follows directly from standardization. If \(\theta_j\mid D\sim N(m_j,s_j^2)\), then \(Z=(\theta_j-m_j)/s_j\) has a standard normal posterior distribution. The event \(\theta_j\geq\delta_H\) is equivalent to \(Z\geq(\delta_H-m_j)/s_j\). Therefore its probability is the upper standard normal tail, \(1-\Phi((\delta_H-m_j)/s_j)\).

One common source of such a posterior is a normal likelihood with known standard error and a normal prior. Suppose an estimate \(Y\), conditional on the true arm effect, satisfies \(Y\mid\theta\sim N(\theta,SE^2)\), and the prior is \(\theta\sim N(m_0,\tau^2)\). The posterior is normal, with variance and mean

v=\left(\frac{1}{\tau^2}+\frac{1}{SE^2}\right)^{-1}, \qquad m=v\left(\frac{m_0}{\tau^2}+\frac{Y}{SE^2}\right).

These formulas require the stated normal likelihood, known sampling standard error, and normal prior. For a multi-arm analysis, shared-control structure may affect the joint posterior covariance; it does not invalidate the marginal calculation when the arm's marginal posterior mean and variance have been correctly obtained.

Worked Example: Obtaining a Harm Probability From a Prior and Estimate

Suppose a hypothetical treatment-minus-control estimate on a worsening-oriented symptom scale is \(Y=0.35\) points, with known standard error \(SE=0.30\) points. Use a prior \(\theta\sim N(0,0.60^2)\) and a clinically important harm threshold of \(0.80\) points. Assume the normal likelihood and prior just stated.

The posterior variance is

v=\left(\frac{1}{0.60^2}+\frac{1}{0.30^2}\right)^{-1} =\left(2.7778+11.1111\right)^{-1} =0.07200,

so the posterior standard deviation is \(s=\sqrt{0.07200}=0.2683\) points. The posterior mean is

m=0.07200\left(\frac{0}{0.60^2}+\frac{0.35}{0.30^2}\right) =0.07200(3.8889) =0.2800\text{ points}.

The posterior harm probability is consequently

P(\theta\geq0.80\mid Y) =1-\Phi\left(\frac{0.80-0.2800}{0.2683}\right) =1-\Phi(1.938) \approx0.0263.

Under this model and prior, about \(2.63\%\) of the posterior distribution lies at or above the harm threshold. This is not the probability that the model is correct; it is conditional on the specified model, prior, and observed estimate.

Assumptions, Diagnostics, and Alternatives

A harm probability is meaningful only if the event and the posterior distribution are meaningful. Before calculation, verify the following:

Effect direction: positive values really do represent worsening for the outcome being analyzed. If the scale instead increases with health or function, reverse the definition or the inequality.
Threshold scale and units: \(\delta_H\) is expressed on the same scale as \(\theta_j\), and its clinical rationale is established independently of a convenient numerical result.
Posterior model: the likelihood, prior, missing-data handling, and treatment-control contrast are appropriate for the data and design. Normal-tail calculations require a normal marginal posterior, or a justified normal approximation.
Arm-specific posterior summaries: each mean and variance used belongs to the correct arm and contrast. Shared-control analyses may induce dependence even when each marginal posterior is normal.

Useful diagnostics include checking the posterior density around \(\delta_H\), verifying the direction of the standardized threshold, and comparing analytic tail probabilities with numerical integration or simulation when the posterior is not normal. A probability near zero or one deserves particular scrutiny: it may reflect strong evidence, but it can also expose a reversed scale, a unit mismatch, or an overly restrictive model. Sensitivity to reasonable prior choices is also important when data are sparse; the resulting probability can be prior-sensitive even when the arithmetic is exact.

The posterior mean alone is not a substitute for a threshold probability. Two arms can have the same posterior mean but different uncertainty and therefore different probabilities of exceeding the harm boundary. A credible interval can show the range of posterior values and whether it crosses the threshold, but the threshold probability states directly how much posterior mass is on the harm side. A frequentist confidence interval has a different repeated-sampling interpretation and should not be described as a posterior probability.

These probabilities also do not, on their own, specify a treatment decision. They do not account for the severity of harm relative to benefit, treatment burden, or other decision consequences. Nor do separate arm-specific probabilities answer a joint question across arms. Those are distinct targets and require their own definitions and calculations.

Common Mistakes

Calling any positive effect harm. A positive effect is unfavorable under the orientation used here, but it is clinically important harm only if it reaches \(\delta_H\).
Using the wrong tail. With larger \(\theta_j\) meaning worse outcomes, harm is the upper tail. If larger values mean improvement, the event direction must change.
Assuming benefit and harm thresholds are symmetric. Clinical importance may differ by direction; define each threshold on its own rationale.
Adding arm-specific probabilities. Marginal probabilities cannot generally be added to obtain a probability of any or all arms being harmful, especially with shared-control dependence.
Rounding too early. Calculate the probability from unrounded standardized values and round only the final result. For example, the interval probability in the first worked example is approximately \(0.78330\), or \(0.7833\), when calculated using the unrounded normal tails.

Key takeaway: State the effect orientation and clinically important harm threshold first. Then calculate the posterior mass beyond that boundary from the correct arm-specific posterior, and interpret it as conditional on the model, prior, and data.

Check Your Understanding

Use the worsening-oriented effect scale throughout: larger values indicate worse outcomes. Solutions follow the questions.

If \(\theta\mid D\sim N(0.2,0.5^2)\) and \(\delta_H=0.7\), what is the posterior probability of clinically important harm?
An arm has posterior probability \(0.35\) of exceeding its harm threshold. State the interpretation without describing it as a probability that the null hypothesis is true.
Why can the posterior probability of harm for each of two arms not generally be added to obtain the probability that at least one arm is harmful?
If a benefit threshold is \(-0.9\) and a harm threshold is \(0.4\), what region represents neither clinically meaningful benefit nor clinically important harm?

Solutions. (1) The standardized threshold is \((0.7-0.2)/0.5=1\), so \(P(\theta\geq0.7\mid D)=1-\Phi(1)\approx0.1587\). (2) Given the specified model, prior, and observed data, \(35\%\) of the posterior distribution for the arm's effect is at or above the clinically important harm threshold. (3) The harm events may be dependent, particularly when arms share a control; adding marginal probabilities ignores their overlap and dependence. A joint calculation is needed. (4) The neither region is \(-0.9<\theta<0.4\); its posterior probability is the mass between those two thresholds.