Probability That Benefits Exceed Harms

From an Average Trade-Off to a Probability

A treatment can have positive expected net clinical benefit while still having substantial uncertainty about whether its benefits outweigh its harms. For a development decision, that distinction matters: the average trade-off describes the center of the uncertainty distribution, while the probability of a positive trade-off describes how often the net benefit is favorable under that distribution. Neither quantity alone summarizes the full decision.

The calculation must preserve the joint uncertainty in benefits and harms. If a treatment’s benefit and harm contributions tend to move together, treating them as independent can change the estimated probability that benefits exceed harms. The result also depends on what the uncertain quantities represent: uncertainty about average treatment effects is not the same as variability in outcomes among individual patients.

For a comparison with a treatment alternative, \(B\) and \(H\) should represent the benefit and harm contributions for that comparison, using the same direction and preference weights as the net clinical benefit model. The zero threshold is meaningful only after those choices have been made. If the decision instead requires a clinically meaningful margin \(c\), the relevant probability is \(P(N>c)\), not \(P(N>0)\).

The probability distribution also needs a clear interpretation. For example, it may represent posterior uncertainty about treatment-level benefit and harm summaries, conditional on the available evidence and model. If it represents future individual outcomes instead, the probability answers a different question. In either case, state the target, the evidence or conditioning information, and which sources of uncertainty are included. The earlier material on joint uncertainty in benefits and harms motivates retaining the dependence between \(B\) and \(H\); the calculation here uses that joint distribution to evaluate the sign of net benefit.

Estimation From Joint Uncertainty

A direct approach is to draw paired values \((B_m,H_m)\) from the specified joint distribution, calculate \(N_m=B_m-H_m\) for every draw, and count the proportion for which \(N_m>0\). Pairing matters: each benefit draw must remain with its corresponding harm draw. Shuffling or independently resampling the two columns changes their dependence and can therefore change the result.

Here, \(M\) is the number of joint draws and \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. This proportion estimates the probability under the distribution from which the draws were generated. With a finite number of simulation draws, it also has Monte Carlo error. That error reflects numerical estimation from a finite simulation, not necessarily all uncertainty in the underlying evidence or model.

A convenient parametric result applies when \(B\) and \(H\) have a joint bivariate normal distribution. Write their means as \(\mu_B\) and \(\mu_H\), standard deviations as \(\sigma_B\) and \(\sigma_H\), and covariance as \(\operatorname{Cov}(B,H)\). Then \(N=B-H\) is normally distributed, with mean equal to the difference in means and variance determined by both marginal variances and their covariance.

If \(\sigma_N>0\), standardizing \(N\) gives \((N-\mu_N)/\sigma_N\sim N(0,1)\). Therefore, with \(\Phi\) denoting the standard normal cumulative distribution function,

This formula requires the normal model for the joint distribution, not merely normal-looking marginal distributions considered separately. If \(\sigma_N=0\), net benefit is constant under the model: the probability is 1 if that constant is positive and 0 otherwise. If there is probability mass exactly at zero, the strict event \(N>0\) excludes that mass; it should not be silently counted as favorable.

Worked Examples

Why Expected Net Benefit Is Not the Same Quantity

Expected net benefit is \(E(N)\), the probability-weighted average of net benefit. The probability of positive net benefit is \(P(N>0)\), the probability mass on one side of a threshold. A mean does not determine how probability is distributed around that threshold. For example, a net benefit that is always 0.020 utility points has mean 0.020 and probability of being positive equal to 1. A different distribution that gives net benefit 0.100 with probability 0.60 and −0.100 with probability 0.40 has the same mean, \(0.60(0.100)+0.40(-0.100)=0.020\), but its probability of being positive is only 0.60.

The normal result gives another way to see the distinction: for a normal net benefit distribution, \(P(N>0)=\Phi(\mu_N/\sigma_N)\). The probability depends on the mean relative to the standard deviation, not on the mean alone. A positive expected net benefit can coexist with a probability below one-half if the distribution is sufficiently skewed; under a normal model specifically, that cannot happen because the normal distribution is symmetric and the sign of the mean determines whether the probability is above or below one-half.

These measures support different questions. Expected net benefit summarizes average utility on the chosen scale. The probability of positive net benefit summarizes the chance, under the chosen uncertainty distribution, that the sign is favorable. A decision-maker may consider both, along with whether the threshold is appropriate and the consequences of unfavorable outcomes. Neither statistic by itself establishes that treatment causes a particular outcome for an individual patient.

Assumptions, Diagnostics, and Limitations

Before reporting a probability, check that the benefit and harm components are defined consistently, that the utility weights and direction of the score are explicit, and that their joint uncertainty distribution represents the intended target. Check that paired simulations preserve dependence. For a normal calculation, verify that the joint normal approximation is defensible and that the resulting variance is nonnegative. An impossible negative variance usually signals an arithmetic error or an inconsistent covariance specification.

For simulation-based estimates, inspect the number of draws, Monte Carlo error, and the distribution of \(N\), especially near zero. If only a small number of draws are positive or negative, the estimated probability can be sensitive to a few draws. Check whether zero values occur and state how ties are handled. Compare the estimated probability across reasonable model specifications or uncertainty inputs when those choices are consequential; a precise simulation estimate from one questionable model is not evidence that the probability itself is known precisely.

The principal alternative is expected net benefit, which emphasizes average utility rather than the proportion above a threshold. A further alternative is a probability of exceeding a prespecified positive margin \(c\), which asks whether net benefit is not merely favorable but larger than that margin. These summaries are not interchangeable: the threshold must be chosen for a clinical or decision reason, and moving it changes the probability. Reporting a mean, a probability, and—when relevant—a clinically motivated threshold can make the trade-off more transparent than relying on one summary.

The probability remains conditional on the utility model, the evidence, and the uncertainty included. It does not resolve uncertainty about whether the chosen preferences represent all relevant stakeholders, whether the outcomes or weights omit an important consequence, or whether the model is appropriate. A probability based on treatment-level parameter uncertainty also should not be described as the proportion of individual patients who will benefit; that interpretation requires a different target distribution.