Rare Severe Harms in Benefit-Risk Models

Why Rare Severe Harms Need a Tail Summary

An event can be uncommon enough to have little effect on the average utility of treatment, yet severe enough to matter greatly to patients and decision makers. A benefit-risk assessment based only on expected utility may therefore conceal an important feature of the outcome distribution: how poor the outcomes are in its lower tail. A tail summary does not replace expected utility. It complements it by making the frequency and severity of the least favorable outcomes more visible.

The calculations below treat utility as a numerical measure on a common scale, with larger values more preferred. The example utilities are hypothetical and are not estimates from a clinical study. Each utility applies to a specified patient and treatment horizon. Event probabilities must refer to that same population and horizon; a probability per dose, for example, cannot be used as though it were a probability over an entire treatment course.

For a discrete distribution, the lower quantile and expected shortfall answer different questions. The quantile identifies a cutoff utility, while expected shortfall averages the utilities within the worst specified fraction of outcomes. If fewer than \(\alpha\) of patients experience the worst outcome, the lower quantile may be a more favorable value even though the worst outcomes are still included in expected shortfall.

A Discrete Lower-Tail Calculation

Suppose the severe event has utility \(-L\) with probability \(p\), and all other patients have utility 0. Let \(Q(v)\) be the lower quantile function, so that \(Q(v)\) gives the utility at cumulative probability \(v\). If \(p<\alpha\), then \(Q(v)=-L\) for the first \(p\) of the distribution and \(Q(v)=0\) from \(p\) through \(\alpha\). Thus the worst \(\alpha\) fraction contains all severe events and enough of the zero-utility outcomes to fill the remainder. Its average is:

This expression also shows why a small event probability does not make severity irrelevant: the tail summary depends on the product \(Lp\), scaled by the chosen tail fraction. If \(p\geq\alpha\), the worst \(\alpha\) fraction consists entirely of severe-event outcomes, so \(ES_\alpha=-L\) in this two-outcome model.

Adding Benefits and Comparing Event-Rate Assumptions

In a benefit-risk model, the severe event is usually only one outcome among several. Its contribution to expected utility is probability weighted, just like every other outcome. The event rate should be varied transparently when its value is uncertain or when plausible assumptions are being compared. Keep the other outcome probabilities and utilities explicit so that a change in the summary can be traced to the assumption that changed.

Assumptions, Diagnostics, and Interpretation

These summaries are meaningful only when the model inputs are coherent. Check that the outcome categories are mutually exclusive and collectively exhaustive, and that their probabilities sum to 1. If benefit and harm can occur in the same patient, a model that lists them as separate exclusive categories may be wrong; represent their joint outcomes or use a utility model that handles their co-occurrence. The earlier material on correlated outcomes in benefit-risk assessment is relevant when dependence changes which combinations occur.

Utilities must also refer to the same scale, patient perspective, and time horizon. If severe harm is already incorporated in a composite outcome utility, adding a separate severe-event disutility can count the harm twice. Check whether the event probability is per patient, per exposure, or per treatment course, and align its denominator with the utility horizon. For a time-dependent event risk, the probability over the modeled horizon—not an unadjusted rate at a single time point—must be used.

A useful diagnostic is to calculate the probability mass function and cumulative distribution at the tail boundary, rather than inferring a quantile from the event rate alone. With discrete outcomes, a quantile can coincide with a common outcome even when rarer outcomes below it occur. Expected shortfall then needs the fractional boundary mass calculation; treating it as simply the conditional mean among patients with \(U\leq q_\alpha\) can be wrong when the probability at the quantile is large.

The event probability may itself be uncertain. Expected utility computed at a fixed event rate describes the modeled patient-outcome distribution under that assumption. Uncertainty about the event rate is a different source of uncertainty and should not be silently folded into the patient-level outcome distribution. Compare explicit low, central, and high rate assumptions, or otherwise propagate parameter uncertainty with a stated model; label each result by its assumption. Do not interpret a scenario calculation as a precise estimate if the available evidence cannot support the input rate.

Sparse data are particularly problematic for rare harms. Observing no events in a limited sample does not establish that the event rate is zero. For example, if the true event probability were \(0.01\), the probability of seeing zero events in 300 independent patients would be \(0.99^{300}\), about \(0.0490\). A zero count is therefore possible even when the true rate is nonzero. Relevant diagnostics include the event definition, exposure time, number of patients at risk, ascertainment completeness, and whether follow-up is long enough to capture the harm.

How Tail Summaries Complement Other Approaches

The summaries are complementary, not competing replacements for the whole outcome model. Expected utility remains useful for comparing average preferences across outcomes. The quantile marks a tail boundary, and expected shortfall gives a severity-sensitive average below that boundary. A scenario table can show whether conclusions are robust to plausible severe-event assumptions. None of these summaries identifies the right utility values or event rates automatically, and none eliminates uncertainty in the evidence or preferences.