Decision Curves and Net Benefit

Why Evaluate Decisions, Not Only Predictions?

A prediction model can rank people well yet offer little help with a particular clinical decision. Whether a person should receive an intervention depends not only on their predicted risk, but also on the consequences of acting when they would not have the outcome and of not acting when they would. A decision curve makes these trade-offs explicit by calculating net benefit at a range of risk thresholds.

In genetic prediction, this allows a polygenic score (possibly combined with clinical predictors) to be evaluated against a baseline model and two simple strategies: intervene for everyone or intervene for no one. The comparison concerns the decisions implied by predicted risks in a specified validation population. It does not establish that a score improves health outcomes; that requires evidence about the intervention and its consequences as well.

What the Threshold Represents

A threshold is not chosen because it makes a model look favorable. It represents a decision-maker’s balance between the expected benefit of acting for someone who would experience the outcome and the harm or burden of acting for someone who would not. To see the connection, suppose a true-positive decision has benefit \(B\), a false-positive decision has harm \(H\), and the decision is to act when expected benefit exceeds expected harm. At risk \(p_t\), the two quantities are \(p_tB\) and \((1-p_t)H\). The boundary between acting and not acting satisfies

Thus the weight in the net-benefit formula corresponds to the relative harm \(H/B\) under this simplified decision framework. For example, a threshold of 0.20 assigns a false-positive decision a weight of \(0.20/0.80=0.25\) relative to a true-positive decision. The threshold and the relevant consequences must be defensible for the clinical decision being studied; the data alone do not determine them.

A decision curve is a plot of net benefit against \(p_t\). At each threshold, the model classifies people with predicted risk at least \(p_t\) as candidates for action, counts the resulting TP and FP classifications, and calculates net benefit. A model therefore has a curve rather than one universal net-benefit value. The curve is usually evaluated over a range of thresholds that could reasonably guide the decision.

Reference Strategies and a Short Derivation

Two reference strategies help interpret a model’s curve. Under treat none, nobody is selected for action, so there are no TP or FP classifications and net benefit is zero. Under treat all, everybody is selected. If the outcome prevalence in the validation population is \(\hat{p}\), then \(\mathrm{TP}/n=\hat{p}\) and \(\mathrm{FP}/n=1-\hat{p}\). Substitution gives the treat-all net benefit.

These expressions follow from the same weighting used for the model: treat-all receives credit for each person with the outcome and incurs the threshold-weighted cost for each person without it. Treat-none is the zero reference because it makes no positive decisions. Treat-all can have negative net benefit at a sufficiently high threshold; that means its weighted false-positive burden exceeds its true-positive benefit.

A model’s net benefit should be judged relative to these strategies and to the clinical baseline model at the same threshold. A higher value indicates better weighted classification utility under the threshold’s assumptions. It is not a measure of discrimination or calibration by itself, and a small numerical difference is not automatically clinically important.

Comparing a Genetic Model With a Baseline Model

To assess the incremental decision value of genetic prediction, compare its net benefit with that of the baseline strategy using the same people, observed outcomes, and thresholds. The baseline may use clinical predictors without the genetic score; the genetic model may add that score. For a fair calculation, both sets of predicted risks must be evaluated in the same target validation population, and each model’s classification rule must use the same \(p_t\). This comparison focuses on the decision curves, rather than treating a difference in one performance metric as proof of clinical usefulness.

Building and Reading a Decision Curve

In practice, net benefit is computed at a sequence of clinically plausible thresholds. At each threshold, the predicted risks from each strategy are compared with that threshold, TP and FP counts are obtained, and the relevant formula is applied. Plotting those values against the thresholds produces the decision curve. Because classifications change when a threshold passes a predicted risk, the estimated curve from a finite sample can change in steps.

The following hypothetical grouped counts illustrate how a curve can compare differently across thresholds. There are 20 people, including 8 with the outcome and 12 without it. For one model, the predicted risks fall into four groups: among those with the outcome, 3 are at least 0.60, 3 are from 0.30 up to but below 0.60, 1 is from 0.10 up to but below 0.30, and 1 is below 0.10. Among those without the outcome, the corresponding counts are 1, 3, 4, and 4. These groups imply the TP and FP counts needed at each threshold.

Assumptions and Interpretation

The arithmetic is straightforward, but its interpretation depends on the study and decision setting. First, the observed outcomes and predicted risks should come from a validation sample relevant to the population where decisions are contemplated. The treat-all calculation uses the sample outcome prevalence, so a prevalence distorted by case-control sampling will generally not give the target-population treat-all net benefit without an appropriate correction. Second, predicted values must represent risks for the outcome and time horizon that define the decision. A threshold applied to a score that is not on a meaningful risk scale does not have the interpretation used here.

The threshold range should also correspond to plausible clinical choices. A model can have high net benefit at thresholds that no decision-maker would use and still be of little practical value. Conversely, a modest but consistent advantage over the baseline model within the relevant range may be worth investigating. Net benefit summarizes the specified balance of true-positive and false-positive decisions; it does not include every possible cost, such as intervention capacity, patient preferences, or downstream effects, unless those are represented by the decision framework.

A decision curve is an estimated comparison, not a guarantee of future performance. In a small validation sample, a few people crossing a threshold can noticeably change the curve. Uncertainty should be considered when judging apparent differences, and comparisons should not be made at thresholds selected only after inspecting which ones favor the genetic model.

Common Mistake

Wrong reasoning: “The genetic model has a higher net benefit at one threshold, so adding the score is clinically useful for every patient and should replace the baseline model.” This confuses a threshold-specific sample comparison with a universal clinical conclusion.

Correction: Net benefit is conditional on the threshold and its implied trade-off, the population’s outcome prevalence, the model’s risk predictions, and the consequences represented by the decision framework. Compare curves over a prespecified, clinically plausible threshold range, include treat-all and treat-none, and assess whether any advantage over the baseline model is meaningful in the intended setting. A favorable decision curve supports a case for further evaluation; it does not by itself establish improved health outcomes.