Decision-Curve Analysis and Net Benefit

Why Evaluate Decisions Rather Than Predictions Alone?

A risk model can rank people well or estimate their absolute risks accurately without necessarily improving a clinical decision. In practice, a decision often depends on whether estimated risk exceeds a threshold: for example, whether to offer a preventive intervention. Decision-curve analysis evaluates such threshold-based strategies by accounting for both people who would benefit from treatment and people treated unnecessarily.

The key quantity is net benefit. It puts true-positive decisions and false-positive decisions on a common scale, with the relative penalty for a false positive determined by the risk threshold. A decision curve plots net benefit across a range of thresholds, allowing a polygenic risk model to be compared with a clinical-only model and with simple default strategies.

Definition: At a risk threshold $p_t$, a person is classified as positive for a decision if their predicted risk is at least $p_t$. Given $n$ evaluated people, $TP$ is the number of people classified positive who experience the outcome by the specified time horizon, and $FP$ is the number classified positive who do not experience it. The model's net benefit is $$ NB(p_t)=\frac{TP}{n}-\frac{FP}{n}\frac{p_t}{1-p_t}. $$

Here the outcome must be defined for a particular horizon, such as disease within five years. The threshold is the risk at which the decision-maker would choose intervention rather than no intervention. Its false-positive weight, $p_t/(1-p_t)$, is the threshold odds. Thus, a higher threshold imposes a larger penalty for treating someone who would not experience the outcome. The formula treats each true-positive classification as one unit of benefit and scales false-positive classifications by this threshold-specific weight.

The threshold should reflect the consequences of the decision, not simply be chosen because it makes a model look favourable. As in the earlier material on choosing risk thresholds with benefit and harm, a threshold summarizes a trade-off between the value of preventing an outcome and the burden or harm of intervention. Net benefit expresses that trade-off in a standardized form; it is not automatically a count of events prevented or a monetary measure.

Reference Strategies and the Decision Curve

A model's net benefit is most informative when compared with strategies that do not require its predictions. Treat none classifies nobody as positive, so it has $TP=FP=0$ and net benefit zero at every threshold. Treat all classifies everybody as positive. If the outcome prevalence in the evaluation sample is $\hat{p}$, its net benefit is

NB_{\mathrm{all}}(p_t)=\hat{p}-(1-\hat{p})\frac{p_t}{1-p_t}.

The treat-all expression follows because $TP/n=\hat{p}$ and $FP/n=1-\hat{p}$. Treat-none is also called the zero-net-benefit reference. A model's net benefit greater than both references at a threshold indicates better standardized decision value than either default strategy at that threshold, under the assumptions underlying the net-benefit interpretation.

To form a decision curve, calculate net benefit for each model and reference strategy at each threshold of interest, using the same evaluation population and outcome definition. The horizontal axis is the risk threshold; the vertical axis is net benefit. A curve above another has greater net benefit at those thresholds, but that comparison is not a claim that the model will improve health outcomes in actual practice.

Interpretation: Net benefit is a weighted difference between true-positive and false-positive classifications per evaluated person. For example, $NB=0.04$ means 0.04 net-benefit units per person at that threshold. It can equivalently be reported as 40 units per 1,000 people. These are threshold-weighted units, not necessarily 40 events prevented.

Worked Calculations on a Hypothetical Evaluation Cohort

Suppose two risk models are evaluated in the same hypothetical, independent cohort of 1,000 people, with complete outcome ascertainment by a prespecified time horizon. There are 100 people who experience the outcome and 900 who do not, so the outcome prevalence is 0.10. The polygenic model here means a risk model that incorporates a polygenic score along with the intended clinical information; the comparison model uses clinical predictors only. At each threshold, people are classified positive when predicted risk meets or exceeds that threshold.

Assume the following threshold-specific classifications. Counts decrease or stay the same as thresholds rise, as expected for a fixed set of predicted risks.

Strategy	Threshold	TP	FP	Number classified positive
Polygenic model	5%	80	320	400
Polygenic model	10%	65	185	250
Polygenic model	20%	40	110	150
Clinical-only model	5%	75	325	400
Clinical-only model	10%	60	240	300
Clinical-only model	20%	35	165	200

Worked Example: Polygenic Model at a 10% Threshold

At $p_t=0.10$, the polygenic model classifies 65 of the 100 people with the outcome and 185 of the 900 without it as positive. The sample size is $n=1{,}000$. First calculate the false-positive weight:

\frac{p_t}{1-p_t}=\frac{0.10}{0.90}=0.1111.

Substituting the counts into the net-benefit formula gives

NB(0.10)=\frac{65}{1{,}000}-\frac{185}{1{,}000}(0.1111) =0.065-0.02056 =0.04444.

The polygenic model therefore has about 0.0444 net-benefit units per person, or 44.4 per 1,000 people, at this threshold. The number 44.4 is not a count of events prevented; it is the net-benefit measure after weighting false positives according to the 10% threshold.

Worked Example: Comparing Models at a 10% Threshold

For the clinical-only model at the same threshold, $TP=60$ and $FP=240$. The threshold weight remains $0.1111$, because both models are being evaluated for the same decision:

NB_{\mathrm{clinical}}(0.10) =\frac{60}{1{,}000}-\frac{240}{1{,}000}(0.1111) =0.060-0.02667 =0.03333.

Treat-all has $TP=100$ and $FP=900$; treat-none has neither:

NB_{\mathrm{all}}(0.10) =\frac{100}{1{,}000}-\frac{900}{1{,}000}(0.1111) =0.100-0.100 =0, $$ $$ NB_{\mathrm{none}}(0.10)=0.

At 10%, the polygenic model has higher net benefit than the clinical-only model by $0.04444-0.03333=0.01111$ units per person, and both models exceed the two default strategies. This is a threshold-specific comparison in this hypothetical cohort; it does not establish that adding a polygenic score will improve decisions in other populations or in clinical practice.

Worked Example: Comparing Strategies at a 20% Threshold

At $p_t=0.20$, the threshold weight is $0.20/0.80=0.25$. For the polygenic model, $TP=40$ and $FP=110$:

NB_{\mathrm{polygenic}}(0.20) =\frac{40}{1{,}000}-\frac{110}{1{,}000}(0.25) =0.040-0.0275 =0.0125.

For the clinical-only model, $TP=35$ and $FP=165$:

NB_{\mathrm{clinical}}(0.20) =\frac{35}{1{,}000}-\frac{165}{1{,}000}(0.25) =0.035-0.04125 =-0.00625.

For treat-all, the prevalence is 0.10, so

NB_{\mathrm{all}}(0.20) =0.10-0.90(0.25) =0.10-0.225 =-0.125.

Treat-none remains at zero. At this threshold, the polygenic strategy is above both the clinical-only strategy and the default strategies, while the clinical-only and treat-all strategies are below treat-none. These values indicate relative net benefit for the stated threshold and hypothetical data, not that treatment decisions have been proven to prevent a specified number of outcomes.

Why the Threshold Weight Has This Form

A short utility argument explains the odds weight. Suppose a true-positive decision has benefit $B>0$, while treating a person who would not experience the outcome has harm $H>0$. If the threshold is the risk $p_t$ at which intervention and no intervention have equal expected utility, then

$$ p_t B-(1-p_t)H=0. $$

Rearranging gives $p_tB=(1-p_t)H$, and hence $H/B=p_t/(1-p_t)$. If each true-positive classification is assigned one benefit unit, a false positive must therefore be assigned a penalty of $p_t/(1-p_t)$ benefit units. Applying those weights to the observed proportions of true positives and false positives yields the net-benefit formula. The argument assumes the threshold summarizes the relevant benefit-harm trade-off and that the same trade-off applies across people being compared.

Reading the Whole Decision Curve

Calculating only one threshold can hide how performance changes across decisions. For the hypothetical cohort above, the complete arithmetic at the three selected thresholds is summarized below. Treat-all values use the cohort prevalence of 0.10; treat-none is zero at all three thresholds.

Threshold	Polygenic model	Clinical-only model	Treat all
5%	0.06316	0.05789	0.05263
10%	0.04444	0.03333	0
20%	0.01250	-0.00625	-0.1250

At each of these thresholds, the polygenic model has the largest net benefit in the example. The margins and rankings vary by threshold, however, so a decision curve should be examined over a clinically meaningful range rather than reduced to a single summary point. Outside that range, a threshold may not correspond to a realistic decision or may have too few people classified positive to support a stable estimate.

For real evaluation data, net-benefit estimates are uncertain because the observed classifications and outcomes come from a finite sample. Comparisons should therefore be interpreted with appropriate uncertainty assessment and a design that avoids evaluating a fitted model on the same data used to build it. Prediction risks should be relevant to the evaluation population and time horizon; calibration matters because the decision rule explicitly compares predicted absolute risk with a threshold. The curve describes the consequences of applying the specified rule to the evaluated data, not the performance of every possible future implementation.

Assumptions and Common Pitfalls

A decision curve is meaningful only when the components of the calculation line up with the intended decision. The outcome must be defined consistently for all strategies, and the threshold must refer to the same intervention and outcome horizon. In a survival setting, incomplete follow-up or censoring requires an appropriate method for estimating threshold-specific classifications and outcomes; simply treating unobserved outcomes as absent can bias net benefit. Similarly, a cohort selected in a way that changes outcome prevalence or risk distributions may not represent the population where the decision will be used.

Comparing models on different people. Different evaluation samples can produce different outcome prevalences and case mixes. Use the same target population and outcome definition for a direct comparison.
Using a threshold as an arbitrary plotting choice. A threshold should reflect a decision-relevant balance of benefit and harm. A curve cannot determine that balance for the decision-maker.
Calling net benefit events prevented. Net benefit is a threshold-weighted classification measure. It is not automatically a causal estimate of outcomes prevented, and actual clinical benefit also depends on whether the intervention works and is implemented as intended.
Assuming a higher curve proves the polygenic score is clinically useful. A higher curve means higher estimated net benefit at the evaluated thresholds under the chosen assumptions. It does not alone establish feasibility, equitable performance, acceptable costs, or improved patient outcomes.

Key takeaway: Decision-curve analysis compares threshold-based strategies using $NB=TP/n-(FP/n)\{p_t/(1-p_t)\}$. Compare the polygenic model, clinical-only model, treat-all, and treat-none at the same clinically relevant thresholds and in the same target population; interpret the curve as decision-value evidence, not as proof of clinical effectiveness.

Check Your Understanding

Use the definitions and calculations above to answer these questions.

At a 15% threshold, what is the false-positive weight $p_t/(1-p_t)$, rounded to three significant figures?
In a cohort of 2,000 people, a model has 120 true positives and 280 false positives at a 10% threshold. Calculate its net benefit and state the units of interpretation.
If outcome prevalence is 8%, calculate treat-all net benefit at a 5% threshold. Is it above or below treat-none?
Why can a model have positive net benefit but still not be a suitable clinical strategy?
In the hypothetical table, why is it inappropriate to interpret the polygenic model's net benefit of 0.04444 at 10% as 44 outcomes prevented per 1,000 people?