Decision-Curve Analysis for Dynamic Risk Predictions

Why Evaluate Predictions as Decisions?

A prediction model can rank people well yet provide little help with a particular decision. For example, a clinician may need to decide whether a patient should receive an intervention because their risk of an event over the next year is high enough to justify its burdens. Decision-curve analysis evaluates predictions at thresholds that represent such trade-offs. It complements discrimination and calibration: it asks whether acting on the predictions yields more benefit, after accounting for false-positive decisions, than simple alternatives.

For a dynamic prediction, the decision is made at a specified landmark \(s\), using information available up to that time. The prediction concerns a specified future horizon \(\tau\), measured from the landmark. The target event and horizon must be stated before calculating net benefit. In a competing-risks setting, the target outcome is occurrence of the event of interest by \(s+\tau\), conditional on being eligible at \(s\); a competing event before that time means the target event did not occur by the horizon.

The threshold \(p_t\) is the risk at which the expected benefit of intervention just balances its expected harm. A low threshold makes intervention relatively easy to justify and therefore penalizes false positives less. A high threshold makes intervention harder to justify and gives false-positive decisions a larger penalty. Thresholds should therefore be clinically meaningful, not selected merely because they make a model's curve look favorable.

Net Benefit and Its Interpretation

Suppose the horizon outcome is fully observed for \(n\) eligible people. Let \(TP\) be the number with the target event by the horizon who are classified above threshold, and \(FP\) the number without that event who are classified above threshold. Then:

Net benefit is expressed in true-positive-equivalent decisions per person. For example, \(NB=0.08\) means a net gain of 0.08 true-positive-equivalent decisions per eligible person, relative to a strategy with net benefit zero, under the threshold's assumed trade-off. Multiplying by 100 gives the equivalent net gain per 100 people. It is not a probability that an individual will benefit from intervention.

The two basic comparison strategies are to intervene for nobody and to intervene for everybody. Treat-none has \(NB=0\). If the event proportion by the horizon is \(\bar Y\), treat-all has:

A model is potentially useful at a threshold when its net benefit exceeds both relevant alternatives. A positive value alone is not enough: treat-all may have a still larger value. Comparisons apply only to the chosen population, landmark, event definition, horizon, and threshold range.

Why the Threshold Odds Appear

The false-positive weight follows from the decision trade-off, rather than being an arbitrary adjustment. Suppose a true-positive intervention has benefit \(B>0\), a false-positive intervention has harm \(H>0\), and true-negative and false-negative decisions define the zero reference. For a person with event risk \(p\), the expected utility of intervening is \(pB-(1-p)H\). Intervention is favored when this quantity is positive:

Setting \(p_t=H/(B+H)\) gives \(H/B=p_t/(1-p_t)\). Thus, after dividing utility by the benefit \(B\), a false-positive decision receives penalty \(p_t/(1-p_t)\), which yields the net-benefit formula. This interpretation assumes that the threshold adequately represents the relevant benefit-harm trade-off and that the same trade-off applies to the people being evaluated.

Dynamic Predictions, Competing Risks, and Censoring

For dynamic analysis, calculate net benefit separately for the intended landmark-horizon pair. A person's prediction must use only longitudinal measurements and other information available by \(s\); using measurements recorded after \(s\) would give an unrealistic evaluation. The risk set should also match the prediction target: for example, people still free of the relevant events at \(s\). If predictions are evaluated at multiple landmarks, each landmark defines a distinct decision problem and potentially a different eligible population.

For competing risks, define a positive outcome as the target cause occurring by \(s+\tau\). A competing event observed first is a negative outcome for this target-event classification by that horizon, not ordinary loss to follow-up. The event of interest's cumulative incidence is the relevant predicted risk; treating competing events as if they were non-informative censoring would change the target and can distort the predicted risk and decision curve.

If some people are censored before the horizon, their horizon outcome is unknown. They should not simply be counted as non-events. One common approach is inverse-probability-of-censoring weighting (IPCW). Let \(G(u\mid Z)\) denote the probability of remaining uncensored through time \(u\), conditional on suitable observed variables \(Z\). Observed target events are weighted by the inverse of the estimated censoring probability at their event time; observed non-events, including observed competing events, are weighted according to their observed follow-up through the horizon or competing event. The weighted true- and false-positive totals are then divided by the number eligible at the landmark to estimate their proportions.

This approach requires correct handling of the event and censoring times, adequate follow-up probability (positivity), and conditional independent censoring given variables used in the censoring model. If these assumptions are implausible, the resulting curve may be biased. The weighting procedure and its assumptions should be reported; a complete-case calculation is not automatically a valid substitute.

Reading Curves, Checking Assumptions, and Choosing Alternatives

A decision curve plots net benefit against \(p_t\) over a range where intervention thresholds are clinically plausible. At each threshold, compare the model curve with treat-all and treat-none. A curve that is higher across a meaningful range suggests greater decision value across those trade-offs; it does not imply superiority at every threshold or in every population. Uncertainty matters: resampling should respect the validation design and, when a person contributes predictions at multiple landmarks, should generally resample people rather than treating their records as independent.

Check that the target event, landmark eligibility, horizon, and prediction time origin agree throughout. Confirm that classifications use the intended threshold and that competing events are handled as target-event non-occurrences when observed before the horizon. With censoring, inspect the estimated censoring probabilities and the size of the weights; very large weights can make estimates unstable. Also verify that the threshold range reflects real decisions and that the model's predictions are evaluated in data not used to fit or tune it.

Decision-curve analysis differs from discrimination measures, such as those for competing risks, because it incorporates a threshold-specific penalty for false positives and compares actions. It also differs from calibration: calibration assesses agreement between predicted risks and observed risks, while net benefit evaluates a decision rule derived from those risks. A well-calibrated model may have little advantage over a simple strategy at a particular threshold, and an apparent net-benefit advantage does not excuse poor calibration or validation. Alternatives include choosing an action based on a single prespecified threshold or conducting a fuller decision analysis with more detailed utilities, costs, and treatment effects. Decision curves are most useful when the threshold trade-off can reasonably be summarized by \(p_t\).

The analysis is not a causal estimate of intervention effectiveness. It assumes a meaningful risk threshold and evaluates how decisions based on predicted risk classify outcomes under the observed outcome process. It does not by itself account for treatment changing subsequent risk, unequal intervention costs across individuals, or all consequences of a decision. These limitations should shape any clinical interpretation.