Decision Curve Analysis for Dosing Policies

Why evaluate a dosing policy with a decision curve?

A dosing policy can predict toxicity accurately yet still recommend dose changes that are not worthwhile. Decision curve analysis gives one way to evaluate whether a policy’s recommendations have a favorable balance of correctly selected patients and unnecessary actions, across a range of action thresholds. The central comparison is not simply whether a model predicts well, but whether using its recommendations is preferable to acting on everyone or no one under a specified trade-off.

For a dosing example, suppose the action is to reduce or otherwise modify a dose when predicted toxicity risk is high. A patient who would experience the defined toxicity under a reference strategy and is selected for the action is counted as a true positive. A patient selected for action who would not experience that toxicity is counted as a false positive. This orientation must be stated explicitly: if the positive event or action is defined differently, the counts and interpretation change.

Here \(p_t\) is the action threshold, not the event prevalence. It represents a trade-off: a higher threshold means that an unnecessary action is considered more costly relative to missing a patient with the event. At each threshold, compare the policy’s net benefit with the treat-all and treat-none strategies, evaluated in the same patients with the same event definition.

What the threshold weight means

The threshold weight follows from a simple utility comparison. Suppose that acting for a patient who truly has the event provides benefit \(B>0\), while acting for a patient who would not have the event incurs harm \(H>0\). For a patient with event risk \(r\), the expected utility of acting, relative to not acting, is \(rB-(1-r)H\). Acting is favored when $$ rB-(1-r)H>0 \quad\Longleftrightarrow\quad r>\frac{H}{B+H}. $$ Thus the threshold \(p_t=H/(B+H)\) implies \(H/B=p_t/(1-p_t)\). Dividing the summed utility by \(B\) gives the decision-curve expression: each true positive contributes one unit, and each false positive subtracts \(p_t/(1-p_t)\) units.

This is a standardized decision score, not a complete pharmacometric utility model. It encodes a particular exchange rate between correctly acting in a patient with the event and unnecessarily acting in a patient without it. It does not, by itself, represent differing toxicity severity, different treatment effects across patients, dose-response consequences, monitoring costs, or other outcomes. If those features are essential, an expected-utility analysis with the relevant outcomes and utilities is needed; a decision curve can still be a useful, narrower comparison.

The standard comparators have simple forms. Treat-none means no patient is selected for action, so its net benefit is zero. Treat-all selects every patient. If \(\hat p\) is the observed event proportion, treat-all has $$ \mathrm{NB}_{\mathrm{all}}(p_t)=\hat p-(1-\hat p)\frac{p_t}{1-p_t}. $$ Treat-none has \(\mathrm{NB}_{\mathrm{none}}(p_t)=0\). A policy is favorable relative to these comparators at a particular threshold when its net benefit is greater than both values at that threshold.

Constructing a decision curve

For each policy and each selected threshold, classify patients according to the policy’s action rule and count \(\mathrm{TP}\) and \(\mathrm{FP}\). Use the same evaluation set and outcome labels for every policy. Convert counts to net benefit using the formula, then calculate treat-all and treat-none at that threshold. Repeating those calculations over thresholds produces the decision curve. Thresholds should cover a clinically defensible range; presenting a value outside a meaningful range can make the curve appear informative where the action trade-off is not credible.

A few assumptions deserve attention before counting. The event must be binary and defined for a clear follow-up period. The action must be consistently defined, such as a dose reduction under a specified rule, rather than a mixture of incompatible clinical actions. The event label must represent the outcome relevant to the decision. In particular, if the question concerns toxicity under a reference dosing strategy, the labels need to represent that reference outcome—not merely toxicity observed after patients received different, policy-influenced doses.

This last issue is important. If a dose modification prevents toxicity, patients who were correctly selected may not show the event after the modification. Counting their observed post-action outcomes as non-events would misclassify the very cases the policy was intended to help. A valid evaluation therefore needs an outcome definition and design that support the intended decision comparison, such as appropriate prospective evaluation or defensible estimates of outcomes under the reference strategy. Predicted-risk calibration, as covered in Calibration of Predicted Toxicity and Target Attainment, does not by itself solve this counterfactual-outcome problem.

Comparing policies across thresholds

A policy can be advantageous at one threshold and not another because the false-positive penalty changes with \(p_t\). At low thresholds, the penalty is relatively small and selecting more patients may be reasonable. At high thresholds, each false positive is penalized more heavily. Therefore a single net-benefit number cannot summarize a policy for every clinical setting.

The table below shows a hypothetical policy evaluated in \(n=100\) patients, with 20 events. Its nested action rules yield the listed counts: as the threshold increases, fewer patients are selected, so the true-positive and false-positive counts do not increase.

For example, at \(p_t=0.30\), the policy calculation is \(12/100-(12/100)(0.30/0.70)=0.06857\). The treat-all value is \(0.20-(0.80)(0.30/0.70)=-0.1429\), while treat-none remains zero. This policy exceeds both comparators at all three displayed thresholds. A plotted decision curve would show the same pointwise comparisons, but the numerical calculations should be checked before drawing conclusions from a graph.

Assumptions, checks, and failure modes

Before interpreting a curve, verify that each policy’s risk is oriented toward the event for which action is being considered, that the action occurs at or above the stated threshold, and that all counts use the same eligible patients and outcome ascertainment. A threshold comparison is not meaningful if, for example, one policy is evaluated for severe toxicity and another for any-grade toxicity, or if their analysis sets differ without an explicit reason.

Decision curves are especially vulnerable to errors in the event/action mapping. If positive means toxicity but the policy recommends an action when predicted toxicity is low, the usual true-positive and false-positive labels do not represent the intended decision. Likewise, a recommendation that reduces dose may be clinically appropriate for toxicity risk but could compromise target attainment; the standard binary curve will not automatically count that loss. Define the decision question before computing a curve, and use a richer utility analysis when important benefits and harms are omitted.

Useful checks include confirming that \(0\leq \mathrm{TP}\leq\) number of events and \(0\leq\mathrm{FP}\leq\) number of non-events, verifying that threshold-specific selected sets are nested for a simple risk-threshold rule, and independently recomputing the comparator values. If treat-all or treat-none is missing from a plot, the reader cannot tell whether the policy improves on these basic strategies. If net benefit is reported without its threshold and event definition, the number is incomplete.

How decision curves relate to other evaluations

Confusion matrices, sensitivity, specificity, and predictive values describe classification performance; they do not by themselves apply the threshold-specific trade-off used in net benefit. Overall accuracy can also obscure the fact that false positives and false negatives may have different consequences. Decision curve analysis makes one particular trade-off explicit, but it does not replace calibration assessment or an evaluation of clinical outcomes.

A full expected-utility comparison is the main alternative when reliable utilities and action-specific outcome consequences are available. It can represent multiple benefits and harms, but requires those quantities to be specified and supported. Decision curves are simpler and make comparisons across thresholds visible, but their standardized score can be mistaken for actual clinical utility. The two approaches answer related but distinct questions.

The key limitation is that net benefit is conditional on the event definition, threshold range, action interpretation, and validity of the outcome labels. A positive curve difference is not proof of causal benefit, safety, or generalizability. It indicates that, under the decision-curve assumptions and on the evaluated data, the policy has a more favorable weighted balance of true-positive and false-positive classifications than the comparator at that threshold.

Common mistake

A common mistake is to call a policy with more true positives “better” without accounting for its false positives, or to compare a policy’s net benefit at one threshold with treat-all at another. The correction is to calculate all strategies at the same threshold in the same evaluation population. For instance, at \(p_t=0.25\), Policy A and Policy B in the example above had 16 and 14 true positives, respectively, but B had higher net benefit because its reduction in false positives outweighed its two fewer true positives under the threshold weight. Net benefit is a weighted comparison, not a ranking by sensitivity alone.