Decision Thresholds and Net Benefit From PRS

From a Risk Estimate to a Clinical Decision

A polygenic risk score (PRS) can contribute to an estimate of disease risk, but a risk estimate is not itself a treatment decision. A decision requires a specified action, such as offering a preventive intervention, and a comparison between the likely benefits and harms of taking that action. A threshold makes this comparison operational: act when estimated risk reaches or exceeds a chosen value.

The threshold should apply to a risk estimate for a defined outcome and time horizon, not automatically to the raw PRS. The same PRS value can correspond to different absolute risks in different populations. The introductory material on calibration is relevant here: risk-based decisions depend on whether predicted risks match observed risks in the target population. In practice, a PRS may be combined with clinical covariates to produce the risk used for the decision.

Definition: For a binary outcome assessed over a specified time horizon, a risk threshold \(p_t\) is the predicted-risk value at which a decision changes. Under a threshold policy, individuals with estimated risk at least \(p_t\) are classified as candidates for the action; those below \(p_t\) are not. Sensitivity is the proportion of individuals with the outcome who are classified as candidates. Specificity is the proportion of individuals without the outcome who are not classified as candidates.

These performance measures describe what a threshold does to the observed cases and non-cases; they do not determine whether the resulting decisions are worthwhile. A threshold with high sensitivity may capture more future cases but also lead to more interventions among people who would not experience the outcome. Decision-curve analysis puts those two consequences on a common scale, using the clinical trade-off represented by the threshold.

Net Benefit and the Meaning of the Threshold

Consider \(N\) individuals with a binary outcome observed over the chosen horizon. At threshold \(p_t\), let \(TP\) be the number who have the outcome and are classified for action, and let \(FP\) be the number who do not have the outcome but are classified for action. The decision-curve net benefit is

NB(p_t)=\frac{TP}{N}-\frac{FP}{N}\frac{p_t}{1-p_t}.

The first term rewards identifying an individual who will experience the outcome. The second penalizes an unnecessary action in someone who will not. The weight \(p_t/(1-p_t)\) is the relative penalty for a false-positive action, expressed in units of a true-positive benefit. Net benefit is often reported as true-positive equivalents per person; multiplying by 1,000 gives the corresponding number per 1,000 people.

The threshold has a decision-theoretic interpretation under a simplified utility model. Suppose correctly acting for a person who would experience the outcome gives benefit \(B>0\), and acting for a person who would not experience it causes harm \(H>0\). Suppose, for this comparison, that withholding action has utility zero in either case and that the action's relevant consequences are captured by \(B\) and \(H\). If the person's risk is \(p\), the expected utility of acting is \(pB-(1-p)H\). Acting is preferred when this quantity is nonnegative:

pB-(1-p)H\geq 0 \quad\Longleftrightarrow\quad p\geq \frac{H}{B+H}.

Thus \(p_t=H/(B+H)\), and \(p_t/(1-p_t)=H/B\). This explains why the false-positive penalty in net benefit is the threshold odds: higher harm relative to benefit implies a higher threshold and a larger penalty for unnecessary action. Real decisions may involve additional consequences, such as treatment burdens for all patients or benefits from avoiding an intervention. The simplified model clarifies the threshold logic; it does not replace clinical specification of the relevant outcomes and utilities.

Two reference policies help interpret a decision curve. Treating no one has \(TP=FP=0\), so its net benefit is zero. Treating everyone has \(TP/N\) equal to the outcome prevalence and \(FP/N\) equal to one minus that prevalence. If prevalence is \(\pi\), then:

NB_{\text{all}}(p_t)=\pi-(1-\pi)\frac{p_t}{1-p_t}.

A model-based policy is useful at a threshold when its net benefit exceeds the relevant reference policies, provided the threshold reflects a clinically defensible trade-off. Sensitivity and specificity remain useful for describing classification performance, but neither alone incorporates the relative consequences of false positives and false negatives.

Worked Calculations

Worked Example: Classifying Risk at a 10% Threshold

Suppose a hypothetical validation cohort contains 1,000 people, of whom 100 experience the outcome during the specified horizon and 900 do not. A calibrated PRS-based risk model classifies 70 of the 100 outcome cases and 180 of the 900 non-cases at or above \(p_t=0.10\). The remaining counts are \(FN=30\) and \(TN=720\).

Sensitivity is the fraction of outcome cases classified for action:

\text{Sensitivity}=\frac{TP}{TP+FN}=\frac{70}{70+30}=0.700.

Specificity is the fraction of non-cases classified below the threshold:

\text{Specificity}=\frac{TN}{TN+FP}=\frac{720}{720+180}=0.800.

At \(p_t=0.10\), the false-positive weight is \(0.10/(1-0.10)=1/9\). Therefore:

NB(0.10)=\frac{70}{1000}-\frac{180}{1000}\frac{0.10}{0.90} =0.070-0.020=0.050.

The estimate is 0.050 true-positive equivalents per person, or 50 per 1,000, at this threshold. The treat-none policy has net benefit zero. The cohort prevalence is \(100/1000=0.10\), so treating everyone has net benefit \(0.10-0.90(1/9)=0\). In this hypothetical cohort, the model-based policy has greater net benefit than either reference policy at the 10% threshold. This comparison does not establish that the intervention is beneficial in every setting; it relies on the threshold and the validation data being relevant to the intended decision.

Worked Example: Evaluating a 20% Threshold

Using the same hypothetical cohort, suppose a 20% risk threshold classifies 55 of the 100 outcome cases and 80 of the 900 non-cases for action. Then \(FN=45\) and \(TN=820\). The operating characteristics are:

\text{Sensitivity}=\frac{55}{100}=0.550, \qquad \text{Specificity}=\frac{820}{900}=0.911.

At \(p_t=0.20\), the false-positive weight is \(0.20/0.80=0.25\). Net benefit is:

NB(0.20)=\frac{55}{1000}-\frac{80}{1000}(0.25) =0.055-0.020=0.035.

Treating no one has net benefit zero. Treating everyone has net benefit \(0.10-0.90(0.25)=-0.125\), so the model-based policy has greater net benefit than either reference at the 20% threshold in this example. The higher threshold has lower sensitivity and higher specificity, which is consistent with classifying fewer people for action. The net-benefit values at 10% and 20% should not be read as a simple contest between thresholds: each threshold encodes a different false-positive penalty and therefore a different clinical trade-off. The threshold should be justified by consequences, not selected merely because its calculated net benefit is numerically largest.

Worked Example: Does Adding a PRS Improve Decisions?

In another hypothetical cohort, 160 of 2,000 people experience the outcome, giving prevalence \(160/2000=0.080\). At a clinically specified threshold of 10%, a clinical-covariate model classifies 80 cases and 360 non-cases for action. A model that adds a PRS classifies 112 cases and 288 non-cases for action. Both models are evaluated in the same people at the same threshold.

For the clinical-only model, sensitivity is \(80/160=0.500\), and specificity is \((1840-360)/1840=1480/1840=0.804\). For the model including the PRS, sensitivity is \(112/160=0.700\), and specificity is \((1840-288)/1840=1552/1840=0.843\). The respective net benefits are:

NB_{\text{clinical}}(0.10) =\frac{80}{2000}-\frac{360}{2000}\frac{1}{9} =0.040-0.020=0.020,

NB_{\text{clinical+PRS}}(0.10) =\frac{112}{2000}-\frac{288}{2000}\frac{1}{9} =0.056-0.016=0.040.

The treat-none net benefit is zero. Treating everyone gives \(0.080-0.920(1/9)=-0.0222\). At this threshold, the model including the PRS has a net-benefit advantage of \(0.040-0.020=0.020\) per person over the clinical-only model, or 20 true-positive equivalents per 1,000 people. The same result can be checked from the changes in counts: it identifies 32 additional cases, worth \(32/2000=0.016\) per person, and avoids 72 unnecessary actions, worth \((72/2000)(1/9)=0.004\) per person. Together these changes give \(0.016+0.004=0.020\). This is evidence of improved decision value in the hypothetical validation cohort at this threshold, not evidence that the PRS causes better health outcomes.

Assumptions, Diagnostics, and Alternatives

A decision-curve calculation is only interpretable when the decision and validation setting are clearly specified. The key conditions are:

A defined outcome and horizon. Cases and non-cases must be determined for the same outcome over a stated time period. Mixing horizons or changing the phenotype definition changes the quantities being evaluated.
A relevant target population. The validation cohort should represent the people for whom the policy is intended, or differences in population and sampling must be addressed. In particular, outcome prevalence affects the numbers of true and false positives and the treat-all reference.
Risk estimates suitable for thresholding. A predicted 10% risk should correspond to about a 10% observed event rate among comparable people assigned that risk. Poor calibration can lead to inappropriate threshold decisions even when ranking or discrimination is useful. A raw PRS percentile is not an absolute risk unless it has been translated to risk for the target population and horizon.
A clinically meaningful action and threshold. The threshold should represent an explicit trade-off in the consequences of acting and not acting. If intervention benefit varies across people, or action has important harms not represented by the threshold, a single-threshold calculation may not capture the decision fully.
Comparable evaluation. Competing models and reference policies should be assessed on the same target population, outcome definition, horizon, and threshold range. Otherwise a difference in net benefit may reflect differences in evaluation rather than in decision performance.

Diagnostics should focus on whether the reported operating point is stable and interpretable. Check the \(TP\), \(FP\), \(FN\), and \(TN\) counts, especially when few cases occur near a threshold or the validation cohort is small. Confirm that sensitivity and specificity use the intended denominators and that the risk threshold is applied consistently. Plot or tabulate net benefit across a clinically plausible range of thresholds, and show treat-all and treat-none references. A favorable result at one isolated threshold is less persuasive if nearby values perform poorly or if the threshold lies outside the range relevant to practice. Uncertainty in validation metrics also matters; a point estimate of net benefit does not guarantee that its advantage over another policy is precise.

Decision curves complement, rather than replace, other evaluation approaches. Discrimination metrics summarize how well a model separates cases from non-cases, but do not encode a particular clinical trade-off. Calibration diagnostics assess agreement between predicted and observed risks, but do not by themselves establish that acting on predictions is worthwhile. Sensitivity and specificity describe a chosen threshold without applying a relative consequence weight. Decision-curve net benefit combines the threshold-specific counts with that weight, making it useful for comparing actions at clinically relevant thresholds. None of these summaries alone demonstrates that implementing the policy improves patient outcomes; that question may require prospective evaluation of the decision strategy.

Common Pitfall and Practical Interpretation

Wrong reasoning: “The PRS has higher sensitivity, so it is the better clinical policy,” or “the threshold with the largest net-benefit number is automatically the right threshold.” Higher sensitivity can come at the cost of many unnecessary actions, and net-benefit values at different thresholds reflect different false-positive penalties. Correction: specify the outcome horizon and action, choose or justify a clinically defensible threshold range, compare models at the same threshold, and evaluate both reference policies. Report sensitivity and specificity to show the operating characteristics, then interpret net benefit as a threshold-dependent comparison of decision value in the target setting.

Key takeaway: A PRS is decision-relevant only insofar as its risk estimates support better choices at thresholds that reflect clinical consequences. Net benefit makes the cost of false-positive actions explicit, but it does not choose the threshold or establish clinical utility without a justified decision context.

Check Your Understanding

Answer each question, then check the concise solutions below.

In a cohort with 40 outcome cases and 360 non-cases, a threshold identifies 30 cases and 90 non-cases for action. Calculate sensitivity and specificity.
At \(p_t=0.25\), what is the false-positive weight in the net-benefit formula, and what clinical trade-off does it represent?
For a cohort of 1,000 with prevalence 0.08, calculate the net benefit of treating everyone at a threshold of 0.10.
Why is it not sound to select a threshold just because it has the largest net-benefit value among several thresholds?

Solutions. (1) Sensitivity is \(30/40=0.750\). Specificity is \((360-90)/360=0.750\). (2) The weight is \(0.25/0.75=1/3\); in the simplified utility interpretation, an unnecessary action is penalized by one-third of the benefit assigned to a true-positive action. (3) Treat-all net benefit is \(0.08-0.92(0.10/0.90)=0.08-0.1022=-0.0222\) true-positive equivalents per person. (4) Each threshold implies a different relative penalty for false positives and therefore a different clinical trade-off. Thresholds should be justified by consequences; model comparisons should be made at the same threshold.