Consequences of Risk-Based Eligibility Rules

From a Risk Threshold to Its Consequences

A risk threshold is not just a number attached to a prediction. Once a health system uses it to determine who is eligible for an intervention, it creates a rule with practical consequences: how many people are selected, how many of them experience the outcome, and how many eventual events occur among people not selected. These quantities make a threshold’s implications concrete for both service planning and clinical interpretation.

The consequences depend on the population being evaluated, the outcome definition, and the prediction time horizon. A threshold chosen for 10-year risk cannot be evaluated against events measured over a different horizon without changing the question. The examples below use hypothetical cohorts and a fixed outcome horizon.

Definition: For person \(i\), let \(r_i\) be the predicted absolute risk of the specified outcome over a stated time horizon, and let \(Y_i=1\) if the person experiences that outcome within the horizon and \(Y_i=0\) otherwise. At threshold \(t\), the eligibility rule is \(E_i(t)=1\) when \(r_i\geq t\), and \(E_i(t)=0\) otherwise. People with \(E_i(t)=1\) are eligible under the rule; eligibility does not mean that they will experience the outcome or that an intervention will necessarily benefit them.

The convention here includes a person whose predicted risk equals the threshold. A service may define a different tie rule, but it should state that rule because it can affect counts when many predictions equal \(t\). To assess consequences, compare eligibility with the outcome actually observed over the specified horizon.

	Outcome by the horizon: yes	Outcome by the horizon: no
Eligible: \(r_i\geq t\)	True positive (TP)	False positive (FP)
Not eligible: \(r_i<t\)	False negative, or missed event (FN)	True negative (TN)

Here, “false positive” means eligible under the rule but not observed to experience the outcome within the chosen horizon. It does not mean that the person’s risk estimate was necessarily erroneous: a person with nonzero predicted risk may not experience the event. Similarly, a “missed event” is an observed event in someone below the threshold; it does not establish that the event could have been prevented by offering the intervention.

Counts and Denominators

The four cells of the table are counts. They should be reported with clear denominators, because the same count can answer different questions depending on the denominator. The number eligible is \(TP+FP\); the number of observed events is \(TP+FN\); and the total cohort size is \(TP+FP+FN+TN\).

Key measures: The eligibility proportion is \((TP+FP)/n\). The proportion of observed events among eligible people, also called the positive predictive value, is \(TP/(TP+FP)\). The event-detection proportion, or sensitivity, is \(TP/(TP+FN)\). The missed-event proportion is \(FN/(TP+FN)=1-\text{sensitivity}\). Each measure describes a different aspect of the threshold’s consequences.

The positive predictive value answers, “Among those made eligible, what proportion experienced the outcome?” Sensitivity answers, “Among those who experienced the outcome, what proportion were eligible?” Neither is the proportion of eligible people who benefit from intervention. The eligibility proportion is especially useful for estimating workload, while the count of false positives describes how many eligible people did not have the observed event during follow-up. Costs, harms, and benefits of intervention require additional information.

Worked Example: Counting Consequences at One Threshold

Hypothetical data. A validation cohort contains 1,000 people followed for 10 years. A rule makes people eligible when their predicted 10-year risk is at least 10%. Among the 250 people who meet this threshold, 70 experience the outcome; among the 750 below the threshold, 30 experience it. Calculate the confusion-matrix counts and key proportions.

Among eligible people, \(TP=70\), so \(FP=250-70=180\). Among people below threshold, \(FN=30\), so \(TN=750-30=720\). The cohort contains \(70+180+30+720=1{,}000\) people, and its observed number of events is \(70+30=100\).

The eligibility proportion is \((70+180)/1{,}000=250/1{,}000=0.25\), or 25.0%. The positive predictive value is \(70/250=0.280\), or 28.0%. Sensitivity is \(70/100=0.700\), or 70.0%, so the missed-event proportion is \(30/100=0.300\), or 30.0%.

In this hypothetical cohort, the rule makes 250 people eligible. Seventy of the 100 observed events occur among those eligible, while 30 occur below the threshold. These figures describe classification over 10 years; they do not state how many events an intervention would prevent.

Comparing Alternative Thresholds

When thresholds are compared, the cohort and outcome horizon should remain the same. Otherwise, changes in the counts may reflect changes in the people or follow-up being assessed rather than the threshold itself. Lowering a threshold can only add eligible people under the rule \(r_i\geq t\); raising it can only remove eligible people. The number of false positives and true positives can both change as a result.

For a fixed set of predictions and observed outcomes, lowering the threshold cannot reduce the number of eligible people or true positives, and it cannot increase the number of missed events. It may, however, add many people without an observed event. Raising the threshold has the reverse set-inclusion effect: it cannot increase eligibility or true positives, and it cannot reduce missed events. How large these changes are depends on the distribution of risks and events near the thresholds.

Worked Example: Comparing Two Thresholds

Hypothetical data. Use a cohort of 1,000 people followed for 10 years. The predicted-risk groups and observed outcomes are: 500 people below 5%, with 10 events; 250 people from 5% to below 10%, with 20 events; 150 people from 10% to below 20%, with 30 events; and 100 people at least 20%, with 40 events. Compare thresholds of 10% and 20%.

At 10%, the last two groups are eligible: \(150+100=250\) people. They include \(30+40=70\) events, so \(TP=70\) and \(FP=250-70=180\). The first two groups are below threshold: \(500+250=750\) people, including \(10+20=30\) events. Thus \(FN=30\) and \(TN=750-30=720\). Eligibility is \(250/1{,}000=25.0\%\), sensitivity is \(70/100=70.0\%\), and the positive predictive value is \(70/250=28.0\%\).

At 20%, only the group at least 20% is eligible. It contains 100 people and 40 events, giving \(TP=40\) and \(FP=100-40=60\). The 900 people below threshold include the other 60 events, so \(FN=60\) and \(TN=900-60=840\). Eligibility is \(100/1{,}000=10.0\%\), sensitivity is \(40/100=40.0\%\), and the positive predictive value is \(40/100=40.0\%\).

Raising the threshold from 10% to 20% reduces eligibility by 150 people and reduces the number of observed events among eligible people by 30. It also increases missed events from 30 to 60. In this cohort, the higher threshold selects fewer people and has a higher positive predictive value, but it captures a smaller share of all observed events. The counts alone do not establish which threshold is preferable; that judgment also depends on intervention effects, burdens, resources, and the consequences of missed events.

A Short Accounting Derivation

The threshold divides every observed event into exactly one of two groups: an event in an eligible person or an event in a person below threshold. Therefore, the total number of events is \(TP+FN\). Dividing both sides by that total gives the relationship between event detection and missed events:

\frac{TP}{TP+FN}+\frac{FN}{TP+FN} =\frac{TP+FN}{TP+FN}=1.

The first term is sensitivity and the second is the missed-event proportion. This identity holds for any threshold and any cohort in which the outcome is recorded consistently. It is a useful check on calculations: if the two proportions do not add to 1, a count or denominator has been misclassified. It also shows why sensitivity and missed-event proportion must be interpreted with the observed-event count as their denominator, not with the number eligible.

Observed Events and Expected Events Are Different

A validation cohort allows direct counting of observed outcomes. A planning exercise may instead use predicted risks to estimate how many events might occur among eligible people. If the risks are calibrated for the target population and time horizon, the sum of predicted risks among eligible people is the expected number of events in that group. This is an expectation, not a promise that the eventual count will equal the sum.

Calibration matters to that interpretation. If predicted absolute risks systematically overestimate or underestimate outcome frequency in the relevant population, their sum will not provide a reliable estimate of the expected event count there. Even when risks are calibrated, realized event counts vary. The distinction is also important because an expected number of events is not an expected number of events prevented: prevention additionally depends on what the intervention does.

Worked Example: Expected Events Among Eligible People

Hypothetical planning data. A clinic is considering a threshold that would select 100 people over a fixed follow-up horizon. Of these, 20 have predicted risk 30%, 30 have predicted risk 20%, and 50 have predicted risk 10%. Assume the risks are calibrated for this clinic and horizon. Calculate the expected number of events among those selected.

For a binary outcome, a person with predicted risk \(r_i\) contributes \(r_i\) to the expected event count. Summing the risks gives \(20(0.30)+30(0.20)+50(0.10)=6+6+5=17\) expected events. The average predicted risk among the 100 eligible people is \(17/100=0.17\), or 17.0%.

The clinic would therefore plan around 17 expected events in this group under the stated calibration assumption. If follow-up later records 15 or 20 events, that difference does not by itself show that the individual risk values were wrong; a realized count can differ from its expectation. And neither 17 expected events nor the observed count gives the number preventable without evidence about intervention effectiveness.

Why the Consequences Matter

A threshold translates a prediction model into an operational policy. The eligible count indicates the approximate number of people who may need to be contacted, assessed, or offered an intervention. The false-positive count indicates how many eligible people would not experience the outcome during the defined horizon, while the missed-event count identifies how many observed events occurred below the threshold. These figures help make trade-offs visible to clinicians and service planners.

They should be reported in context: state the population, the outcome, the time horizon, the threshold, and whether the counts come from observed outcomes or summed predicted risks. Include both counts and denominators. For example, “30 of 100 events occurred below the 10% threshold” is more informative than “30 events were missed” alone. If the evaluation cohort is not representative of the population where the rule will be used, its consequences may not transfer directly.

Most importantly, a risk-based eligibility rule is a classification policy, not a treatment-effect model. A threshold does not prove that people above it benefit from treatment, that people below it cannot benefit, or that all eligible people should receive the same intervention. Decisions about benefits and harms require evidence about intervention outcomes as well as predicted baseline risk.

Common Mistakes

Calling every eligible person a true positive. Eligibility is determined by predicted risk; a true positive also requires an observed event within the specified horizon. Eligible people without an observed event are false positives for this classification.
Using the wrong denominator. Sensitivity and the missed-event proportion use all observed events, \(TP+FN\), as their denominator. The positive predictive value uses eligible people, \(TP+FP\).
Interpreting a missed event as a preventable event. An event below threshold is an observed outcome in a person not selected by the rule. Whether it could have been prevented is a separate question about intervention effects.
Comparing counts from different horizons. Events observed over 5 years and events observed over 10 years are not interchangeable. Keep the outcome horizon consistent when comparing thresholds.
Assuming a higher positive predictive value means a better policy. It may come with lower sensitivity and more missed events. No single count or proportion determines the preferred threshold.

Key takeaway: For each threshold, count eligible people and partition both eligible and ineligible groups by observed outcome. Compare thresholds in the same population and horizon, report the denominators, and keep classification consequences separate from intervention benefit.

Check Your Understanding

Use a fixed population and outcome horizon in each answer.

A rule selects 300 people, 45 of whom experience the outcome. Among the 700 people not selected, 35 experience it. Calculate \(TP\), \(FP\), \(FN\), and \(TN\).
Using the counts in question 1, calculate the eligibility proportion, positive predictive value, sensitivity, and missed-event proportion. State each denominator.
In a fixed cohort, explain why lowering the risk threshold cannot increase the number of missed events, even though it may increase the number of false positives.
A calibrated-risk planning group contains 10 people at predicted risk 20% and 40 people at predicted risk 5%. What is the expected number of events in the group, and why is it not a guaranteed observed count?
Why does a count of events among eligible people not, by itself, tell a clinic how many events its intervention will prevent?