Threshold-Based Classification From Predicted Risk

From Predicted Risk to a Classification

A predicted risk is a number; a threshold turns it into an action-oriented label. For example, a prediction rule might label a person “positive” when their predicted probability of an event within the next year is at least 20%. That classification makes it possible to summarize how well the rule detects events and how often it flags people who remain event-free. Such summaries are useful only when the event definition, horizon, threshold, and handling of censoring are all made explicit.

Consider a validation cohort assessed at a prediction time \(t\). Let \(h\) be the prediction horizon, and include people who are event-free and under observation at \(t\). A joint model supplies each person’s predicted risk using information available at \(t\), including the longitudinal history observed up to that time. Let \(T_i\) be the time from \(t\) to the event for person \(i\), and let \(\hat r_i(t,h)\) be their predicted probability of experiencing the event by \(t+h\), conditional on being event-free at \(t\) and on the information available then.

These are horizon-specific, threshold-specific quantities. Sensitivity describes event detection among people who experience the event by the horizon; it is not the proportion of all positive classifications that are events. That latter proportion is PPV. “Event detection” is sometimes used informally for sensitivity, so a report should state the exact denominator. The proportion of the cohort classified positive is another quantity, the positive classification rate, and is not itself a measure of accuracy.

Complete Outcomes: The Basic Calculation

If every person’s event status at the horizon is known, the measures come from a two-by-two table. True positives (TP) are predicted positives with an event by the horizon; false negatives (FN) are predicted negatives with an event. False positives (FP) are predicted positives without an event by the horizon; true negatives (TN) are predicted negatives without an event.

Sensitivity and specificity condition on the observed horizon outcome. PPV and NPV condition on the classification. In particular, PPV and NPV depend on how common the event is in the population being evaluated, as well as on the threshold and the model’s performance.

Why Censoring Changes the Calculation

In time-to-event validation, some people are lost to follow-up or reach the end of data collection before \(t+h\). If a person has not had the event when follow-up ends at month 7, their 12-month outcome is unknown; they cannot simply be counted as event-free. Excluding everyone censored before the horizon can also distort performance if those people differ systematically from those with complete follow-up.

Inverse-probability-of-censoring weighting (IPCW) addresses this by giving more weight to people whose observed follow-up represents others with similar measured censoring patterns. Let \(C_i\) be the time from \(t\) to censoring, and let \(Z_i\) denote variables used in an appropriate censoring model. Define \(G_i(s)=P(C_i\ge s\mid Z_i)\), the probability of remaining uncensored through time \(s\), conditional on those variables. Write \(\hat G_i(s)\) for its estimate. For an observed event at time \(T_i\), use the probability of remaining uncensored just before that event, \(\hat G_i(T_i-)\); for an observed event-free outcome at the horizon, use \(\hat G_i(h)\).

Let \(D_i\) indicate an event observed before censoring and by \(h\). A person censored before \(h\) without a prior event has an unknown binary outcome and contributes to neither observed event cases nor observed event-free controls. Define the IPCW contribution \(W_i\) as \(1/\hat G_i(T_i-)\) for an observed event by \(h\), \(1/\hat G_i(h)\) for a person observed event-free through \(h\), and zero for someone censored before \(h\) without a prior event. If \(A_i=I\{\hat r_i(t,h)\ge q\}\), the weighted totals are:

The weighted sensitivity, specificity, PPV, and NPV use the same ratios as in the complete-outcome case, replacing the four counts with their weighted versions. These are estimates: the weighted totals need not be integers, and their sum need not equal the number of people in the cohort.

Why the Weights Work

The key idea is to compensate for the chance that an outcome would be observed. For a person whose event occurs at time \(s\), the event is observed if censoring has not occurred by \(s\). Conditional on the censoring-model variables, this observation probability is \(G_i(s-)\). If censoring is conditionally independent of the event time, weighting an observed event by \(1/G_i(s-)\) cancels that observation probability. Similarly, an event-free status at \(h\) is observed only if the person remains uncensored through \(h\), with probability \(G_i(h)\).

For event-free people at the horizon, the corresponding identity is \(E[I(C_i\ge h)/G_i(h)\mid T_i>h,Z_i]=1\). Thus, under the stated assumptions, the weighted event and event-free totals estimate the totals that would have been available with complete follow-up. Dividing the appropriate weighted totals estimates the conditional proportions of interest. The argument fails if censoring depends on unmeasured event-related information after conditioning, or if some relevant group has essentially no chance of remaining under observation.

Thresholds, Comparisons, and Diagnostics

A lower threshold generally classifies more people as positive, which can increase sensitivity while reducing specificity; a higher threshold generally does the reverse. The change is not guaranteed to be strict at every threshold, because predicted risks may tie or because of the observed data. The threshold should be selected for a stated practical purpose and, when possible, prespecified rather than chosen to maximize apparent performance on the validation cohort.

Threshold measures answer a different question from discrimination measures such as the area under a receiver operating characteristic curve: they describe performance at a particular decision boundary rather than ranking across many boundaries. They also differ from decision-curve analysis, which combines threshold-based classifications with specified relative consequences of false positives and false negatives. Reporting threshold measures alongside predicted-risk calibration and other relevant performance summaries gives a more complete evaluation than reporting any single measure alone.

Several checks help identify problems. Confirm that each prediction corresponds to the intended horizon and uses only information available at its prediction time. Check that the horizon outcome and event definition are applied consistently. Examine estimated censoring probabilities across relevant times and covariate patterns: values near zero create large weights and unstable estimates. Report how many outcomes were censored before the horizon and, where useful, summarize the distribution of weights or the impact of unusually large weights. Weight truncation can reduce instability but changes the estimator and may introduce bias; it should not be presented as a cure for severe positivity violations. Uncertainty intervals should account for the estimation process, including censoring-model estimation; resampling people and repeating the relevant fitting steps is one possible approach.

The formulas above assume a single event type. When the target is one event type in the presence of competing events, define the horizon outcome using the target cumulative incidence and treat competing events as observed competing outcomes, not as ordinary censoring. The event definition and estimator must match the cumulative-incidence estimand; treating competing events as if they were loss to follow-up can give misleading results.

Common Mistakes and Limits

• Calling a censored person event-free. No event observed before censoring does not establish that the person remained event-free through the horizon. Their horizon outcome is unknown unless the event occurred before censoring.
• Using the wrong denominator. Sensitivity divides by all observed event cases; PPV divides by all predicted positives. A statement about “the proportion detected” should identify which of these is intended.
• Ignoring the prediction horizon. A 12-month sensitivity is not interchangeable with a 6-month sensitivity. State the horizon and the time origin for the prediction.
• Using future information in predictions. A prediction for time \(t\) must be based on information available by \(t\). This restriction does not mean the censoring model must ignore later follow-up: appropriate post-prediction follow-up data may be used to estimate censoring probabilities for validation.
• Interpreting a high PPV as universal performance. PPV depends on event frequency in the evaluated population. A change in population can change PPV even if sensitivity and specificity are similar.

IPCW cannot recover information without assumptions. Misspecified censoring models, unmeasured predictors of both censoring and event time, or near-zero probabilities of remaining observed can undermine the estimates. Threshold metrics also discard information about how far predicted risks lie above or below the cutoff. They should be interpreted as summaries for a particular horizon, population, and decision threshold, not as complete evidence of model quality or as evidence that using the rule improves outcomes.