Sensitivity and Specificity of Safety Classifiers

Why Evaluate a Safety Classifier at a Clinical Threshold?

A toxicity classifier turns information available at a dosing decision into a safety signal. For example, it might flag a patient for additional monitoring when a predicted risk score exceeds a chosen cutoff. Evaluation must answer two distinct questions: among patients who experience the defined toxicity, how often did the classifier flag them; and among patients who do not experience it, how often did the classifier leave them unflagged? These are measured by sensitivity and specificity.

The cutoff is not merely a technical setting. A low cutoff may catch more patients who will experience toxicity, while also triggering more alerts for patients who will not. The clinical consequences depend on what follows an alert: an alert might prompt a low-burden laboratory check, or it might lead to a disruptive dose reduction. The rates describe classification performance; they do not, by themselves, establish whether an alert policy improves outcomes.

The reference outcome and time window must be set before interpreting either rate. “Toxicity” could mean a particular adverse event, a laboratory abnormality meeting a defined criterion, or a composite endpoint; these are not interchangeable. Similarly, a prediction of toxicity within 7 days is evaluated against the 7-day outcome, not an unspecified lifetime outcome. The classifier inputs must also be available at the intended decision time. Otherwise, the evaluation can inadvertently use information that would not have been known when the alert was supposed to operate.

Computing and Interpreting the Rates

Sensitivity conditions on observed toxicity; specificity conditions on its absence. Their denominators are different, so the metrics answer different questions. A sensitivity of 0.85 means that 85% of the patients who experienced the defined toxicity were flagged at this threshold. A specificity of 0.80 means that 80% of patients who did not experience it were not flagged. Neither statement gives the probability that a flagged patient will experience toxicity. That is a predictive-value question, and it depends on the event prevalence in the evaluated population.

The corresponding error rates within each outcome group are complements:

These identities follow because, among toxicity cases, every patient is either a true positive or a false negative, so \(TP+FN\) is the full number of cases. Therefore \(TP/(TP+FN)+FN/(TP+FN)=1\). Among non-cases, \(TN+FP\) is the full number of non-cases, giving the analogous result for specificity and false-positive rate. Thus, a sensitivity of 0.85 implies that 15% of observed toxicity cases were missed; a specificity of 0.80 implies that 20% of non-cases received an alert.

Threshold Choice and the Missed-Case Tradeoff

For a score where larger values indicate greater toxicity risk and alerts occur when \(S \geq c\), lowering \(c\) cannot remove an alert from a patient who was already flagged. It can add alerts among patients whose scores fall between the old and new cutoffs. On the same set of scored patients, lowering the threshold therefore cannot decrease sensitivity and cannot increase specificity. With discrete scores or few observations, the rates may stay unchanged across a range of thresholds and then change in steps.

This is a mathematical property of nested alert groups, not a guarantee that the lower threshold is clinically better. More sensitive classification can reduce missed toxicity while creating more false alerts, follow-up work, and possible unnecessary interventions. The relevant tradeoff depends on the clinical action and the relative consequences of a missed case and an unnecessary alert. A threshold should therefore be justified in relation to that action, not chosen solely to maximize a generic summary such as overall accuracy.

A threshold chosen after inspecting performance in the same data can look better than it will perform in new patients. If several thresholds are tried and the one with the most attractive observed rates is selected, the reported rates are partly optimized to random features of that dataset. A stronger evaluation specifies the threshold using clinical reasoning or development data, then assesses it in a separate validation cohort. If a threshold must be selected during development, the final evaluation should account for that selection rather than presenting the best development result as independent validation.

Uncertainty, Diagnostics, and Failure Modes

Sensitivity and specificity are estimates when calculated from a finite validation sample. Their precision depends on the relevant denominators: the number of toxicity cases for sensitivity and the number of non-cases for specificity. A large total cohort does not ensure a precise sensitivity if only a small number of patients experience toxicity. Report the numerator and denominator alongside each estimate, and use an appropriate interval to show sampling uncertainty. For an independent binomial sample, a Wilson interval is one commonly used interval for a proportion; its assumptions and the sampling design must fit the data.

For example, a Wilson 95% interval for a sample proportion \(\hat p=x/n\) can be calculated using \(z=1.96\):

This interval procedure treats the observed labels as outcomes from an independent binomial sampling process. Clustered or repeated observations, for example multiple records from the same patient, require an approach that accounts for that dependence. An interval quantifies sampling uncertainty under its assumptions; it does not correct a biased cohort, an inaccurate reference assessment, or a poorly defined toxicity endpoint.

Several checks help identify an evaluation that may be misleading:

• Verify the denominators. The sensitivity denominator must include all eligible toxicity cases, and the specificity denominator all eligible non-cases. Missing classifier scores or outcomes can alter those groups; report exclusions and how they were handled.
• Check the reference outcome and timing. Outcome ascertainment should be applied consistently. If only alerted patients receive definitive follow-up, missed toxicity may go undetected, biasing the estimated sensitivity.
• Check the threshold and score direction. Confirm that the intended positive rule is applied consistently, including how scores exactly equal to the cutoff are treated.
• Check the evaluation population. Case mix, clinical practice, monitoring intensity, and event definitions can differ between development and deployment settings. Sensitivity and specificity may not transport unchanged.
• Examine counts as well as percentages. A reported sensitivity of 100% based on a handful of cases is weak evidence of reliable detection. Show the numerator, denominator, and uncertainty.

What These Metrics Do—and Do Not—Tell You

Sensitivity and specificity complement but do not replace other evaluation approaches. Overall accuracy, \((TP+TN)/N\), can be dominated by the larger outcome group and conceal poor performance for the less common group. Balanced accuracy, the average of sensitivity and specificity, gives the two rates equal numerical weight, but that weighting need not match the clinical consequences. A receiver operating characteristic curve summarizes sensitivity against false-positive rate across thresholds; it can describe discrimination across cutoffs but does not select a clinically appropriate action threshold. Predictive values answer different questions and change with event prevalence, so sensitivity and specificity cannot be used as substitutes for them.

Nor do these rates establish calibration, clinical utility, or causal benefit from using the classifier. A classifier can have useful sensitivity and specificity yet prompt actions that do not improve patient outcomes. Conversely, a modest rate may still support a useful safety net if the follow-up action is low burden and the missed-event consequences are substantial. Evaluation should connect the rate estimates to the exact alert-triggered policy and its intended use.