Evaluating Rare-Outcome Predictions

Why Rare Outcomes Need Careful Evaluation

When a genomic classifier predicts an uncommon outcome, a high overall accuracy can hide poor detection of the cases that matter. A classifier that labels every sample negative is 99% accurate when prevalence is 1%, yet it finds no positive cases. Conversely, even a classifier with useful discrimination can have modest precision: among the samples it flags, many may be false positives because the negative group is much larger.

Evaluation should therefore describe performance in terms that expose both missed cases and false alarms. Precision–recall curves focus on the fraction of flagged samples that are truly positive and the fraction of all positive samples that are detected. Sensitivity at a specified specificity answers a related operational question: how many positive cases are detected while limiting false positives among negative cases? These quantities are meaningful only when the evaluation samples represent the prevalence and sampling process to which the results are intended to apply.

Assume a binary outcome \(Y\), with \(Y=1\) denoting the rare outcome and \(Y=0\) its absence. Each evaluation unit has an observed label and a model score, with larger scores indicating stronger evidence for \(Y=1\). The model and any threshold used for evaluation must be fixed without using the evaluation labels. In cross-validation, this means using predictions made for held-out units by a pipeline fitted without those units; the earlier material on leakage-safe pipelines and nested cross-validation explains how to preserve that separation.

Confusion Counts and Prevalence

Sensitivity and specificity condition on the true class. Precision instead conditions on a positive prediction. The distinction matters when the positive class is rare: sensitivity and specificity alone do not determine precision unless prevalence is also known.

Precision–Recall Curves and Average Precision

A precision–recall curve shows precision against recall as the score threshold varies. Here recall is the same quantity as sensitivity. Starting with a threshold above every score yields no positive predictions, so precision is undefined at that endpoint; the curve is usually displayed from recall zero with a conventional endpoint. As the threshold is lowered, more samples are flagged, recall cannot decrease, and precision may rise or fall.

A common summary is average precision (AP). With distinct score groups considered from highest to lowest, let \(P_k\) be the precision after including score group \(k\), and let \(\Delta R_k\) be the increase in recall contributed by that group. The non-interpolated grouped definition is \(AP=\sum_k \Delta R_k P_k\). Thus each group’s precision is weighted by the fraction of all positive samples newly recovered at that step. Samples tied at a score are included as a group; arbitrarily ordering tied samples can otherwise create spurious changes in the curve.

AP is not necessarily equal to the trapezoidal area under a precision–recall curve. Different software may also use distinct interpolation conventions, so a report should identify the summary definition used. In particular, the grouped calculation below uses unrounded precisions and recall increments.

Why the Prevalence Baseline Matters

For a classifier that predicts every evaluation sample positive, recall is 1 and precision equals the evaluation prevalence. This gives a natural reference level for precision: unlike the diagonal reference sometimes used for ROC curves, the precision baseline changes with prevalence. A precision–recall curve should therefore be read alongside the positive fraction in the evaluation set.

The dependence can be derived directly. Let \(\pi=P(Y=1)\) denote prevalence in the population being evaluated, and suppose a threshold has sensitivity \(s\) and false-positive rate \(f=1-\text{specificity}\). Among a large set of cases, the fraction that are true positives is \(\pi s\), while the fraction that are false positives is \((1-\pi)f\). Therefore the fraction of positive predictions that are true is

This is the population relationship; in a finite evaluation set, the observed precision is calculated from its counts. The derivation shows why preserving prevalence is essential when interpreting precision. Changing the class mix changes the denominator even if sensitivity and specificity remain the same.

Sensitivity at a Fixed Specificity

Some applications specify a tolerable false-positive rate. Sensitivity at a fixed specificity reports the fraction of positive samples detected at an operating point constrained by that requirement. For example, an evaluation may report sensitivity at 95% specificity. To make this an honest held-out estimate, the threshold must be chosen without using the evaluation labels—for instance, using training data or an inner validation procedure—and then applied unchanged to the held-out scores.

A stated specificity target does not guarantee that an estimated test-set specificity will equal the target exactly. Specificity changes in increments of \(1/N_-\), where \(N_-\) is the number of negative evaluation samples, and the selected threshold’s true population specificity is uncertain. The report should give the observed specificity and the number of positive and negative evaluation samples, not just the target.

Preserving Evaluation Prevalence and Diagnosing Failure

If evaluation samples are downsampled or oversampled to balance the classes, observed precision and the precision–recall curve describe that altered sample mix, not the target population’s prevalence. The default is to leave the evaluation set’s class composition intact. Class weighting or resampling may be used during model fitting when appropriate, but evaluation predictions should be assessed on samples drawn to represent the intended evaluation population. If evaluation sampling intentionally changes prevalence, precision must be re-expressed for a clearly specified target prevalence using valid sampling information; a balanced-set precision should not simply be reported as population precision.

Rare outcomes also make metrics unstable. With few positive samples, each additional detected case changes recall by \(1/N_+\), often substantially. A small number of false positives can sharply reduce precision when the number of true positives is small. Report the positive and negative counts, use uncertainty methods appropriate to the validation design, and avoid treating small differences in AP as decisive when the estimate is noisy. In cross-validation, pooled held-out predictions and an average of fold-level metrics answer different questions, particularly when folds contain different positive counts.

ROC curves summarize sensitivity against false-positive rate and can be useful for ranking discrimination across thresholds, but a low false-positive rate can still produce many false positives when negatives vastly outnumber positives. Accuracy is prevalence-sensitive and can be dominated by the majority class. A single threshold metric such as F1 combines precision and recall but does not enforce a specified specificity and does not use true negatives directly. These summaries answer different questions; none removes the need to report class counts, prevalence, and a metric aligned with the intended use.

Common Mistakes

• Calling a high accuracy good rare-outcome detection. Accuracy can be high when a model mostly predicts the majority class. Report sensitivity and precision or a threshold-independent ranking summary as appropriate.
• Comparing precision across differently balanced evaluation sets as if prevalence were equal. Precision changes with prevalence even when sensitivity and specificity are fixed. Preserve the target evaluation prevalence or explicitly adjust to a stated target population.
• Choosing the threshold on the held-out labels. That makes the reported fixed-specificity performance optimistic. Set the threshold using data separate from the held-out evaluation and apply it without adjustment.
• Rounding each precision before calculating AP. Small rounding errors can alter the final reported AP. Use exact counts or unrounded values through the weighted sum, then round the final result.
• Splitting tied scores into an arbitrary order. A score does not distinguish samples tied at that value. Include the tied group together when constructing the curve and calculating grouped AP.