Why Metric Choice Changes the Question
A multi-omics classifier may produce a score or estimated probability for each person. Reporting a single number as “performance” can conceal what the model does well and where it fails. For example, a model can correctly rank most cases above controls yet perform poorly at a clinically chosen decision threshold. Or it can classify well at one threshold while producing inaccurate probabilities. These are different properties, and they call for different metrics.
Metric selection should follow the prediction target and deployment population, as established in the earlier material on Prediction Targets and Deployment Populations. In particular, outcome prevalence affects some measures directly, while unequal consequences of missed cases and false alarms may make overall accuracy a poor summary. No metric, by itself, establishes that a model is clinically useful.
Threshold-dependent metrics describe the classifications at a particular threshold. Ranking metrics describe how well scores order positive and negative subjects, without fixing a single threshold. Probabilistic scoring metrics evaluate the probabilities themselves. Before calculating any of them, the evaluation data must be held out from model fitting and metric-driven choices; reusing a held-out test set to select a model or metric can make reported performance optimistic.
Threshold-Specific Metrics
For a chosen threshold, sensitivity measures the share of event subjects identified, while specificity measures the share of non-event subjects correctly left negative. Precision, also called positive predictive value, is the share of predicted-positive subjects who are events. Accuracy is the share of all subjects classified correctly.
Sensitivity and specificity condition on the actual outcome class. Precision and accuracy depend on the mix of outcome classes in the evaluated population as well as on classifier performance. Thus, precision can change when prevalence changes, even if sensitivity and specificity remain the same. Balanced accuracy gives the two actual outcome classes equal weight:
Worked Example: A Rare Outcome at a Fixed Threshold
Suppose a hypothetical evaluation includes 200 subjects, of whom 20 experience the event and 180 do not. At a specified threshold, the classifier identifies 18 of the 20 event subjects, but also labels 36 non-event subjects positive. Thus \(TP=18\), \(FN=2\), \(FP=36\), and \(TN=144\). The rows and columns are consistent: \(18+2=20\) events, \(36+144=180\) non-events, and the total is 200.
Sensitivity is \(18/(18+2)=0.900\), or 90.0%. Specificity is \(144/(144+36)=0.800\), or 80.0%. Precision is \(18/(18+36)=18/54=0.333\), or 33.3%. Accuracy is \((18+144)/200=162/200=0.810\), or 81.0%. Balanced accuracy is \((0.900+0.800)/2=0.850\), or 85.0%.
A rule that predicts every subject negative would be 90.0% accurate because 180 of 200 subjects are non-events. Its sensitivity would be 0%, its specificity 100%, and its balanced accuracy 50%. Accuracy alone therefore makes that rule look better than the example classifier, despite its missing every event. Conversely, the example classifier’s sensitivity and specificity do not tell a clinician whether 36 false positives are acceptable. That judgment requires consequences and context, not a performance metric alone.
When false negatives and false positives have unequal clinical consequences, report measures that make those errors visible rather than relying on accuracy. Sensitivity and specificity can clarify the two error types; precision can show the proportion of positive alerts that are events in the target population. None assigns a clinical cost to an error. Expected utility or net benefit requires additional, explicit information about consequences and decisions; it cannot be inferred from these rates alone.
Ranking Metrics Under Imbalance
The area under the receiver operating characteristic curve (AUROC) summarizes discrimination across possible thresholds. In the absence of ties, it has a direct pairwise interpretation: the probability that a randomly selected positive subject receives a higher score than a randomly selected negative subject. A tied positive-negative pair contributes one-half. AUROC is not the probability that a prediction is correct, nor does it show the performance at any one threshold.
Precision-recall summaries focus on the positive class. At each score threshold, recall is sensitivity, and precision is the fraction of predicted positives that are true positives. One commonly used summary is average precision (AP). With distinct score values considered in descending order, let \(P_k\) be precision after including the first \(k\) subjects, and let \(R_k\) be recall at that point. Then:
This definition weights each precision value by the increase in recall at that step. Software may report other versions of PR-curve area, including interpolation-based areas, so the reported convention should be named. Unlike AUROC, AP is closely tied to prevalence: a non-informative ranking has expected precision-recall performance near the event prevalence, and the exact summary depends on its convention. Report prevalence alongside AP, especially when comparing populations. Neither AP nor AUROC specifies a clinical decision threshold.
Worked Example: Computing AUROC and Average Precision
Consider ten hypothetical subjects: four events and six non-events. Their scores, ordered from highest to lowest, have labels \(1,1,0,0,1,0,1,0,0,0\), where 1 denotes an event. There are \(4\times6=24\) positive-negative pairs. The first event score beats all six non-event scores, the second beats all six, the third beats four, and the fourth beats three. Thus 19 pairs are correctly ordered, so AUROC is \(19/24=0.7917\).
For AP, each event adds \(1/4=0.25\) to recall. The first two events occur at ranks 1 and 2, where precision is 1 for each. The third event is at rank 5, where precision is \(3/5=0.600\). The fourth is at rank 7, where precision is \(4/7=0.5714\). Therefore:
The event prevalence is \(4/10=0.400\). AUROC indicates how often a positive outranks a negative across pairs; AP summarizes precision as recall increases and should be read in light of that 40.0% prevalence. Neither result says how many subjects a particular clinical threshold would label positive.
Probabilistic Scoring: Brier Score and Log Loss
If predictions are intended as probabilities, evaluate their probability quality as well as their ranking or thresholded classifications. For observed outcomes \(y_i\in\{0,1\}\) and predicted probabilities \(\hat{p}_i\), the Brier score is the mean squared probability error. Log loss is the mean negative log probability assigned to the outcome that occurred. Smaller values are better for both.
A useful justification for these scores is that, for a subject whose true event probability is \(p\), the expected Brier loss from reporting \(q\) is minimized at \(q=p\). Since \(Y=1\) with probability \(p\) and \(Y=0\) with probability \(1-p\), this expectation is:
The final expression is minimized at \(q=p\). The expected log loss is \(-p\log(q)-(1-p)\log(1-q)\); its derivative with respect to \(q\) is \(-p/q+(1-p)/(1-q)\), which is zero at \(q=p\), and its second derivative \(p/q^2+(1-p)/(1-q)^2\) is positive for \(0<q<1\). Thus it too is minimized at the true probability. These are strictly proper scoring rules: in expectation, truthful probabilities outperform systematically different reports. In a finite test sample, however, one realized score does not prove that the probabilities are well calibrated.
Worked Example: Comparing Probability Scores
For four hypothetical subjects, outcomes are \(1,0,1,0\). Model A reports probabilities \(0.8,0.7,0.4,0.1\); model B reports \(0.7,0.2,0.6,0.3\). For model A, squared errors are \(0.04,0.49,0.36,0.01\), giving Brier score \((0.04+0.49+0.36+0.01)/4=0.225\). For model B, they are \(0.09,0.04,0.16,0.09\), giving \((0.09+0.04+0.16+0.09)/4=0.0950\). By Brier score, model B performs better on these four predictions.
Using natural logarithms, model A's log loss is \([-\log(0.8)-\log(0.3)-\log(0.4)-\log(0.9)]/4=0.6122\). Model B's is \([-\log(0.7)-\log(0.8)-\log(0.6)-\log(0.7)]/4=0.3618\). Log loss also favors model B on these observations. A probability of exactly 0 or 1 assigned to an outcome that does not occur gives infinite log loss; in practice, predictions may need principled numerical clipping, and the convention should be reported.
Choosing and Reporting a Complementary Set
Metrics answer different questions, so a defensible report usually includes more than one, selected before examining held-out test results. A practical distinction is:
| Metric family | What it summarizes | What it does not establish |
|---|---|---|
| Sensitivity, specificity, precision, accuracy, balanced accuracy | Classification performance at a specified threshold | Performance at other thresholds, probability accuracy, or clinical value |
| AUROC | Positive-negative ranking over thresholds | Precision in a target population or performance at one operating point |
| Average precision | Precision-recall behavior, emphasizing the positive class | A prevalence-independent measure or clinical utility |
| Brier score, log loss | Quality of predicted probabilities on a proper scoring scale | That a model has acceptable discrimination or clinical value |
For an imbalanced outcome, a useful report can pair AUROC with AP and prevalence, then add threshold-specific sensitivity, specificity, and precision if a threshold has been specified for the evaluation. If probabilities matter for downstream decisions, add a proper scoring rule such as Brier score or log loss. Comparisons require the same outcome definition and evaluation population; otherwise changes in prevalence or case mix may change the metric independently of model improvements. Report the threshold and the positive-class definition for threshold-specific results, and state the AP convention used.
A common mistake is to say, “AUROC is high, so the model will find nearly all cases with few false alarms.” That conclusion does not follow: AUROC averages ranking behavior across thresholds and does not determine the sensitivity, precision, or number of false positives at a particular threshold. The correction is to report the threshold-specific quantities relevant to the intended use, alongside a ranking metric when ranking is also of interest. Metric choice is not a substitute for considering the target population, uncertainty in estimates, or consequences of decisions.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A classifier has \(TP=12\), \(FN=8\), \(FP=30\), and \(TN=150\). Calculate sensitivity, precision, and accuracy.
- Why can precision fall when a classifier is applied in a lower-prevalence population, even if sensitivity and specificity do not change?
- What does an AUROC of 0.80 mean in pairwise terms, and what does it not tell you about a particular threshold?
- A non-informative ranking is evaluated where event prevalence is 0.05. Why is an AP near 0.05 a relevant reference point, and why should prevalence be reported?
- For a single subject with event probability \(p=0.25\), compare the expected Brier loss from reporting \(q=0.25\) and \(q=0.50\).
Solutions. (1) Sensitivity is \(12/(12+8)=0.600\); precision is \(12/(12+30)=0.286\); accuracy is \((12+150)/(12+8+30+150)=162/200=0.810\). (2) Among predicted positives, a lower share of subjects are events when prevalence falls, so precision can decrease even with unchanged class-conditional rates. (3) It means a randomly selected positive receives a higher score than a randomly selected negative about 80% of the time, counting ties as half; it does not specify sensitivity, precision, or false positives at a particular threshold. (4) For an uninformative ranking, precision-recall performance is near the positive-class prevalence, so AP should be interpreted against that baseline; reporting prevalence makes comparisons more meaningful. (5) Expected Brier loss is \((q-p)^2+p(1-p)\). At \(q=0.25\), it is \(0+0.25(0.75)=0.1875\); at \(q=0.50\), it is \((0.50-0.25)^2+0.1875=0.2500\). The true probability report has lower expected loss.