Why the Choice of Metric Matters
A classifier can appear highly accurate simply by predicting the common class. When a positive event is rare, that result may conceal poor detection of the cases a model is intended to identify. Even among classifiers, a single metric can favor a different model from another metric because the measures count errors differently.
For numerical risk predictions, the same issue arises in another form. Mean absolute error treats errors in direct proportion to their size, whereas root mean squared error gives relatively more weight to large errors. This tutorial develops the calculations and interpretations needed to compare these measures carefully. Its applications are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.
A confusion matrix organizes these four counts. Sensitivity measures the proportion of actual positives detected. Specificity measures the proportion of actual negatives correctly rejected. Accuracy is the proportion of all cases classified correctly. Balanced accuracy averages sensitivity and specificity, giving the two actual classes equal weight in the metric regardless of their proportions in the evaluation data.
How Class Prevalence Affects Accuracy
Let \(\pi\) be the proportion of actual positives in the evaluation population. If sensitivity is \(s\) and specificity is \(c\), the expected fraction correctly classified is the fraction of positives times the fraction of those detected, plus the fraction of negatives times the fraction of those correctly rejected. Thus,
This identity explains why accuracy depends on prevalence. In contrast, balanced accuracy is \((s+c)/2\), so it assigns one-half weight to each class. It does not estimate the fraction correct at the actual prevalence unless the classes have equal prevalence or the two class-specific rates happen to make the summaries agree. Neither measure is universally best: the choice depends on which errors matter and on the intended evaluation setting.
Identify the positive class and locate TP, FN, TN, and FP. Do not infer the positive class from which outcome is more common.
Use the actual-positive total, TP + FN, for sensitivity and the actual-negative total, TN + FP, for specificity.
Accuracy weights classes according to their prevalence; balanced accuracy gives the classes equal weight.
Worked Example: A High Accuracy From Mostly Negative Cases
A hypothetical screening rule is evaluated on 100 policy files. Twenty files have the event of interest and 80 do not. The rule identifies 12 of the 20 positive files and correctly labels 72 of the 80 negative files. Calculate sensitivity, specificity, accuracy, and balanced accuracy. Compare them with an all-negative rule.
Method. First fill in all four counts. There are 12 true positives, so the 8 remaining positive files are false negatives. There are 72 true negatives, so the 8 remaining negative files are false positives.
Sensitivity: \(12/(12+8)=12/20=0.6000\). The rule detects 60% of the positive files.
Specificity: \(72/(72+8)=72/80=0.9000\).
Accuracy: \((12+72)/100=84/100=0.8400\). As a check using prevalence, \(0.20(0.6000)+0.80(0.9000)=0.1200+0.7200=0.8400\).
Balanced accuracy: \((0.6000+0.9000)/2=0.7500\).
The all-negative rule has sensitivity \(0/20=0\) and specificity \(80/80=1\). Its accuracy is \(80/100=0.8000\), and its balanced accuracy is \((0+1)/2=0.5000\). The evaluated rule improves on the all-negative rule under both accuracy and balanced accuracy, but its sensitivity of 0.6000 also shows that it misses 8 of the 20 positive files.
Check: Each rate is between 0 and 1, and the sensitivity and specificity denominators sum to the corresponding actual-class totals, 20 and 80. The common mistake is to mistake the negative proportion, 0.80, for the specificity. The specificity is \(72/80=0.9000\); 0.80 is the prevalence weight used in the accuracy identity.
Threshold-Based Measures and AUC
Many classifiers produce a score, such as an estimated probability, rather than a class label directly. A threshold converts scores into labels: if the rule declares positive when the score is at least the threshold, changing the threshold changes TP, FN, TN, and FP. Sensitivity and specificity are therefore threshold-specific.
The receiver operating characteristic (ROC) curve plots sensitivity against the false-positive rate, \(1-\text{specificity}\), as the threshold varies. The area under this curve (AUC) summarizes ranking performance across thresholds. It is not the accuracy at a particular threshold, and it does not by itself specify which threshold to use.
The pairwise definition gives a direct calculation for a small sample. Compare every positive score with every negative score; count a positive score above a negative score as one win, an equal score as one-half, and a lower score as zero. Divide the total by the number of positive-negative pairs. The tie rule is necessary because a tied pair provides neither a strict ranking in favor of the positive case nor a strict ranking in favor of the negative case.
Worked Example: Computing AUC With Tied Scores
A hypothetical model assigns scores to three positive cases of \(0.90, 0.70, 0.70\) and four negative cases of \(0.80, 0.60, 0.70, 0.20\). Calculate AUC and compare two threshold choices: positive if the score is at least \(0.75\), or positive if it is at least \(0.70\).
AUC method. There are \(3(4)=12\) positive-negative pairs. The positive score \(0.90\) exceeds all four negative scores, contributing 4 wins. Each positive score of \(0.70\) exceeds \(0.60\) and \(0.20\), ties \(0.70\), and is below \(0.80\); each therefore contributes \(2+0.5=2.5\). The total is \(4+2.5+2.5=9\), so AUC is \(9/12=0.7500\).
At threshold \(0.75\), only the positive case scored \(0.90\) is classified positive, so TP \(=1\) and FN \(=2\). Among the negatives, the \(0.80\) score is a false positive and the other three are true negatives, so FP \(=1\) and TN \(=3\). Sensitivity is \(1/3=0.3333\), specificity is \(3/4=0.7500\), accuracy is \((1+3)/7=0.5714\), and balanced accuracy is \((0.3333+0.7500)/2=0.5417\), using unrounded rates in the calculation.
At threshold \(0.70\), all three positive cases are classified positive, so TP \(=3\) and FN \(=0\). Two negative cases have scores at least \(0.70\), so FP \(=2\) and TN \(=2\). Sensitivity is \(1.0000\), specificity is \(2/4=0.5000\), accuracy is \(5/7=0.7143\), and balanced accuracy is \((1+0.5)/2=0.7500\).
The AUC remains \(0.7500\) regardless of which one of these thresholds is selected; it summarizes ranking across thresholds. The common mistake is to call AUC the accuracy at the threshold. The calculations show that the accuracy at \(0.75\) is \(0.5714\), not \(0.7500\).
MAE and RMSE for Numerical Risk Predictions
For observed outcomes \(y_1,\ldots,y_n\) and predictions \(\hat y_1,\ldots,\hat y_n\), define the prediction error as \(e_i=y_i-\hat y_i\). Mean absolute error (MAE) averages the absolute errors. Root mean squared error (RMSE) takes the square root of the average squared errors. Both are nonnegative and use the same units as the outcome.
Squaring magnifies large errors relative to small ones: doubling an absolute error doubles its contribution to MAE but quadruples its contribution to the squared-error sum before the square root. Consequently, RMSE is more sensitive to a few large errors. That can be desirable when large misses are especially costly, but it can also make RMSE unstable when a small number of extreme observations dominate. MAE is less sensitive to such cases, but it does not make large misses irrelevant.
A fair comparison uses the same evaluation observations, outcome definition, and units for both models. If one model is scored on a different sample or a different target scale, its metric cannot be compared directly without accounting for that difference. Also, minimizing one metric need not minimize the other.
Worked Example: A Few Large Errors Reverse the Model Ranking
A hypothetical portfolio analyst compares two models on five evaluation cases. The signed errors \(y_i-\hat y_i\), measured in thousands of dollars, are \(0,0,0,0,10\) for Model A and \(3,3,3,3,3\) for Model B. Calculate MAE and RMSE for each model, then state which model each metric favors.
Method. Use absolute errors for MAE and squared errors for RMSE. For Model A, the absolute errors sum to \(0+0+0+0+10=10\), giving MAE \(=10/5=2\) thousand dollars. The squared errors sum to \(0+0+0+0+100=100\), so RMSE \(=\sqrt{100/5}=\sqrt{20}=4.4721\) thousand dollars, rounded to four decimal places.
For Model B, the absolute errors sum to \(3+3+3+3+3=15\), giving MAE \(=15/5=3\) thousand dollars. The squared errors sum to \(9+9+9+9+9=45\), so RMSE \(=\sqrt{45/5}=\sqrt{9}=3\) thousand dollars.
Model A has the smaller MAE, \(2\) versus \(3\), while Model B has the smaller RMSE, \(3\) versus \(4.4721\). Thus, choosing a model requires deciding whether typical absolute error or stronger penalization of large errors better reflects the objective. The common mistake is to say that a model with smaller MAE must also have smaller RMSE; the calculation disproves that claim.
Check: All errors and both metrics are in thousands of dollars. Model A's one error of 10 contributes \(10/5=2\) to MAE but contributes \(100\) to the squared-error sum, explaining the change in ranking.
Choosing and Reporting a Metric
Metric selection should follow the prediction task, not the most flattering result. For an imbalanced classifier, report class-specific rates as well as an overall summary when the costs or consequences of missed positives and false alarms differ. Accuracy is useful when the evaluation prevalence is relevant and the fraction of all correct classifications is the target. Balanced accuracy is informative when equal emphasis on sensitivity and specificity is intended. AUC is useful for comparing score rankings across thresholds, but a chosen operating threshold still needs to be assessed using the class-specific rates and relevant costs.
For numerical predictions, MAE gives an interpretable average absolute miss, while RMSE emphasizes large misses. Neither metric alone explains why errors occur, and neither replaces examination of the evaluation design. In all cases, calculate metrics on data appropriate for assessing performance rather than relying on the training fit as if it were an independent estimate.
Check Your Understanding
For each problem, calculate the requested quantity from the stated evaluation data.
- A hypothetical alert model is tested where positive events have prevalence \(0.05\). Its sensitivity is \(0.80\) and its specificity is \(0.90\). Calculate accuracy and balanced accuracy. (Answer: accuracy \(=0.05(0.80)+0.95(0.90)=0.8950\); balanced accuracy \(=(0.80+0.90)/2=0.8500\).)
- For a binary classification sample, TP \(=24\), FN \(=6\), FP \(=30\), and TN \(=240\). Calculate sensitivity, specificity, accuracy, and balanced accuracy. (Answer: sensitivity \(=24/30=0.8000\); specificity \(=240/270=0.8889\); accuracy \(=(24+240)/300=0.8800\); balanced accuracy \(=(0.8000+0.8889)/2=0.8444\), rounded to four decimals.)
- Two positive cases have scores \(0.80,0.50\), and three negative cases have scores \(0.70,0.50,0.20\). Calculate AUC, counting tied pairs as one-half. (Answer: positive-negative pair wins total \(=3+(1+0.5)=4.5\) out of \(2(3)=6\), so AUC \(=0.7500\).)
- On four cases, a model's absolute errors are \(1,1,1,7\). Calculate MAE and RMSE in the outcome's units. (Answer: MAE \(=(1+1+1+7)/4=2.5000\); RMSE \(=\sqrt{(1+1+1+49)/4}=\sqrt{13}=3.6056\), rounded to four decimals.)
- A rule predicts every case negative in an evaluation population with positive prevalence \(0.03\). Calculate accuracy and balanced accuracy. (Answer: accuracy \(=0.97\); sensitivity \(=0\), specificity \(=1\), and balanced accuracy \(=(0+1)/2=0.5000\).)