Why Report Performance by Subgroup?
A model’s overall performance can look acceptable while its errors are concentrated among a particular part of the population. That concentration may matter operationally: one group could face more missed referrals, larger estimates of claim cost, or more delays in receiving service. Reporting performance by subgroup makes those differences visible, but only when the groups, metrics, and denominators are clearly defined.
A subgroup comparison is a descriptive evaluation on a specified data set. It does not, by itself, explain why results differ, establish that a model caused an outcome, or prove that a difference will persist in future use. A careful report identifies the evaluated population, gives the number of observations in each group, and presents metrics that match the decision being considered. This reporting practice is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Choose Metrics and Denominators Deliberately
For classification, each subgroup can have its own confusion matrix at the chosen classification threshold. A confusion matrix counts true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN). These counts support different measures, and each measure answers a different question.
- Recall is TP divided by TP + FN. Its denominator is the actual positive cases; it describes the share of those cases identified as positive.
- Specificity is TN divided by TN + FP. Its denominator is the actual negative cases.
- Precision is TP divided by TP + FP. Its denominator is the cases predicted positive.
- Accuracy is (TP + TN) divided by the total number of observations. It is the share classified correctly.
These are not interchangeable. A report about missed eligible applicants, for example, should not rely on precision alone: precision says what share of referrals are truly eligible, not what share of eligible people the model misses. When comparing groups, state the denominator in words as well as showing the formula or counts.
For a quantitative prediction, common subgroup measures include mean absolute error (MAE) and root mean squared error (RMSE). For observations \(i=1,\ldots,n\), let \(y_i\) be the observed value and \(\hat{y}_i\) the predicted value. MAE averages the absolute prediction errors, while RMSE takes the square root of the average squared errors. Both are in the units of the outcome. RMSE gives greater influence to larger errors.
A measure that is appropriate overall may not capture every subgroup’s operational concern. For instance, average dollar error does not show how often a model underestimates cost. The report should use the measures that answer the stated question, while avoiding a long list of metrics with no explanation of their relevance.
Worked Example: Comparing Classification Results
Worked Example: Comparing Classification Results
A hypothetical service program evaluates a referral model on 300 cases. It reports results for two regions at the same fixed threshold. In Region North, TP = 36, FN = 9, FP = 15, and TN = 140. In Region South, TP = 18, FN = 12, FP = 10, and TN = 60. Calculate subgroup sizes, recall, specificity, precision, and accuracy, then compare them with the combined results.
Method. Calculate each metric from the counts in its own region, then add the corresponding counts before calculating a combined metric. This avoids confusing the metric’s denominator with the subgroup’s total size.
North. The subgroup size is \(36+9+15+140=200\). Actual positive cases number \(36+9=45\), and actual negative cases number \(15+140=155\). Predicted positives number \(36+15=51\). Therefore:
South. The subgroup size is \(18+12+10+60=100\). Actual positive cases number \(18+12=30\), actual negative cases number \(10+60=70\), and predicted positives number \(18+10=28\). Thus recall is \(18/30=0.6000\), specificity is \(60/70=0.8571\), precision is \(18/28=0.6429\), and accuracy is \((18+60)/100=0.7800\).
Combined results. The total size is \(200+100=300\). The combined counts are TP \(=36+18=54\), FN \(=9+12=21\), FP \(=15+10=25\), and TN \(=140+60=200\). Thus combined recall is \(54/(54+21)=54/75=0.7200\); combined specificity is \(200/(200+25)=200/225=0.8889\); combined precision is \(54/(54+25)=54/79=0.6835\); and combined accuracy is \((54+200)/300=254/300=0.8467\).
The results show lower recall and accuracy in South on this evaluation sample. If a false negative means a person who might have qualified is not referred, the recall difference could be relevant to access; false positives could also create extra review work. The counts alone do not establish why the metrics differ or how they would behave after deployment. The common mistake avoided here is comparing correct classifications against each region’s total and calling that recall; recall uses actual positive cases as its denominator.
Aggregate Metrics Must Respect the Subgroup Sizes
An overall metric is not generally the simple average of subgroup metrics. For an additive error measure such as MAE, the combined value is a size-weighted average of the subgroup values. For RMSE, first combine the subgroup sums of squared errors and observation counts, then take the square root. Averaging subgroup RMSE values directly does not generally give the overall RMSE.
Why the MAE formula holds. In subgroup \(g\), the sum of absolute errors is \(n_gM_g\), because the subgroup MAE is that sum divided by \(n_g\). The sum of absolute errors across all disjoint subgroups is therefore \(\sum_g n_gM_g\). Dividing this total by the overall count \(\sum_g n_g\) gives the stated combined MAE. For RMSE, the subgroup sum of squared errors is \(n_gR_g^2\); adding those sums and dividing by the total count gives the combined mean squared error, whose square root is the combined RMSE.
Worked Example: Combining Regression Errors Correctly
A hypothetical claims team evaluates a cost prediction on six cases. For four rural cases, the signed errors \(\hat{y}_i-y_i\), in thousands of dollars, are \(-2, 2, -4, 4\). For two urban cases, they are \(0, 10\). Calculate each subgroup’s MAE and RMSE and the overall values.
Method. Take absolute values to compute MAE and squares to compute RMSE. Combine error totals and counts, rather than giving each subgroup equal weight.
Rural. Absolute errors are \(2,2,4,4\), with sum \(2+2+4+4=12\); hence MAE is \(12/4=3\) thousand dollars. Squared errors are \(4,4,16,16\), with sum \(40\); hence RMSE is \(\sqrt{40/4}=\sqrt{10}=3.1623\) thousand dollars, rounded to four decimals.
Urban. Absolute errors are \(0,10\), with sum \(10\); hence MAE is \(10/2=5\) thousand dollars. Squared errors are \(0,100\), with sum \(100\); hence RMSE is \(\sqrt{100/2}=\sqrt{50}=7.0711\) thousand dollars, rounded to four decimals.
Overall. The total absolute error is \(12+10=22\) thousand dollars across six cases, giving MAE \(=22/6=3.6667\) thousand dollars, rounded to four decimals. The total squared error is \(40+100=140\), so overall RMSE is \(\sqrt{140/6}=\sqrt{23.3333\ldots}=4.8305\) thousand dollars, rounded to four decimals.
The overall MAE can also be checked by weighting the subgroup MAEs: \((4(3)+2(5))/6=22/6=3.6667\). In contrast, the unweighted average subgroup MAE is \((3+5)/2=4\), which is not the overall MAE. The common mistake is to average subgroup RMSEs or MAEs without regard to subgroup sizes. The rural and urban results may also imply different review or budgeting needs, but these six cases are not enough to establish a stable operational pattern.
Report Sample Size and Uncertainty With the Estimate
A subgroup percentage without its denominator can be misleading. A recall of 80% based on 10 actual positive cases is less precise than the same observed recall based on 1,000 such cases. Report both the subgroup’s total size and, when the metric has a more specific denominator, that denominator too. For recall, for example, report the number of actual positive cases, not only the total number of records.
For an initial description of uncertainty in a subgroup recall, suppose validation cases are independent and each actual positive case has the same probability \(p\) of being identified as positive. If \(x\) of \(m\) actual positive cases are identified, the estimated recall is \(\hat{p}=x/m\). A common approximate standard error is \(\sqrt{\hat{p}(1-\hat{p})/m}\). An approximate 95% Wald interval is \(\hat{p}\) plus or minus 1.96 standard errors. This approximation can perform poorly with small denominators or estimates near zero or one; it should not be presented as an exact guarantee.
Worked Example: Interpreting a Recall Difference With Uncertainty
A hypothetical screening review finds 40 detected cases among 50 actual positive cases in Group A, and 12 detected among 20 actual positive cases in Group B. Calculate recall and approximate 95% Wald intervals for both groups, then assess whether the observed difference is conclusive.
Method. Use the actual positive count as the denominator for recall. Calculate each standard error using its own denominator, then calculate the interval as estimate \(\pm 1.96\) standard errors. The two groups are treated as independent for this approximate calculation.
Group A. Recall is \(40/50=0.8000\). The estimated standard error is \(\sqrt{0.8(1-0.8)/50}=\sqrt{0.16/50}=\sqrt{0.0032}=0.0566\), rounded to four decimals. The margin is \(1.96(0.0566)=0.1109\), rounded to four decimals. The approximate interval is \(0.8000\pm0.1109\), or \([0.6891,0.9109]\).
Group B. Recall is \(12/20=0.6000\). The standard error is \(\sqrt{0.6(1-0.6)/20}=\sqrt{0.24/20}=\sqrt{0.012}=0.1095\). The margin is \(1.96(0.1095)=0.2147\). The approximate interval is \(0.6000\pm0.2147\), or \([0.3853,0.8147]\), with probabilities rounded to four decimals.
The observed recall difference, Group A minus Group B, is \(0.8000-0.6000=0.2000\). The intervals overlap substantially, and Group B’s estimate is less precise because it has fewer actual positive cases. These approximate intervals do not establish that the population recalls differ or are equal. The common mistake is to interpret the 20-percentage-point observed gap as a proven, stable gap while ignoring the denominators and sampling variability.
Explain the Operational Meaning Without Overclaiming
A useful subgroup report connects measured errors to plausible consequences, while distinguishing what the data show from what remains uncertain. For example, a lower recall may mean that more actual positive cases were missed in the evaluation sample. Depending on the setting, those misses could delay service, reduce access, or increase later costs. A higher false-positive rate may require more manual review, create unnecessary interventions, or consume limited capacity. These are possible implications to investigate, not outcomes proven by a confusion matrix alone.
Use careful language: “The validation sample showed lower recall for Group B” is supported by the calculation. “The model caused lower access for Group B” requires evidence beyond a subgroup metric. Also specify whether subgroup assignment was known at the time of prediction and whether the evaluation data represent the population in which decisions will be made.
State the subgroup criteria, evaluation period, and who is included or excluded.
For each metric, state what error or performance question it answers and give its denominator.
Report subgroup sizes and the metric-specific counts; describe uncertainty in proportion to the available data.
Explain possible service, cost, or access implications, and separate observed evidence from explanations or causal claims.
A balanced report does not hide an unfavorable subgroup result, but it does not overstate one either. Present the subgroup estimates beside the overall result, identify which differences matter for the decision, and disclose limitations such as small denominators or a validation population that may not match future users. The goal is a fair account of model performance that supports informed follow-up, not a single ranking that erases different experiences.
Check Your Understanding
For each problem, calculate the requested measure and interpret it in the stated context.
- A hypothetical routing model has 24 TP, 6 FN, 8 FP, and 62 TN for one group. What are the group size, recall, and precision? (Answer: size \(=100\); recall \(=24/30=0.8000\); precision \(=24/32=0.7500\).)
- Two subgroups have MAEs of 2 and 6 dollars, based on 30 and 10 cases respectively. What is their combined MAE? (Answer: \((30(2)+10(6))/40=3.00\) dollars.)
- For two groups, the sums of squared errors are 72 over 8 cases and 50 over 2 cases. What is the combined RMSE? (Answer: \(\sqrt{(72+50)/(8+2)}=\sqrt{12.2}=3.4928\) dollars, if errors are in dollars.)
- A subgroup has 18 true positives and 12 false negatives. Calculate recall and explain its denominator. (Answer: \(18/(18+12)=0.6000\); the denominator is the 30 actual positive cases.)
- A group’s observed recall is 0.75 based on 40 actual positive cases. Using the approximate standard-error formula, calculate its standard error and 95% Wald interval. (Answer: standard error \(=\sqrt{0.75(0.25)/40}=0.0685\); interval \(=0.75\pm1.96(0.0685)\), approximately \([0.6157,0.8843]\).)