Why Performance Must Be Summarized Across Cohorts Carefully
A classifier evaluated on several cohorts can have a single pooled score, a score for each cohort, or an average of the cohort scores. Those summaries answer different questions. The pooled score describes performance over all evaluated subjects together, so large cohorts and common outcome classes contribute more. An unweighted average gives each cohort the same influence, regardless of size. Neither summary reveals the whole pattern by itself: a high pooled score can coexist with substantially worse performance in a smaller cohort.
The calculations below assume that each subject belongs to one cohort and has an observed binary outcome and a prediction made by the evaluation procedure. The outcome definition and prediction rule, including any classification threshold, should be consistent across cohorts. Predictions must come from an evaluation design that keeps test outcomes from influencing model fitting or threshold choice. These summaries describe the evaluated subjects; by themselves they do not quantify sampling uncertainty or establish performance in a different target population.
For a pooled summary, add the counts across cohorts first, then calculate the metric from those totals. This is often called a micro-averaged summary: each subject contributes to the pooled counts. A macro average instead calculates each cohort's metric and averages those cohort values with equal cohort weights, among cohorts where the metric is defined. State which summary is being reported; the word “average” alone is not precise enough.
What the Pooled Metric Weights
The weighting behind a pooled metric follows from its denominator. For sensitivity, the relevant denominator is the number of actual positives in each cohort. For specificity, it is the number of actual negatives. Accuracy uses cohort size, and precision uses the number of positive predictions. Consequently, pooled sensitivity is not generally the same as an unweighted mean of cohort sensitivities, nor is pooled precision generally an unweighted mean of cohort precisions.
This identity applies when the pooled denominator is positive. The sum on the right includes only cohorts with at least one actual positive. A cohort with no actual positives has undefined sensitivity; it contributes zero to the pooled true-positive and actual-positive totals, but it must not be assigned a sensitivity or included as a weighted term. The same restriction applies to each of the corresponding identities below.
Each expression is defined only when its pooled denominator is positive. Cohorts with zero denominator for that metric contribute zero to the pooled numerator and denominator, but their cohort-specific metric is undefined and they are omitted from the weighted-average identity. In particular, an empty cohort has no accuracy to average. Reporting “not defined” for such a cohort is more informative than silently assigning it a value.
Worked Counts and Two Different Summaries
Worked Example: Pooled Metrics and Macro Averages
Suppose two hypothetical cohorts have the following confusion counts. The same outcome definition and classification threshold were used in both, and the evaluation predictions were generated without using test outcomes to fit the model or choose the threshold.
| Cohort | TP | FN | TN | FP | Actual positives | Actual negatives |
|---|---|---|---|---|---|---|
| A | 72 | 18 | 180 | 20 | 90 | 200 |
| B | 18 | 12 | 63 | 27 | 30 | 90 |
For cohort A, sensitivity is \(72/(72+18)=72/90=0.8000\), specificity is \(180/(180+20)=180/200=0.9000\), accuracy is \((72+180)/290=252/290=0.8690\), and precision is \(72/(72+20)=72/92=0.7826\).
For cohort B, sensitivity is \(18/(18+12)=18/30=0.6000\), specificity is \(63/(63+27)=63/90=0.7000\), accuracy is \((18+63)/120=81/120=0.6750\), and precision is \(18/(18+27)=18/45=0.4000\).
The pooled counts are \(TP=72+18=90\), \(FN=18+12=30\), \(TN=180+63=243\), and \(FP=20+27=47\). Therefore pooled sensitivity is \(90/(90+30)=90/120=0.7500\); pooled specificity is \(243/(243+47)=243/290=0.8379\); pooled accuracy is \((90+243)/410=333/410=0.8122\); and pooled precision is \(90/(90+47)=90/137=0.6569\).
For comparison, the unweighted macro sensitivity is \((0.8000+0.6000)/2=0.7000\); macro specificity is \((0.9000+0.7000)/2=0.8000\); macro accuracy is \((0.8690+0.6750)/2=0.7720\); and macro precision is \((0.7826+0.4000)/2=0.5913\), with rounding. These are different summaries, not competing calculations of the same quantity. For example, pooled sensitivity weights the two cohort sensitivities by their actual-positive counts, 90 and 30, so \(0.75=(90/120)(0.80)+(30/120)(0.60)\). Macro sensitivity gives both cohorts equal weight.
How Prevalence Changes Some Metrics
Outcome prevalence in a cohort is the proportion of subjects with positive outcomes. Accuracy and precision depend on prevalence as well as on the classifier's sensitivity and specificity. If sensitivity is \(Se\), specificity is \(Sp\), and prevalence is \(\pi\), then the expected fraction classified correctly is \(\pi Se+(1-\pi)Sp\). Precision, the fraction of positive predictions that are truly positive, is \(\pi Se/[\pi Se+(1-\pi)(1-Sp)]\), provided the denominator is positive. These relationships explain why accuracy and precision can differ across cohorts even when sensitivity and specificity are unchanged.
Worked Example: Equal Sensitivity and Specificity, Different Prevalence
Consider two hypothetical cohorts of 200 subjects each. In both, sensitivity is \(0.8000\) and specificity is \(0.9000\), but their positive-outcome prevalences differ.
Cohort A has 20 positive and 180 negative outcomes. Its counts are \(TP=16\), \(FN=4\), \(TN=162\), and \(FP=18\). Accuracy is \((16+162)/200=178/200=0.8900\). Precision is \(16/(16+18)=16/34=0.4706\).
Cohort B has 160 positive and 40 negative outcomes. Its counts are \(TP=128\), \(FN=32\), \(TN=36\), and \(FP=4\). Accuracy is \((128+36)/200=164/200=0.8200\). Precision is \(128/(128+4)=128/132=0.9697\).
Thus the same sensitivity and specificity do not imply the same accuracy or precision. The prevalence is \(20/200=0.1000\) in A and \(160/200=0.8000\) in B. For example, in A the expected accuracy from the prevalence formula is \((0.10)(0.80)+(0.90)(0.90)=0.8900\), matching the count-based result. In B it is \((0.80)(0.80)+(0.20)(0.90)=0.8200\). Precision is much higher in B because true positives are common relative to false positives.
Pooling gives \(TP=144\), \(FN=36\), \(TN=198\), and \(FP=22\). Pooled sensitivity is \(144/180=0.8000\), specificity is \(198/220=0.9000\), accuracy is \(342/400=0.8550\), and precision is \(144/166=0.8675\). The pooled accuracy is the size-weighted average of the cohort accuracies; here the sizes are equal, so it is \((0.8900+0.8200)/2=0.8550\). Pooled precision is not the unweighted mean of the two cohort precisions because the cohorts have different numbers of positive predictions.
When a Strong Pooled Score Hides Heterogeneity
A pooled score can be useful when the target quantity is performance over the combined set of evaluated subjects. It can nevertheless obscure poor performance in a cohort that is smaller, has a different outcome mix, or is otherwise clinically important. Cohort-specific results help reveal whether performance is consistent enough for the intended use. Differences may reflect chance variation, prevalence, measurement practices, case mix, or a real change in how the model performs; descriptive summaries alone do not establish which explanation is responsible.
Worked Example: A High Pooled Accuracy With a Poor Cohort
Suppose a hypothetical large cohort has 450 positive and 450 negative outcomes. The classifier has \(TP=405\), \(FN=45\), \(TN=405\), and \(FP=45\), giving accuracy \((405+405)/900=0.9000\), sensitivity \(405/450=0.9000\), and specificity \(405/450=0.9000\).
A second, smaller hypothetical cohort has 50 positive and 50 negative outcomes, with \(TP=10\), \(FN=40\), \(TN=10\), and \(FP=40\). Its accuracy is \((10+10)/100=0.2000\), sensitivity \(10/50=0.2000\), and specificity \(10/50=0.2000\).
Across both cohorts, \(TP=415\), \(FN=85\), \(TN=415\), and \(FP=85\). Pooled accuracy is \((415+415)/1000=0.8300\), pooled sensitivity is \(415/500=0.8300\), and pooled specificity is \(415/500=0.8300\). The pooled accuracy is numerically high, but it does not describe the smaller cohort's performance: that cohort has only 20% accuracy, sensitivity, and specificity. The unweighted macro accuracy is \((0.9000+0.2000)/2=0.5500\). The large gap between pooled and macro accuracy is a warning that one summary alone would conceal substantial cohort variation.
Diagnostics, Comparisons, and Reporting
A practical summary begins with a cohort-by-cohort table of the four confusion counts, actual positive and negative counts, and the chosen performance metrics. Add pooled counts and pooled metrics, and state whether any reported cross-cohort average is pooled or macro. Include the cohort sizes and outcome prevalences: without them, a difference in accuracy or precision can be difficult to interpret. When the intended use gives cohorts equal importance, an unweighted macro summary may be relevant; when the target is the combined subject population, pooled performance may be relevant. The choice should follow the target question, not whichever number looks more favorable.
Check denominators before calculating or comparing metrics. A cohort with no actual positives has undefined sensitivity, and one with no actual negatives has undefined specificity. If there are no positive predictions, precision is undefined. An empty cohort has no accuracy. Do not replace these values with zero: zero would assert observed failures where the metric cannot be calculated. State the undefined metric and the denominator that is zero.
Comparing pooled and macro results is a quick heterogeneity check, not a complete diagnosis. Inspect the cohort-level values and denominators directly; a macro average can also hide variation in the opposite direction, and a mean alone does not show the range. Accuracy may look favorable in a cohort dominated by negative outcomes even if sensitivity is low, while precision can vary substantially with prevalence. These summaries also do not by themselves establish statistical significance or explain why cohorts differ. A receiver operating characteristic area under the curve, when appropriate, summarizes ranking across thresholds rather than performance at one fixed threshold; it should not be substituted for a threshold-specific metric or treated as a simple average of confusion counts.
Confirm that the outcome, prediction rule, and threshold have the same meaning across cohorts, and identify the population or cohorts the summary is intended to represent.
For each cohort, record TP, FN, TN, and FP, then calculate metrics only when their denominators are positive.
Add each confusion-count category across cohorts and calculate pooled metrics from the summed counts; do not average cohort metrics as a substitute.
Show cohort sizes, outcome prevalence, cohort metrics, and pooled results. If useful, report a clearly labelled macro average and inspect the spread for heterogeneity.
Check Your Understanding
Use the definitions and denominator rules above. Each solution is included so the calculation and interpretation can be checked.
- A cohort has \(TP=24\), \(FN=6\), \(TN=56\), and \(FP=14\). Calculate sensitivity, specificity, accuracy, and precision.
- Two cohorts have sensitivities \(0.90\) and \(0.60\), with 80 and 20 actual positives, respectively. What is pooled sensitivity, and what is the unweighted macro sensitivity?
- A cohort has no actual positive outcomes. Is its sensitivity zero, one, or undefined? How does it contribute to pooled sensitivity?
- Why can accuracy and precision differ across cohorts even when sensitivity and specificity are the same?
Solutions. (1) Sensitivity is \(24/(24+6)=24/30=0.8000\); specificity is \(56/(56+14)=56/70=0.8000\); accuracy is \((24+56)/100=0.8000\); precision is \(24/(24+14)=24/38=0.6316\). (2) The pooled sensitivity is \([(80)(0.90)+(20)(0.60)]/(80+20)=(72+12)/100=0.8400\); the macro sensitivity is \((0.90+0.60)/2=0.7500\). (3) Sensitivity is undefined because its denominator, the number of actual positives, is zero. That cohort contributes zero to both the pooled true-positive total and the pooled actual-positive total and is omitted from the weighted-average identity; it is not assigned a sensitivity of zero. (4) Accuracy and precision depend on prevalence as well as sensitivity and specificity. With different outcome prevalence, the proportions of actual positives, actual negatives, and positive predictions change, so accuracy and precision can change even when sensitivity and specificity remain fixed.