Why Classwise Performance Needs More Than One Number
A cell-type annotation can appear successful overall while performing poorly for a particular label. Overall accuracy counts correct predictions across all evaluated cells, but it does not show which true cell types are being missed or which labels are attracting incorrect predictions. A confusion matrix makes those patterns visible by arranging correct and incorrect assignments according to both the reference label and the predicted label.
The matrix is useful only when its orientation is clear. In this tutorial, rows are reference (true) labels and columns are predicted labels. The reference labels are treated as the operational standard for evaluation; this does not establish that they are biologically infallible. The examples use hypothetical held-out validation cells, with one reference label and one predicted label per cell. Each cell is counted once, and the label set is mutually exclusive and exhaustive for the evaluation.
A raw confusion matrix contains counts. Row normalization divides each entry in row \(i\) by \(R_i\); it describes the distribution of predictions among cells whose reference label is \(i\). Column normalization divides each entry in column \(i\) by \(K_i\); it describes the distribution of reference labels among cells predicted as \(i\). Each normalized row or column with a positive total sums to 1.
Classwise Measures From the Matrix
For a particular class \(i\), a correct prediction is a true positive for that class. A cell with reference label \(i\) predicted as something else is a false negative for \(i\). A cell with another reference label predicted as \(i\) is a false positive for \(i\). These are one-versus-rest counts defined separately for each class.
The off-diagonal entries are the errors. For class \(i\), the sum of off-diagonal entries across its row is the number of false negatives, while the sum of off-diagonal entries down its column is the number of false positives. This distinction makes the measures complementary: sensitivity focuses on missed members of the reference class; precision focuses on incorrect additions to the predicted class. A high value for one does not guarantee a high value for the other.
Worked Example: Reading Both Normalizations
Worked Example: Reading Both Normalizations
Suppose a hypothetical held-out validation set contains T cells, B cells, and monocytes. The table shows reference labels in rows and predicted labels in columns.
| Reference \ Predicted | T | B | Monocyte | Row total |
|---|---|---|---|---|
| T | 45 | 3 | 2 | 50 |
| B | 4 | 36 | 0 | 40 |
| Monocyte | 1 | 5 | 24 | 30 |
| Column total | 50 | 44 | 26 | 120 |
For T cells, row normalization gives \(45/50=0.900\) predicted as T, \(3/50=0.0600\) predicted as B, and \(2/50=0.0400\) predicted as monocyte. Thus T-cell sensitivity is \(45/50=0.900\). Column normalization for predicted T gives \(45/50=0.900\) reference T, \(4/50=0.0800\) reference B, and \(1/50=0.0200\) reference monocyte. Thus T-cell precision is also \(45/50=0.900\).
For B cells, sensitivity is \(36/40=0.900\), while precision is \(36/44=0.818\), because 8 cells assigned the B label have a different reference label. The B-cell F1 score is \(2(0.900)(0.81818)/(0.900+0.81818)=0.857\), equivalently \(72/(72+8+4)=0.857\).
For monocytes, sensitivity is \(24/30=0.800\), and precision is \(24/26=0.923\). The monocyte F1 score is \(2(24)/(2(24)+2+6)=48/56=0.857\). Here the two off-diagonal monocyte-column entries are false positives, and the six off-diagonal monocyte-row entries are false negatives.
The matrix also shows the actual error destinations: 5 reference monocytes were predicted as B, whereas no reference B cells were predicted as monocytes. These are different directions of confusion, not one symmetric error rate.
Why the F1 Formula Has This Form
The F1 score is the harmonic mean of sensitivity and precision. For positive values \(a\) and \(b\), their harmonic mean is \(2/(1/a+1/b)\). Substituting sensitivity \(TP/(TP+FN)\) and precision \(TP/(TP+FP)\) gives a count-based expression that weights false positives and false negatives symmetrically.
The derivation assumes \(TP>0\), and the resulting score also requires a positive denominator. The harmonic mean is pulled toward the smaller of sensitivity and precision, so F1 is low when either is low. Unlike the arithmetic mean, it does not allow a very high value for one measure to compensate as much for a low value of the other.
Worked Example: Finding a Directional Label Confusion
Worked Example: Finding a Directional Label Confusion
In another hypothetical evaluation, the counts for three labels are as follows.
| Reference \ Predicted | A | B | C | Row total |
|---|---|---|---|---|
| A | 28 | 2 | 0 | 30 |
| B | 9 | 18 | 3 | 30 |
| C | 1 | 4 | 25 | 30 |
| Column total | 38 | 24 | 28 | 90 |
For class B, sensitivity is \(18/30=0.600\): 18 of the 30 reference-B cells received the B prediction. Precision is \(18/24=0.750\): 18 of the 24 cells predicted as B are reference B. Therefore, \(F_{1,B}=2(18)/(2(18)+6+12)=36/54=0.667\).
The row for reference B shows 9 cells assigned to A and 3 assigned to C. The column for predicted A includes 9 reference-B cells, while the row for reference A contains only 2 cells assigned to B. Thus the A–B confusion is strongly directional in these counts: B is much more often predicted as A than A is predicted as B. The matrix identifies this asymmetry; it does not by itself explain its biological or technical cause.
Summaries Across Classes and Their Limits
Overall accuracy is the total number of correct predictions divided by the total number evaluated. It summarizes performance at the cell level, but it can be dominated by classes with many cells. Classwise metrics retain information about each label. When a single summary across classes is needed, the macro-average gives each class equal weight, whereas a support-weighted average weights each class by its reference row total.
Here \(m\) is the number of evaluated classes. These summaries answer different questions and should not be treated as interchangeable. Accuracy asks what fraction of all evaluated cells were assigned correctly. Macro F1 gives every included class equal influence. Weighted F1 reflects class frequencies in the reference set, so common classes contribute more. Reporting classwise values alongside any aggregate helps prevent the aggregate from hiding a specific failure.
Worked Example: Accuracy Can Conceal Uneven Class Performance
Consider a hypothetical two-class validation set with 90 common-class cells and 10 less common-class cells.
| Reference \ Predicted | Common | Less common | Row total |
|---|---|---|---|
| Common | 85 | 5 | 90 |
| Less common | 4 | 6 | 10 |
| Column total | 89 | 11 | 100 |
There are \(85+6=91\) correct predictions, so accuracy is \(91/100=0.910\). For the less common class, sensitivity is \(6/10=0.600\), precision is \(6/11=0.545\), and F1 is \(12/(12+5+4)=12/21=0.571\). Thus the overall accuracy of 91.0% coexists with substantially weaker performance for that class.
For the common class, sensitivity is \(85/90=0.944\), precision is \(85/89=0.955\), and F1 is \(170/(170+4+5)=0.950\). Macro F1 is \((0.950+0.571)/2=0.761\) (using unrounded values). Weighted F1 is \((90(0.9497)+10(0.5714))/100=0.912\). The different summaries reflect different weighting, not contradictory calculations.
Checks, Failure Modes, and Comparisons
Before interpreting a matrix, check that its row and column labels are aligned and that every evaluated cell contributes exactly once. The row totals should match the number of cells in each reference class, the column totals should match the number assigned to each predicted class, and both sets of totals should sum to the same overall count. These checks can catch transposed tables, dropped labels, and inconsistent filtering.
- Do not swap the normalizations. With reference labels in rows, row normalization gives sensitivity; column normalization gives precision. A software package may use the opposite axis convention, so verify the labels and denominators rather than relying on the display alone.
- Do not report normalized values without counts. A proportion based on 2 cells has a different evidential basis from the same proportion based on 200 cells. Keep the raw counts or class totals available.
- Handle empty rows or columns explicitly. If there are no reference cells for a class, sensitivity is undefined. If no cells were predicted as a class, precision is undefined. F1 is also undefined when its denominator is zero. Do not silently replace undefined values with zero; report the reason and any convention used for an aggregate.
- Confirm the label definitions agree. If reference and predicted labels differ in granularity or use incompatible names, a count may represent a label-mapping problem rather than a classifier error. The evaluated classes must have a clearly stated correspondence.
Raw counts are best for auditing how many cells went to each destination. Row-normalized matrices compare the distribution of predictions conditional on reference class, making class-specific misses easy to see. Column-normalized matrices assess the composition of each predicted group, highlighting labels that collect cells from other classes. Accuracy is a concise overall summary, while classwise sensitivity, precision, and F1 show which kinds of errors contribute. None of these summaries alone explains why a label was confused, and cell-level counts do not automatically provide independent evidence about performance across donors. In donor-held-out evaluation, the donor structure and validation design still matter when generalizing beyond the evaluated cells.
Check Your Understanding
Answer each question using the stated convention that rows are reference labels and columns are predicted labels. Solutions follow the questions.
- A class has 24 reference cells, of which 18 are predicted correctly. What is its sensitivity?
- A predicted class contains 30 cells, and 21 of them have that class as their reference label. What is its precision?
- For a class, \(TP=16\), \(FP=4\), and \(FN=8\). Calculate its F1 score.
- In a confusion matrix, there are 7 reference-A cells predicted as B and 2 reference-B cells predicted as A. Which direction of confusion is more frequent?
- Why can a high overall accuracy fail to show that performance is weak for a particular cell type?
Solutions. (1) Sensitivity is \(18/24=0.750\). (2) Precision is \(21/30=0.700\). (3) \(F_1=2TP/(2TP+FP+FN)=32/(32+4+8)=32/44=0.727\). (4) Reference A predicted as B occurs more often: 7 cells versus 2 in the reverse direction. (5) Accuracy pools correct predictions across all cells, so abundant classes can dominate it; a classwise sensitivity, precision, or F1 score can reveal weaker performance in a less frequent class.