Why Compare Annotation Methods?
Marker-based rules, nearest-neighbor transfer, and a trained classifier can assign different labels to the same cell. Comparing their assignments helps locate stable labels and systematic disagreement—for example, a label that one method repeatedly assigns where the others assign another label. These summaries can prioritize labels for review; they do not determine which method is correct.
A meaningful comparison requires the methods to annotate the same cells using a common label vocabulary and compatible label granularity. If one method reports a broad parent type and another reports a specific subtype, their labels must be mapped to a shared level before agreement is calculated. The comparison should also state how unassigned or abstained cells are handled. Report the number of cells each method labels, and calculate agreement on a clearly defined common set rather than silently treating missing labels as ordinary cell types.
Pairwise Agreement and Cohen’s Kappa
For two methods, a cross-tabulation shows both the number of agreements and the direction of disagreement. Let \(n_{ab}\) be the number of cells assigned label \(a\) by method 1 and label \(b\) by method 2. The observed agreement is the sum of diagonal counts divided by the total number of cells. Cohen’s kappa compares this agreement with an expected agreement calculated from the methods’ marginal label proportions.
Worked Example: Pairwise Agreement for Two Methods
Suppose two hypothetical methods annotate 12 cells as A or B. Their cross-tabulation is:
| Method 2: A | Method 2: B | Total | |
|---|---|---|---|
| Method 1: A | 4 | 2 | 6 |
| Method 1: B | 1 | 5 | 6 |
| Total | 5 | 7 | 12 |
The methods agree on the diagonal: 4 cells labeled A by both and 5 labeled B by both. Thus
The marginal proportions for method 1 are \(6/12\) for each label. For method 2 they are \(5/12\) for A and \(7/12\) for B. Therefore,
It follows that
The observed agreement is 75%, and kappa is 0.50 relative to the marginal-product benchmark. Neither number says which method is more accurate; for that, an appropriate reference or validation design is needed.
Summarizing Three or More Methods
With three methods, pairwise rates retain useful detail: marker versus nearest neighbor, marker versus classifier, and nearest neighbor versus classifier. A single summary can complement them, but should not replace them because the same overall value can conceal different disagreement structures.
For \(N\) cells rated by \(m\) methods, let \(n_{ij}\) be the number of methods assigning cell \(i\) label \(j\), and let \(p_j\) be the proportion of all \(Nm\) assignments that use label \(j\). Fleiss’ kappa is a multi-rater chance-adjusted summary. For each cell, the fraction of method pairs that agree is \(P_i=\sum_j n_{ij}(n_{ij}-1)/[m(m-1)]\). The average observed pair agreement is \(\bar P=N^{-1}\sum_i P_i\), and the marginal-product expected agreement is \(P_e=\sum_j p_j^2\). The summary is \(\kappa_F=(\bar P-P_e)/(1-P_e)\), when \(P_e<1\).
This formula follows from counting ordered pairs of distinct methods. A cell with \(n_{ij}\) assignments to label \(j\) contributes \(n_{ij}(n_{ij}-1)\) ordered pairs agreeing on that label. There are \(m(m-1)\) ordered pairs in total, so summing across labels gives \(P_i\). Averaging over cells gives \(\bar P\). Under the marginal-product benchmark, two independently selected assignments have the same label with probability \(\sum_j p_jp_j=\sum_j p_j^2\). Subtracting this benchmark and dividing by the largest possible excess, \(1-P_e\), yields the chance-adjusted statistic.
Worked Example: Comparing Three Methods Across 17 Cells
Consider hypothetical assignments from a marker-based method, a nearest-neighbor method, and a classifier. Ten cells receive A/A/A, three receive A/B/A, and four receive B/A/A, where entries are listed in that method order.
The unanimous fraction is \(10/17=0.5882\). Pairwise agreement rates differ by method pair. The first and second methods agree only on the 10 unanimous cells, so their rate is \(10/17=0.5882\). The first and third agree on the unanimous cells and the three A/B/A cells, giving \(13/17=0.7647\). The second and third agree on the unanimous cells and the four B/A/A cells, giving \(14/17=0.8235\). These rates show that disagreement is not evenly distributed across pairs.
For Fleiss’ kappa, there are 51 assignments. Label A occurs 44 times: 30 in unanimous cells, 6 in the A/B/A cells, and 8 in the B/A/A cells. Label B occurs 7 times. Hence
A unanimous cell has pair agreement 1. In each split cell, two methods assign the same label and one assigns the other, so one of the three method pairs agrees and \(P_i=1/3\). Thus
Therefore,
The raw average pair agreement is \(37/51\approx0.7255\), while the chance-adjusted summary is negative because the pooled assignments are strongly concentrated on A, making the marginal-product benchmark high. The negative value is a warning to inspect the margins and disagreement pattern, not a verdict that the annotations have no biological value.
From Summary Statistics to Review Priorities
A useful review summary keeps the cell-level structure visible. For each label, count how often all methods agree on it, how often one method assigns it while the others agree on another label, and how often assignments are split without a majority. Also report each method’s unassigned or abstained fraction and the number of cells included in each comparison. A high overall agreement can coexist with a problematic minority label, especially when common labels dominate the cell set.
| Pattern | What it can indicate | Useful follow-up summary |
|---|---|---|
| One method repeatedly differs while two others agree | A method-specific labeling pattern or a label definition mismatch | Count this pattern by label and method |
| Disagreement concentrated between related labels | Boundary ambiguity or differing label granularity | Tabulate confusions at the shared hierarchy level |
| Several methods disagree with no repeated majority | Cells may be difficult to classify or evidence may be weak | Report the number and fraction of such cells |
| Many unassigned or abstained cells | Methods may have different coverage or confidence thresholds | Report coverage separately from agreement on labeled cells |
Worked Example: A Prevalence-Sensitive Kappa
In a second hypothetical two-method comparison, 24 cells have the following counts: both methods assign A to 17 cells, method 1 assigns A while method 2 assigns B to 2, method 1 assigns B while method 2 assigns A to 5, and neither assigns B to the same cell. The margins are 19 A and 5 B for method 1, and 22 A and 2 B for method 2.
Observed agreement is \(17/24=0.7083\). Expected agreement from the margins is
Calculating from the exact fractions avoids rounding inconsistency:
Although the methods agree on more than seven in ten cells, their agreement is below the marginal-product benchmark in this example. The imbalanced margins—especially method 2’s assignment of A to 22 of 24 cells—help explain why. Report the margins alongside kappa rather than interpreting kappa in isolation.
Assumptions, Diagnostics, and Limitations
Before comparing methods, verify that each row represents the same cell across methods; labels refer to the same biological level; and the compared set is defined consistently. If an abstention is present, do not silently count it as a biological label. One defensible descriptive analysis is to report each method’s coverage and calculate agreement among cells labeled by all methods, explicitly naming that subset. Because the subset may be easier to annotate than cells that trigger abstention, this agreement does not describe all cells.
Inspect the cross-tabulations and label-specific patterns before summarizing. Unexpectedly high disagreement for one label, a one-sided pattern in which one method repeatedly assigns a label that others do not, or a large change in agreement after harmonizing label granularity can reveal a comparison problem. Check cell identities, label mappings, and the number of cells per label. If cells come from the same donor or batch, cell-level assignments may be dependent; the displayed fractions remain descriptive, but treating every cell as an independent replicate for inferential claims can exaggerate precision.
Agreement measures have important limits. Cohen’s and Fleiss’ kappa depend on the marginal frequencies of labels, so their values can change with prevalence even when a raw agreement rate is similar. They also compress distinct disagreement patterns into one number. Neither kappa nor unanimous agreement establishes accuracy: methods may share a bias, share training or reference information, or agree on a label that is biologically inappropriate. Use an appropriate validation source when the question is performance, and treat disagreement summaries as a way to identify where scrutiny may be most valuable.
Check Your Understanding
Answer the questions, then compare your reasoning with the solutions below.
- Two methods agree on 36 of 50 jointly labeled cells. What is their observed agreement, and what does it mean?
- For three methods, a cell has labels A, A, and B. What fraction of the three method pairs agrees on that cell?
- Why can kappa be low even when observed agreement seems substantial?
- What should be reported if one method abstains on many cells?
Solutions. (1) \(p_o=36/50=0.72\), so the methods assign the same label to 72% of the jointly labeled cells; this does not establish that those labels are correct. (2) One of the three pairs, the two methods assigning A, agrees, so the cell’s pair agreement is \(1/3\). (3) Kappa compares observed agreement with an expected value based on marginal label proportions; imbalanced margins can make that benchmark high. (4) Report the abstention rate or coverage for each method, define the set used for agreement, and state that agreement on jointly labeled cells may not represent abstained cells.