Why Adjudication Requires More Than a Vote
Cell-type annotators can disagree for different reasons. One may rely on a marker panel while another relies on a reference-based prediction; they may use different cell-type vocabularies; or they may agree on a broad lineage but differ about a fine subtype. Sometimes the cells themselves are transitional or lack enough evidence for a precise label. These situations are not interchangeable, so a useful review first identifies what kind of disagreement is present.
Adjudication is a structured assessment of the available evidence and its limits, not a contest in which the most confident annotator automatically wins. The goal is a label that is supported at the finest defensible level. If the evidence does not resolve the disagreement, the review should preserve that uncertainty rather than disguise it with a forced choice.
The review assumes that the labels refer to the same cells or cell group, that the candidate labels can be mapped to a shared ontology or vocabulary, and that the evidence being compared is available and interpretable. If the annotators examined different cell sets, used incompatible label definitions, or had access to different measurements, those differences must be clarified before the annotations can be compared. Adjudication cannot repair missing or incomparable evidence by itself.
A Structured Review
Begin by writing down the competing labels exactly as given, the level of each label, and the evidence each annotator used. Then classify the disagreement. Is it a conflict between different lineages, a distinction between sibling subtypes, a difference in ontology depth, or a disagreement caused by a particular method or reference? The distinction matters: a broad parent label may resolve a granularity difference, while a lineage conflict requires weighing the underlying evidence.
Confirm that both annotations concern the same cells and map each label to a common ontology. Note whether the labels are siblings, parent and child, or biologically distinct alternatives.
Check whether the observed marker pattern supports each proposed identity and whether contradictory markers, missing genes, or state-dependent expression limit interpretation. A marker is evidence, not a standalone proof of identity.
Record the predicted labels and any available uncertainty summaries. Ask whether the reference represents the tissue, condition, cell state, and candidate types under review. A prediction can be internally decisive yet inappropriate for a mismatched reference.
Consider model uncertainty, cell ambiguity, and disagreement among annotators separately. Select a supported label at the finest defensible ontology level, or retain an unresolved status when the evidence does not discriminate.
State which evidence supported the decision, which evidence conflicted, and why the chosen level is appropriate. Keep unresolved alternatives visible rather than silently discarding them.
Marker review should consider a pattern of positive and negative evidence, not just whether one familiar gene appears in a list. Check that the measured features are detected well enough for absence to be informative, and consider whether activation, stress, or a transitional state could alter expression. A marker pattern may support a broad lineage while failing to distinguish its subtypes. Reference predictions should be assessed in the same way: their output is informative only to the extent that the reference and prediction procedure are relevant to the cells under review.
Uncertainty summaries help describe how concentrated a model’s predictions are, but they do not independently settle an annotation. For example, if a method provides probabilities \(p_1,\ldots,p_K\) over \(K\) mutually exclusive labels, the largest probability describes the leading prediction, and the margin is the difference between the largest and second-largest probabilities. A small margin indicates that the leading alternatives are close under that model. Entropy, \(H=-\sum_{k=1}^{K}p_k\log(p_k)\), summarizes the spread across the full prediction distribution: higher entropy indicates a more diffuse distribution. These summaries depend on the probabilities being meaningful for the task; a score that is not a calibrated probability should not be interpreted as one.
A model’s uncertainty is also not identical to biological ambiguity. A model may be uncertain because it has limited training examples or lacks a relevant reference type. Conversely, distinct methods may agree on a label even when the cells show a mixed or transitional marker pattern. The earlier material on separating model uncertainty from cell ambiguity is useful when interpreting these cases. Agreement among methods is supporting evidence, not a guarantee of correctness, especially when methods share data or assumptions.
Ontology Level and a Useful Justification
An ontology represents labels at different levels of specificity. For instance, a broad lineage can have several subtype labels beneath it. If annotators disagree only about which child label applies, and evidence supports the parent but does not distinguish the children, the parent may be the most defensible resolution. This is not the same as asserting that the parent is a uniquely identified fine subtype; it explicitly makes a less specific claim.
The same principle explains why a broad label can be justified when fine-level evidence is unresolved. It does not justify selecting a parent merely because its children’s probabilities look large if those values are uncalibrated, the ontology mapping is wrong, or plausible alternatives have been omitted. Nor does it establish that one specific child is correct.
Worked Example: Close Subtype Predictions Within a Supported Lineage
In a hypothetical cell group, annotator 1 proposes conventional CD4 T cells and annotator 2 proposes regulatory T cells. A reference method assigns probabilities 0.46, 0.41, 0.08, and 0.05 to conventional CD4 T cells, regulatory T cells, CD8 T cells, and other, respectively. Marker review finds a consistent T-lineage pattern, but no convincing subtype-specific pattern that separates the two leading labels.
The margin between the two leading predictions is \(0.46-0.41=0.05\). The lead is small, so the reference prediction does not clearly distinguish the two subtypes. The entropy is approximately \( -[0.46\log_2(0.46)+0.41\log_2(0.41)+0.08\log_2(0.08)+0.05\log_2(0.05)] = 1.550\) bits. For four possible labels the maximum entropy is \(\log_2(4)=2\) bits, consistent with substantial spread across the predictions.
The two leading alternatives are both within the T-cell lineage. Their probabilities sum to \(0.46+0.41=0.87\), but that sum alone does not establish a subtype. Given the stated ontology and marker evidence, the defensible resolution is the broader T-cell label, with the subtype disagreement noted. Calling the cells regulatory T cells solely because that label was proposed by one annotator would claim more specificity than the evidence supports.
When Reference and Marker Evidence Conflict
A reference prediction and a marker-based assessment are not automatically independent witnesses. A reference method may use many of the same genes that inform a marker review, so counting the two as separate votes can overstate support. A conflict should prompt questions about the basis of each claim: which features drove the prediction, whether the reference includes the relevant tissue and state, and whether the observed pattern is coherent with the proposed label. Where the reference appears poorly matched, its output should be treated cautiously rather than accepted simply because it is quantitative.
Worked Example: A Reference Prediction That Conflicts With the Marker Pattern
In a hypothetical tissue sample, two annotators disagree between macrophage and dendritic cell labels. A reference trained mainly on blood cells predicts dendritic cells with probability 0.72. The observed cells show a coherent macrophage-associated marker pattern in the available measurements; the evidence does not show a clear dendritic-cell pattern. The tissue contains resident myeloid cells that may not be well represented in the reference.
The value 0.72 is the method’s prediction for dendritic cells, not a 72% vote among independent reviewers and not proof that the cells are dendritic. The reference’s tissue mismatch is a plausible limitation, so the prediction warrants less weight for this question. The marker evidence favors macrophages, but the review must still check that the pattern is interpretable and that the competing label has not been excluded merely because its markers were poorly measured.
For this hypothetical case, assume that the marker pattern is reproducible across the group and that the relevant measurements are adequate. The review therefore resolves the group as macrophage at the level supported by those data, while documenting that the blood-based reference favored dendritic cells. If the marker pattern were weak or the cells fell between the two identities, the appropriate result would instead be an unresolved myeloid label or another supported parent category—not an automatic choice of either fine label.
Granularity Disagreements and Probability Aggregation
Sometimes reviewers are not making contradictory biological claims: one uses a broad label and another a subtype. Check the ontology before calling this a conflict. If the subtype is a child of the broad label, the question is whether the data support that extra specificity. A fine label should not be preserved just because it is more informative when the evidence does not separate it from its sibling labels.
Worked Example: Harmonizing Labels at the Supported Ontology Level
In another hypothetical group, one reviewer calls the cells endothelial and another calls them capillary endothelial. A classifier provides probabilities over four mutually exclusive leaves under the relevant ontology: capillary endothelial 0.34, arterial endothelial 0.31, venous endothelial 0.20, and pericyte 0.15. Marker evidence supports endothelial identity but does not distinguish the three endothelial subtypes.
The total probability assigned to the endothelial parent is \(0.34+0.31+0.20=0.85\). The capillary probability is only 0.34, and it is not clearly separated from the arterial probability of 0.31. Under the stated assumptions that the leaves form the relevant partition and the probabilities are usable, the broad endothelial label has substantial support, whereas the capillary subtype is not resolved. The adjudicated label is therefore endothelial, not capillary endothelial. The evidence supports the parent-level claim without settling the child-level claim.
If the candidate leaves are not exhaustive, parent probability cannot be obtained by summing only the listed children and treating the result as complete. If labels come from incompatible ontologies, first establish a valid mapping or state that a shared label cannot yet be assigned. A superficially convenient parent name is not a substitute for a biologically valid relationship among categories.
Alternatives, Diagnostics, and Failure Modes
A simple majority vote is quick but can conceal shared biases, disregard evidence quality, and reward the number of annotators rather than the relevance of their evidence. Averaging method scores can be equally misleading when scores have different meanings or are dependent. Forced selection of the most probable fine label hides uncertainty, particularly when the top alternatives are close. At the other extreme, always retreating to a broad parent discards useful resolution when marker and reference evidence genuinely support a subtype. The structured review is an alternative to these shortcuts: it weighs evidence in context and lets the ontology level reflect what is actually resolved.
Warning signs that an adjudication is going wrong include a decision driven by a single marker, a probability treated as certainty, or a reference treated as definitive despite an obvious tissue or state mismatch. Another warning sign is an unexpectedly specific label that cannot be traced to discriminating evidence. Check whether different annotators used different label definitions or examined different cell sets before interpreting their disagreement as biological.
Adjudication also has limits. It cannot create information that was not measured, establish a biological identity from an inadequate reference, or guarantee that a chosen label is correct. A broad label may be the most honest decision but can obscure real heterogeneity within the group. If a group contains cells with distinct evidence patterns, a single group label may not adequately summarize them; the review should not force a uniform conclusion merely for convenience. Retain the unresolved alternatives when the data do not discriminate, and make clear what additional evidence would help distinguish them.
Common Mistake
The practical standard is not that every reviewer must agree on the same fine label. It is that the decision process makes the evidence and its limitations explicit. A defensible adjudication may conclude that one label is better supported, that a parent label is appropriate, or that the disagreement remains unresolved.
Check Your Understanding
For each question, consider what evidence is needed before an annotation decision is justified. Solutions follow the questions.
- Two annotators propose sibling subtypes within the same lineage. The reference probabilities are close, and markers support the lineage but not either subtype. What level of label is defensible?
- A reference assigns probability 0.80 to a label, but it was trained on a different tissue and the marker pattern conflicts. Does 0.80 settle the disagreement? Why or why not?
- Three subtype probabilities under one parent are 0.25, 0.20, and 0.30. A fourth, unrelated label has probability 0.25. Under what conditions can the parent probability be reported as 0.75?
- Why can majority vote fail when several annotators use the same reference method?
Solutions: (1) Use the supported parent lineage and record that the subtype is unresolved; the evidence does not justify choosing a child. (2) No. The probability is a method output whose relevance is limited by reference mismatch and conflicting marker evidence; assess both sources in context. (3) If the three subtype labels are mutually exclusive children of that parent, the listed probabilities use the same valid distribution, and no other child or relevant possibility is omitted, their sum is \(0.25+0.20+0.30=0.75\). (4) Their votes may not be independent: shared reference data or assumptions can reproduce the same error, so counting votes does not provide independent confirmation.