Why Reference Choice Is a Statistical Decision
A reference atlas is not just a large collection of cells to which new cells can be compared. It is a particular sample of tissues, conditions, donors, cell states, and assigned labels. Choosing it determines which biological variation is represented and which labels are available to a downstream label-transfer procedure. A very large atlas can therefore be a poor reference if it comes from the wrong tissue or lacks a cell type important to the target data.
The selection question is not simply “Which atlas has the most cells?” It is “Which candidate has adequate biological relevance, label coverage, and annotation evidence for this target, with limitations we can describe?” The earlier material on cell-type annotation as a statistical decision motivates stating the intended labeling decision first. Here, the focus is the suitability of the reference, not the algorithm that transfers labels.
Specify the Target Before Comparing Atlases
The same atlas may be suitable for one task and unsuitable for another. State the target tissue, species, condition, developmental or disease context, and desired label resolution before inspecting candidates. Also list cell types that must be distinguishable. For example, a reference adequate for labeling broad immune compartments may not contain sufficient coverage to distinguish closely related T-cell states.
Separate eligibility criteria from preferences. Eligibility criteria are requirements that a candidate must satisfy, such as the correct species or inclusion of a tissue needed for the analysis. Preferences are desirable but negotiable, such as a larger number of donors or broader condition coverage. This distinction prevents a high cell count from compensating for a mismatch that makes the atlas biologically inappropriate.
Compare candidates using the same definitions. “Tissue match” should mean the same thing for every candidate; a broad organ label may conceal differences in anatomical region or sample preparation. “Condition match” should reflect the actual target, not just whether a disease name appears in the metadata. Record missing or unclear metadata as uncertainty, rather than silently treating it as a match.
Specify tissue, condition, species, and the cell-type resolution required.
Set aside candidates that fail essential biological or technical requirements, and document why.
Check whether every required label is represented and whether its evidence comes from adequate numbers of cells and donors.
Examine how labels were assigned, how ambiguous cases were handled, and whether there is independent validation.
Compare remaining candidates against the stated priorities and report the limitations that matter for interpretation.
Coverage and Reference Composition
Coverage has at least two meanings. Label coverage asks whether the labels needed for the task occur in the reference at all. Support within a label asks how many cells and independent donors contribute to that category. A label represented by a few cells from one donor is not as broadly supported as the same label observed across several donors, even if both appear in the atlas metadata.
Cell counts describe the atlas, but cells from the same donor are not independent biological replicates. A high cell total can result from deeply sampling a small number of donors. Thus, report both cell counts and donor counts by important label when possible. If donor-level counts are unavailable, that is a limitation, not evidence that the cells represent many independent individuals.
Composition also matters. A reference dominated by one cell type may provide much more representation for that type than for rare types. This imbalance can affect a transfer method, but it does not by itself prove that predictions will be biased by a particular amount. Such an effect depends on the method and data. Do not treat a class proportion as a guaranteed upper bound on prediction accuracy or as a direct estimate of the target population’s prevalence.
The justification is direct. Let \(f\) be any procedure whose output is restricted to \(L_R\). For every target cell \(x\), \(f(x)\in L_R\) by definition. If \(c\notin L_R\), then \(f(x)\ne c\) for every \(x\). This is a limitation of the allowed label set, not a claim about expression similarity or the eventual accuracy of a particular transfer algorithm.
Worked Example: Checking Required Cell-Type Coverage
A hypothetical study needs labels for three target populations: macrophages, conventional dendritic cells, and plasmacytoid dendritic cells. Atlas A contains 1,200 macrophages, 500 conventional dendritic cells, and no plasmacytoid dendritic cells. Atlas B contains 600 macrophages, 300 conventional dendritic cells, and 80 plasmacytoid dendritic cells. Assume the supplied annotations use these exact three label names.
For Atlas A, the required label set is not fully covered because plasmacytoid dendritic cells are absent. Its total number of cells in the three categories is \(1{,}200+500+0=1{,}700\), but a large total does not fix the missing category.
For Atlas B, all three labels are present, with \(600+300+80=980\) cells across them. Its plasmacytoid dendritic cell fraction among these three listed categories is \(80/980\approx0.08163\), or 8.163%. That is evidence of limited cell-count support relative to the other two labels, not a guarantee of an 8.163% prediction ceiling or an estimate of prevalence in the target.
If all three labels are essential, Atlas A fails the coverage requirement. Atlas B is eligible on label presence, but its 80 cells should prompt a check of how many donors contributed them and how confidently those cells were annotated. The example establishes a coverage comparison, not a final selection independent of tissue, condition, and annotation quality.
Annotation Quality and Evidence
An atlas’s labels are measurements with provenance, not unquestionable truth. Useful evidence includes the annotation protocol, the expertise and criteria used, the resolution of the labels, independent review, and validation against evidence that was not used to assign the labels. Labels supported only by broad markers may be less informative for a task that requires distinctions between closely related types. The earlier material on marker panels and negative markers provides relevant context for evaluating whether a fine-grained distinction is biologically supported.
Look for uncertainty recorded in the atlas itself: “unknown,” “unassigned,” or broad parent labels may indicate that the annotators appropriately avoided unsupported precision. Do not automatically count such entries as errors, but do not treat them as fully resolved instances of a required subtype either. A reference that labels every cell at fine resolution is not necessarily better if the evidence does not justify those labels.
Quality comparisons should use comparable evidence. For instance, an audited accuracy proportion from a clearly defined sample is more interpretable than an unqualified statement that an atlas is “high quality.” If accuracy is estimated from an audit, check how cells were sampled and whether the audit spans relevant labels and donors. A pooled audit can conceal poor performance for a rare or difficult class. Moreover, a quality estimate for one tissue or condition need not transfer to another.
Worked Example: Comparing Annotation Audits
Two hypothetical atlases report independent audits against expert-adjudicated labels. Atlas A has 92 correct assignments among 100 audited cells. Atlas B has 76 correct assignments among 80 audited cells. Suppose, for this comparison only, the audit samples were selected using the same protocol and cover the target labels and donors.
The observed audit proportions are $$ \hat{p}_A=\frac{92}{100}=0.9200,\qquad \hat{p}_B=\frac{76}{80}=0.9500. $$
Atlas B’s audited proportion is \(0.9500-0.9200=0.0300\), or 3.000 percentage points, higher. Atlas A has 8 incorrect assignments out of 100; Atlas B has 4 out of 80. These are descriptive audit results, not proof that Atlas B is better in every label or for every target population. The sample sizes differ, and the result depends on the audit design and its coverage.
Before using this comparison, verify the stated assumptions: both audits use the same meaning of “correct,” the adjudication is independent of the original annotation, and the audited cells are relevant to the target. If Atlas B’s audit excluded rare dendritic-cell labels, its overall 95.00% figure would not establish quality for those labels. The appropriate conclusion is that Atlas B has the higher reported overall audit proportion under the stated comparable audit conditions, while class-specific evidence remains necessary for a fine-grained task.
Making a Transparent Comparison
A compact comparison table can make selection more reproducible. Include at least tissue and condition match, required-label coverage, cell and donor support for critical labels, and evidence about annotation quality. Use explicit entries such as “meets,” “does not meet,” or “uncertain,” with brief reasons. Avoid collapsing these different dimensions into one score before deciding which are non-negotiable.
A numerical score can help rank eligible candidates if its construction is transparent. For example, a team might assign prespecified scores from 0 to 2 for tissue match, condition match, coverage, and annotation evidence, then weight them according to the task. Such a score is a decision aid, not a statistical estimate of transfer accuracy. The weights express priorities, and a high total must not override a failed essential criterion.
Worked Example: Applying Eligibility Before Ranking
A hypothetical target is lung tissue from a specified inflammatory condition, and two labels—alveolar macrophage and plasmacytoid dendritic cell—are required. The team sets exact lung tissue and presence of both labels as eligibility requirements. Among eligible atlases, it assigns scores from 0 to 2 for condition match and annotation evidence, with equal weights for these two preferences.
Atlas P is lung tissue and contains both labels, but it is from healthy donors. Its condition-match score is 0 and its annotation-evidence score is 2, giving \(0+2=2\). Atlas Q is lung tissue from the specified condition, contains both labels, and has an annotation-evidence score of 1, giving \(2+1=3\). Atlas R is from the specified condition and scores 2 on annotation evidence, but it lacks plasmacytoid dendritic cells.
Atlas R is ineligible and is not rescued by its score: condition match plus annotation evidence would give \(2+2=4\), but it fails a required-label criterion. P and Q pass the stated eligibility rules; Q ranks higher under the declared preference score, 3 versus 2. This ranking does not establish that Q will transfer labels more accurately. It makes clear which criteria determined the choice and where the team accepted a trade-off.
Composition, Diagnostics, and Failure Modes
Several checks help reveal when an apparently attractive reference is not well suited to the target:
- Check metadata completeness. Unclear tissue region, condition, or donor information makes a claimed match less secure. Record uncertainty rather than assuming equivalence.
- Inspect support by label and donor. A seemingly well-represented cell type may come almost entirely from one donor. A large cell total alone does not demonstrate broad donor representation.
- Review label granularity. If a required subtype is represented only by a broad parent label, do not count the parent as a resolved subtype without a defensible basis.
- Check how audits were assembled. Overall agreement can hide errors concentrated in rare or difficult cell types, and an audit from a different biological context may not be relevant.
- Compare target and reference contexts. A tissue name match does not guarantee matching region, disease state, treatment, or developmental state.
A further composition pitfall is to confuse the pooled cell proportion with the average donor-level proportion. Suppose donor 1 contributes 900 cells, 10% of which have a particular label, while donor 2 contributes 100 cells, 80% of which have that label. The pooled proportion is \((900(0.10)+100(0.80))/1{,}000=170/1{,}000=0.1700\), or 17.00%. The unweighted average of the two donor proportions is \((0.10+0.80)/2=0.4500\), or 45.00%. They answer different questions: the first describes a randomly selected pooled cell, while the second gives equal weight to each of these two donors. Neither should be presented as the other.
Alternatives and Limits
When no single atlas meets the task’s needs, the main alternatives are to select a broader but still biologically relevant atlas, use multiple candidate references, or narrow the intended label resolution to what the available evidence supports. Each has a cost. A broader atlas may dilute context-specific representation; multiple atlases require careful handling of differences in their label definitions; and broader labels sacrifice specificity. Detailed methods for reconciling labels across atlases belong to the separate task of label harmonization.
Reference selection cannot guarantee correct transfer. Suitability is a prerequisite and a source of evidence, not a performance result. The selected reference can still differ from the target in unrecorded ways, and a well-annotated atlas can still have weak support for a particular rare type. Selection also cannot justify inferring a category that is absent from its available label set. Report the chosen atlas, the criteria used, critical counts and donor support, and the most consequential limitations.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- A required cell type is absent from an atlas’s reference label set. What can be concluded about exact-label output from a procedure restricted to that set?
- Why should cell counts and donor counts both be reported for an important reference label?
- A candidate has a high annotation-audit proportion, but the audit omitted a rare required label. What does the audit establish, and what does it not establish?
- In a hypothetical atlas, 40 of 500 listed cells have a label. Calculate its fraction among those cells and state one conclusion that cannot be drawn from that fraction alone.
- Why should an atlas that fails an essential tissue or label-coverage requirement not be selected just because it has a high preference score?
Solutions. (1) It cannot return that exact label if outputs are restricted to the reference label set; it may return an available related label, which is not the absent exact label. (2) Many cells can come from few donors, so cell count does not show independent donor representation. (3) It describes performance only for the audited cells under the audit protocol; it does not establish quality for the omitted rare label. (4) \(40/500=0.08000\), or 8.000%; this alone does not establish the target population’s prevalence or a prediction-accuracy ceiling. (5) A preference score cannot compensate for failing a stated requirement that defines eligibility.