Why Method Selection Is a Study-Specific Decision
An integration method can mix cells across batches well and still be a poor choice. It may blur a biological contrast, lose a rare state, or rely on cell correspondences that the study does not support. Conversely, a method with less dramatic batch mixing may be preferable if it preserves the biological structure needed for the analysis. Method selection therefore requires more than ranking one benchmark column.
The relevant evidence includes the study design, which cell populations are shared across datasets, and the intended biological use of the integrated data. Benchmark results help assess those choices, but they do not establish that an approach will work in every new dataset. The earlier material on benchmarking metrics and fair comparisons provides ways to define and measure performance; here the task is to combine that evidence into a defensible decision.
Start With Design and Required Biological Structure
Before reading a benchmark table, write down which distinctions the analysis must preserve. These may include a condition contrast, donor-to-donor variation, or a rare cell state. Also record which samples share cell populations and which experimental factors are crossed or nested. These details determine whether benchmark results are relevant to the intended application.
MNN and anchor-based methods depend on useful correspondences between datasets. If a population is present in one dataset but genuinely absent in another, treating those cells as if they must correspond can create misleading alignment. MNN requires mutual-neighbor pairs to provide informative matches; anchor-based methods require the selected anchors to represent valid correspondences. Either approach can struggle when overlap is weak, when a shared population changes substantially, or when technical differences obscure true counterparts. Check the relevant populations, not only the overall number of matched cells.
A mixed model can represent structure such as cells measured within donors or samples, and can include biological covariates whose effects should be retained. Its conclusions depend on the model being appropriate for the response and design, and on the technical and biological effects being estimable separately. A model cannot recover a condition effect that is perfectly confounded with batch. Random-effect estimates also need enough independent groups to support the intended inference; many cells from one donor do not substitute for multiple donors.
These are eligibility checks, not claims that one method is always superior. A method whose core assumptions fail should not win simply because it scores well on a metric that does not reveal that failure.
Read the Benchmark as a Set of Complementary Evidence
For a useful comparison, make clear which direction is favorable for every metric. For example, a higher batch-mixing score may be favorable, whereas a lower false-correction rate may be favorable. To combine them, first orient and scale the measures so that larger values consistently mean better performance. A normalized score between 0 and 1 is convenient, but it does not make different biological dimensions interchangeable.
Check whether the benchmark resembles the intended study in its cell populations, sample structure, imbalance, and known biological contrasts. Give particular attention to diagnostics linked to the analysis goal: conservation of known biology, rare-state retention, negative controls, and residual patterns. A global score can conceal a severe failure in one important subgroup. Where benchmark variation or uncertainty is reported, compare that uncertainty as well as the point estimates; small differences may not support a confident ranking.
A weighted summary can help organize evidence, provided its role is explicit. Let \(B_m\), \(C_m\), \(R_m\), and \(N_m\) denote oriented, normalized scores for batch mixing, biological conservation, rare-state retention, and negative-control preservation for method \(m\). If the corresponding weights are nonnegative and sum to 1, a transparent summary is:
The weights express priorities; they are not universal constants. A better batch-mixing score should not automatically compensate for destruction of a biological contrast that is essential to the study. Set critical minimum requirements first, such as acceptable rare-state retention, then use a summary score to compare methods that pass. When the choice changes under reasonable weights, report that sensitivity rather than presenting the ranking as decisive.
Worked Example: Choosing Among Methods When Several Metrics Matter
Suppose a hypothetical benchmark resembles the planned study: each of several batches contains the main cell populations, and the biological condition is represented across batches. All values below are normalized so that larger is better. The team assigns weights of 0.25 to batch mixing, 0.35 to biological conservation, 0.20 to rare-state retention, and 0.20 to negative-control preservation.
| Method | Batch mixing | Biological conservation | Rare-state retention | Negative-control preservation |
|---|---|---|---|---|
| MNN | 0.90 | 0.86 | 0.70 | 0.80 |
| Anchor-based | 0.84 | 0.93 | 0.88 | 0.94 |
| Mixed model | 0.75 | 0.90 | 0.91 | 0.96 |
For MNN, the weighted score is \(S_{\mathrm{MNN}}=0.25(0.90)+0.35(0.86)+0.20(0.70)+0.20(0.80)=0.225+0.301+0.140+0.160=0.826\). For anchor-based integration, \(S_{\mathrm{anchor}}=0.25(0.84)+0.35(0.93)+0.20(0.88)+0.20(0.94)=0.210+0.3255+0.176+0.188=0.8995\). For the mixed model, \(S_{\mathrm{mixed}}=0.25(0.75)+0.35(0.90)+0.20(0.91)+0.20(0.96)=0.1875+0.315+0.182+0.192=0.8765\).
The anchor-based method ranks highest under these stated weights. That is evidence in its favor, not proof that it is best for the planned analysis. Its anchors still need to represent valid cross-batch correspondences, and the modest score difference from the mixed model should be interpreted in light of benchmark uncertainty and sensitivity to weights. If rare-state preservation is a hard requirement and the mixed model has a more appropriate replicate structure, that evidence may justify choosing it instead.
A Decision Process That Includes Assumptions
A practical selection combines eligibility, diagnostic review, and comparative evidence. The following sequence makes it less likely that a single appealing score will dominate the decision.
Name the biological distinctions the integrated data must retain and the technical variation that should be reduced.
Assess whether MNN pairs or anchors can represent valid shared populations. For a mixed model, check replication, factor structure, and whether technical and biological effects are identifiable.
Rule out methods that fail a prespecified essential criterion, such as preservation of a known rare state or biological contrast.
Review batch mixing, biological conservation, negative controls, rare-state results, and method-specific diagnostics. Use a weighted summary only to clarify stated priorities.
Check whether reasonable changes to weights, parameter settings, or benchmark subsets change the ranking. If they do, describe the decision as uncertain.
Worked Example: Weak Overlap and a Critical Rare State
Consider a hypothetical study in which most populations occur in both datasets, but a disease-associated rare state is confidently identified in only one. The benchmark reports the following normalized scores. For this analysis, the team requires rare-state retention of at least 0.80; the threshold is a prespecified decision rule, not a universal standard.
| Method | Batch mixing | Biological conservation | Rare-state retention | Negative-control preservation |
|---|---|---|---|---|
| MNN | 0.91 | 0.82 | 0.56 | 0.78 |
| Anchor-based | 0.83 | 0.91 | 0.84 | 0.92 |
| Mixed model | 0.77 | 0.90 | 0.88 | 0.95 |
MNN fails the critical rare-state requirement because \(0.56<0.80\), despite its highest batch-mixing score of 0.91. The anchor-based and mixed-model approaches pass that requirement, but neither score settles the choice. Since the rare state has no counterpart in one dataset, the team should inspect whether anchors were formed among the truly shared populations and whether the rare state was preserved rather than forced into an inappropriate correspondence. The team should also confirm that the mixed model has sufficient independent samples and a design that separates technical from biological effects.
If those checks support both approaches, the mixed model's higher rare-state and negative-control scores favor it when preserving the rare state is central. If its sampling structure is inadequate, its benchmark score does not fix that design limitation; the anchor-based method may be more defensible if its correspondences are credible. The choice follows the assumptions and target, not the largest value in the batch-mixing column.
Worked Example: Recognizing Perfect Confounding
In a hypothetical two-batch study, all cells in batch 1 are from condition A and all cells in batch 2 are from condition B. The mean normalized expression for a gene is 2 units in batch 1 and 5 units in batch 2. A proposed mixed model includes an intercept, a condition indicator \(C\), and a batch indicator \(B\):
With condition A and batch 1 coded 0, and condition B and batch 2 coded 1, every observation has \(C=B\). The model therefore contains two identical predictor columns. Its fitted mean difference between the observed groups is \(\beta_C+\beta_B=5-2=3\) expression units, but the data cannot determine how much of those 3 units belongs to condition and how much to batch. Many pairs of coefficients with the same sum fit the observed group means.
This is a design failure, not a reason to prefer MNN or anchors automatically. The apparent batch difference and biological difference are inseparable in this benchmark. Integration may make the groups look more mixed, but that visual or numerical change cannot identify the biological condition effect. The defensible conclusion is that these data do not support a separate estimate of condition and batch effects; additional design information or samples with conditions represented across batches are needed.
Diagnostics, Alternatives, and Limits
After the initial comparison, inspect failures in the methods that remain plausible. For MNN or anchor-based approaches, check whether matches are concentrated among expected shared populations, whether any population lacks credible counterparts, and whether biological controls or known contrasts are weakened. A high overall mixing score alongside lost rare-state retention is a warning, not a success. For mixed models, inspect the residual diagnostics and design checks established in the earlier material on mixed-model integration: systematic residual patterns, unsupported variance estimates, or poor separation of modeled effects undermine the interpretation.
The approaches are alternatives with different strengths, not interchangeable settings for one universal correction. MNN can be useful when meaningful mutual matches exist across datasets; anchor-based methods make cross-dataset correspondences explicit and can be assessed through anchor quality; mixed models can represent replicate and covariate structure when the design supports estimation. A method may be unsuitable because its assumptions do not fit, even if it is computationally convenient. Benchmark results can also depend on preprocessing and parameter choices, so sensitivity results matter.
No integration method can establish that removed variation was purely technical, recover an unobserved counterpart with certainty, or resolve perfect confounding by computation alone. The benchmark is evidence about performance under tested conditions, not a guarantee for every dataset. If the methods trade off important objectives or the ranking is unstable, report that limitation and retain the diagnostics that support the final choice.
Common Mistakes
- Choosing the largest batch-mixing score. Mixing is not sufficient if a critical cell state or biological contrast is lost. Check conservation and prespecified requirements alongside mixing.
- Assuming every population should align. A population absent from another dataset may be biologically distinct, not a failed match. Examine whether the overlap assumption is credible.
- Treating the weighted score as objective. Weights encode priorities. State them, check sensitivity, and do not let an average conceal a failure of an essential criterion.
- Expecting a model to repair confounding. If condition and batch are perfectly aligned, their separate effects are not identified. Use a design that separates them or limit the conclusion.
- Equating many cells with many replicates. Cells from the same donor do not provide the same independent information as multiple donors when donor-level variation is relevant.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A method has the highest batch-mixing score but falls below a prespecified minimum for rare-state retention. Should it be selected solely on its mixing result?
- What evidence is needed before interpreting MNN pairs or anchors as valid cross-dataset correspondences?
- Why can a mixed model not separate condition and batch effects when every condition occurs in exactly one batch?
- A weighted score changes its method ranking when the biological-conservation weight is increased slightly. What should be reported?
- Why do 2,000 cells from one donor not replace several independent donors for estimating donor-level variation?
Solutions. (1) No. If rare-state retention is essential and the method fails the prespecified requirement, its high mixing score does not make it eligible. (2) Check that the populations are genuinely shared, that the matches are concentrated among plausible counterparts, and that biological controls and rare states are not incorrectly aligned or erased. (3) The condition and batch indicators are identical, so the design cannot identify their separate effects; only their combined difference is estimable. (4) Report that the ranking is sensitive to priorities, state the weights or requirements used, and avoid presenting one method as an unambiguous winner. (5) Cells within a donor share donor-level structure, so they do not provide independent replication of between-donor variation.