Start With the Scientific Claim
An integration result is not good merely because batches look similar after correction. If batches contain different cell-state compositions, removing every visible batch difference can erase genuine biology. Conversely, preserving every difference can leave technical shifts that interfere with the intended analysis. A benchmark therefore needs an explicit target: what should an integration remove, what should it retain, and for which cells and scientific use?
The target is not yet a score or a particular calculation. It is a statement about the desired behavior of an integrated representation or adjusted expression values. Metrics can later operationalize that statement, but choosing a metric before stating the target risks rewarding an output that is unsuitable for the question.
Let \(B_i\) denote the batch of cell \(i\), \(Z_i\) its biological state, and \(C_i\) a condition such as treatment or disease status. Let \(U_i\) denote its integrated representation; if the method produces adjusted expression, write \(Y^*_{ig}\) for the adjusted value of gene \(g\). A benchmark target should say which of these outputs matters. A representation used to group cells need not preserve gene-level contrasts in the same way as adjusted expression used to estimate a condition effect.
Define Batch Removal Conditionally
A common target is that technical batch information should not distinguish cells that are biologically comparable. The phrase “biologically comparable” must be made explicit. If two batches contain the same cell state, compare their integrated outputs within that state; do not require the overall batch distributions to match when their state compositions differ for biological reasons.
In an idealized formulation, for each state \(z\) with cross-batch support, the target is
where \(\mathcal{L}\) denotes a probability distribution and \(b,b'\) are batches. The approximation allows for sampling variability and for a benchmark’s prespecified tolerance. The equality is not a universal rule that all batches must have identical outputs: it is a conditional target for comparable cells. If the condition itself is biologically relevant, it may also need to be held fixed or included in the definition of the target population.
The reason to condition can be seen from the mixture identity
The overall distribution in a batch depends both on its within-state distributions and on its proportions of each state. Thus different overall distributions may arise even when the within-state distributions match. Conversely, matching overall distributions does not establish that the within-state distributions match: shifts in different states can cancel in the mixture.
This target requires overlap: the state being compared must occur in the batches under consideration, with enough observations to support a meaningful comparison. If a state occurs only in one batch, its batch-specific technical effect cannot be separated from its biology using that comparison alone. Extrapolating a “corrected” version of that state into another batch is not evidence that batch removal succeeded.
Worked Example: Why Overall Mixing Is Not the Target
Suppose a hypothetical integrated coordinate has two cell states, \(A\) and \(B\), and the target coordinate mean for each state is the same across batches. In batch 1, the means are 0 for state \(A\) and 10 for state \(B\), with equal proportions of the two states. In batch 2, the corresponding means are 1 and 11, with proportions 25% and 75%.
Within each state, the batch difference is 1 coordinate unit: \(1-0=1\) for \(A\), and \(11-10=1\) for \(B\). This is evidence of a residual batch shift under the state-conditional target.
The overall mean in batch 1 is \(0.50(0)+0.50(10)=5.00\) coordinate units. In batch 2 it is \(0.25(1)+0.75(11)=0.25+8.25=8.50\) coordinate units. The overall difference is \(8.50-5.00=3.50\) units. It is not a clean estimate of the within-state batch shift: it combines that shift with the difference in state proportions.
A benchmark focused on conditional batch removal should therefore ask whether comparable states retain batch differences, rather than treating the 3.50-unit overall difference as the batch effect. If the scientific use also requires comparing state abundance, composition is a separate biological target; it should not be silently “fixed” by forcing the two batches to have the same proportions.
Specify What Biology Must Be Preserved
“Preserve biology” is too broad to be a usable target. A benchmark should identify the particular biological features needed for the intended analysis: for example, a known condition contrast in a cell state, separation of two established states, or the presence of a state that is rare but important. Different uses can require different outputs and tolerances.
For a condition contrast in a specified cell state, let \(\Delta_g\) be the target difference in mean expression of gene \(g\) between conditions, defined in the relevant population. If adjusted expression is intended for that analysis, a preservation target may ask that the adjusted contrast remain close to \(\Delta_g\), or that it retain a prespecified direction and minimum magnitude. The contrast and its units must be specified. The target is not simply that all adjusted values resemble the unadjusted values, because some differences may be technical and others biological.
For a representation intended to distinguish known states, the target may instead concern retention of their separation or recognizable structure. For a use concerned with a rare state, the target may require that the state remain identifiable rather than being merged into a common state. These are different biological endpoints; success on one does not imply success on the others.
Targets also need a defined population. A condition contrast may apply only within a named cell state and only among biological samples with both conditions represented. A claim about cells from one sample is not automatically a claim about a wider population of donors. Cells from the same biological replicate are not independent biological replicates, so cell counts alone do not establish population-level evidence.
Worked Example: Preserve a Condition Contrast
Suppose a hypothetical study’s stated use is to estimate a treatment-associated change in a gene within one cell state. The reference mean expression is 2.0 units for control and 5.0 units for treatment. The target contrast is therefore \(\Delta=5.0-2.0=3.0\) expression units. Assume, for illustration, that the benchmark specifies retaining at least 90% of this contrast, with the same direction.
An integration produces adjusted means of 2.2 units for control and 5.0 units for treatment. Its adjusted contrast is \(\Delta^*=5.0-2.2=2.8\) expression units. The retained fraction is \(\Delta^*/\Delta=2.8/3.0=0.933\), or 93.3%. The contrast remains positive and exceeds the prespecified 90% retention threshold, so it meets this particular contrast-preservation target.
If instead the adjusted means were 3.5 and 4.0 units, then \(\Delta^*=0.5\) units and \(0.5/3.0=0.167\), or 16.7%, would be retained. That output would fail the stated target, even if its batches appeared well mixed. This calculation is meaningful only if the reference contrast is trustworthy for the intended population and the adjusted values are intended to support this expression-level question.
The Intended Use Sets the Priority
Integration errors have consequences through the analyses that follow. A shift between batches within a shared state may impair joint visualization or clustering. Attenuation of a condition contrast may impair a differential-expression analysis. Merging a rare state with a common one may compromise a study of that state even if common states align well. A benchmark should therefore state the downstream claim before deciding which errors matter most.
When several uses matter, define several targets rather than hiding trade-offs in one vague goal. A useful specification identifies which targets are essential, which are secondary, and what level of failure is unacceptable. A method may satisfy a mixing target while failing a biological-preservation target; reporting both makes that conflict visible instead of allowing one success to conceal the other.
Worked Example: A Rare State With No Cross-Batch Support
Consider a hypothetical experiment with 100 cells in each of two batches. A rare state of scientific interest occurs in 4 cells in batch 1 and in 0 cells in batch 2. The observed within-batch proportions are \(4/100=0.04\), or 4%, and \(0/100=0\), respectively.
A target requiring the rare state to mix across these two batches is not assessable from these data: there are no batch-2 cells in that state to compare with the batch-1 cells. The absence may reflect biology, sampling, or technical failure; the observed counts alone do not identify which explanation is correct.
If the intended use is to retain and study the rare state in batch 1, a defensible target is that the integration not erase its distinguishable structure or its associated biological signal. It is not defensible to claim successful cross-batch alignment for that state without cross-batch support. The benchmark report should mark this target as unassessable from the available comparison, rather than declaring success or failure of batch mixing for the rare state.
Justify Targets and Check Identifiability
A target is informative only if the data and design can address it. For each proposed batch comparison, inspect which biological states and conditions occur in each batch. If batch and condition are perfectly confounded—for example, all controls were processed in one batch and all treated samples in another—then a batch-associated difference cannot be uniquely attributed to technical batch rather than treatment. Integration cannot recover information that the design does not identify.
The same caution applies to a reference contrast. If the reference is based on very few biological replicates, or if the relevant state is absent from some samples, the target may be uncertain or restricted to a narrow population. State the scope rather than implying that the result generalizes to all donors, conditions, or cell states. Where reference uncertainty is material, describe it as a limitation of the target, not as proof that an integration method failed.
State the downstream scientific task, such as joint cell-state analysis or estimation of a condition contrast in a specified state.
Specify the batches and the biological states or conditions that must be held comparable.
Confirm that the relevant states and conditions occur in the batches and biological replicates needed for the claim. Record comparisons that cannot be assessed.
Describe which batch differences should diminish and which biological features or contrasts should remain, including any acceptable loss.
Select metrics that address each stated target, and report targets separately when they can conflict.
Common Mistakes and Alternatives
- Wrong reasoning: “The batches should be indistinguishable, so any remaining batch separation means failure.” This ignores state composition and genuine batch-specific biology. Correction: define the batch-removal comparison within shared biological states and conditions.
- Wrong reasoning: “A batch-mixing result proves the biology was preserved.” Mixing and biological preservation are distinct targets. Correction: state and assess both, using evidence relevant to the intended biological claim.
- Wrong reasoning: “Every observed biological difference must be retained.” Some observed differences may be technical artifacts, and the use may depend on only particular biological features. Correction: identify the contrasts, states, or relationships that matter before judging what was lost.
- Wrong reasoning: “A state absent from one batch should be forced to align with another batch.” The comparison has no within-state support in the absent batch. Correction: label the target unassessable for that comparison, or use additional evidence that genuinely addresses the missing support.
- Wrong reasoning: “Many cells make a condition contrast reliable.” Cells from one sample do not replace independent biological replicates. Correction: define the population-level claim in terms of the biological units that support it.
The main alternative is a single global target such as “remove batch while preserving all biology.” It is concise, but neither the desired conditional comparisons nor the relevant biological endpoints are defined. A vector of use-specific targets is more transparent, even when it reveals that no method performs best on every objective. A single combined summary can be useful only after its component targets and priorities are stated; otherwise, its trade-off is difficult to interpret.
The limitation is fundamental: target specification clarifies what counts as success, but it cannot resolve confounding, create missing overlap, or make an uncertain biological reference certain. Nor does a well-defined target guarantee that a chosen metric measures it adequately. The target, the available evidence, and the measurement must be considered together.
Check Your Understanding
Answer each question using the target, population, and use in the scenario. Brief solutions follow.
- Two batches contain different proportions of two cell states. Why can their overall integrated distributions differ even if batch removal succeeds within each state?
- A treatment contrast is 4.0 expression units before integration and 3.2 units afterward. What fraction of the contrast remains, and what does that calculation not establish by itself?
- A cell state occurs in one batch but not another. Is cross-batch mixing for that state assessable from this comparison? Explain.
- A benchmark reports strong batch mixing but does not assess any biological contrast. What conclusion is justified, and what conclusion is not?
- All treated samples were processed in one batch and all control samples in another. Why can the observed difference not be uniquely assigned to batch or condition?
Solutions. (1) Overall distributions are mixtures of within-state distributions weighted by each batch’s state proportions; different weights can produce different mixtures. (2) \(3.2/4.0=0.80\), so 80% remains. This alone does not show whether the reference contrast is valid for the target population or whether other biological features were preserved. (3) No. There is no cross-batch support for that state, so the comparison is unassessable from these data. (4) The result supports the reported batch-mixing target only; it does not establish preservation of biological contrasts. (5) Batch and condition are perfectly confounded in the design, so their contributions cannot be separated from those observations alone.