Why Dataset Design Determines What a Benchmark Can Show
A real single-cell RNA sequencing dataset can make an integration comparison biologically relevant, but it does not automatically provide a fair test. If a biological condition occurs in only one batch, for example, an apparent batch difference may also be a real condition difference. If a cell state is absent from one sample, an integration method cannot be judged as though that state had been observed there. Dataset selection therefore determines which questions a benchmark can answer before any method is run.
The introductory material on defining benchmark targets distinguishes the target of evaluation from the data used to evaluate it. Here the focus is the data audit: whether a real dataset contains suitable batches, biological overlap, annotations, and replication for the target. The central principle is to select datasets based on their design and documented provenance, not merely on their size or popularity.
A dataset is fit for a particular benchmark only relative to an explicit target. A study designed to compare technical batches within the same biological population requires overlap of that population across batches. A study involving condition-specific states instead requires careful distinction between a true absence and a state that was simply not sampled or annotated. No single dataset is ideal for every integration question.
Audit the Design Before the Cell Matrix
Begin with sample-level metadata. For each sample, record the biological unit, condition, tissue or source, processing batch, sequencing run, and any other technical group that could be relevant. Then construct a sample-by-batch-by-condition table. Cell-level data are not a substitute for this table: thousands of cells from one donor still represent one donor for biological replication.
The key design question is whether batch is separable from biology. Suppose an expression feature is described by a simple additive model, \(Y_{ij}=\mu+\beta C_i+\gamma B_i+\epsilon_{ij}\), where \(C_i\) indicates a biological condition and \(B_i\) indicates a batch. This is only a design argument, not a claim that the model fully describes single-cell data. If every condition-A sample is in batch 1 and every condition-B sample is in batch 2, then the condition indicator and batch indicator are identical. The observed difference cannot identify \(\beta\) separately from \(\gamma\).
Thus the data identify, at most, the combined contrast \(\beta+\gamma\), not the condition and batch contributions individually. Additional cells within the same samples do not resolve this aliasing. A benchmark using such a design may still describe an end-to-end comparison on those samples, but it cannot support an unqualified claim that the method removed technical variation while preserving the condition effect.
Worked Example: Detecting Condition–Batch Confounding
Consider a hypothetical real-dataset audit with four samples. Samples A1 and A2 are condition A in batch 1; samples B1 and B2 are condition B in batch 2. Suppose the intended benchmark is to distinguish technical batch variation from the condition contrast.
Let \(C=0\) for condition A and \(C=1\) for condition B, and let \(B=0\) for batch 1 and \(B=1\) for batch 2. The sample-level design rows \((1,C,B)\) are \((1,0,0)\) for A1 and A2, and \((1,1,1)\) for B1 and B2. The condition and batch columns match exactly.
For a feature with a hypothetical mean of 5 counts on the chosen transformed scale in A samples and 9 in B samples, the observed contrast is \(9-5=4\) units. Under the additive representation, that contrast equals \(\beta+\gamma=4\). Values such as \(\beta=4,\gamma=0\) and \(\beta=0,\gamma=4\) fit the same contrast, as do infinitely many other pairs summing to 4.
Conclusion. The dataset has two samples per group, but replication does not repair the exact design confounding. It is unsuitable for estimating separate condition and batch contributions. It could be retained for a more limited descriptive comparison, provided that limitation is stated.
When batch and biology are not perfectly confounded, check the degree of overlap rather than treating the design as automatically adequate. Partial overlap can support some comparisons, but estimates may depend heavily on the few samples or states that connect groups. State coverage should be examined at the sample level as well as the cell level: a state represented by many cells in one sample is not equivalent to replication of that state across several independent samples.
Shared States, Annotations, and Replication
For each candidate dataset, list the benchmark-relevant states and tabulate their presence by sample and batch. Ask whether the states used to assess shared biological structure occur in the batches being compared, and whether rare or condition-specific populations are intended to be part of the target. A state absent from a batch may indicate genuine biology, insufficient sampling, filtering, or annotation failure. The dataset alone may not distinguish these explanations.
Annotations need their own audit. Record how labels were assigned, whether they are available at a broad or fine resolution, whether the same rules were applied across samples, and whether uncertain or unassigned cells are retained. Labels derived from a common reference can be useful, but they are not independent ground truth if the reference itself lacks the relevant state. Broad, consistent annotations may be more defensible than detailed labels whose distinctions are poorly supported.
Worked Example: Quantifying Shared-State Coverage
Suppose a hypothetical candidate has three batches, each containing two independent samples. The benchmark plans to examine two annotated states, S1 and S2. The audit finds S1 in both samples in all three batches. S2 is found in both batch 1 samples, one of the two batch 2 samples, and neither batch 3 sample.
For S1, batch-level coverage is \(3/3=1\), or 100%, because it occurs in all three batches. For S2, coverage is \(2/3=0.667\), or 66.7%, because it occurs in two of the three batches. Across the six samples, S2 is present in \(3/6=0.500\), or 50.0%, of samples.
These arithmetic summaries describe coverage; they do not establish why S2 is unobserved in batch 3. The benchmark can use S1 for a shared-state comparison across all three batches. It should not treat S2 as a shared state across all batches. A comparison restricted to batches 1 and 2 is possible, but it has only one batch 2 sample containing S2, so that limitation must accompany the result.
The audit also reveals that cell abundance alone could be misleading. If the one batch 2 sample contains 400 S2 cells, those cells still come from one sample. They do not provide two independent sample-level observations.
Replication should be counted at the level at which biological variation arises. For studies with multiple donors, tissues, or animals, record the number of independent units per condition and batch, and whether units are paired across conditions or processing groups. Technical replicates can help characterize technical variability, but they do not create additional independent donors. A benchmark based on a single biological unit may be useful as a case study, but its conclusions should not be generalized as though it sampled between-unit variation.
Worked Example: Separating Cells From Biological Replicates
A hypothetical dataset contains 1,200 cells from four donors: 300 cells from each donor. Each donor contributes cells to both of two processing batches, and all four donors have samples from the same broad tissue state. The intended benchmark concerns whether integration can be compared across the two batches without losing this shared state.
There are \(4\) independent biological units, not \(1{,}200\). Each batch contains cells from all four donors, so the sample-level design has cross-batch overlap in donor identity. The biological replication count for this design is four donors; the 300 cells per donor improve cell-level characterization but do not change that count.
This structure is more informative about batch comparisons than a design in which each batch contains different donors. However, the audit must still verify that sample handling and batch assignment are documented, that the state labels are comparable across donors, and that the relevant state is observed in each batch. If each donor's material was split across batches, note that pairing explicitly; do not describe the two batches as independent donor groups.
A Practical Dataset Selection Workflow
A defensible selection process can be organized as a short sequence of design checks. These checks do not rank integration methods; they establish what evidence the chosen dataset can provide.
Name the batch contrast and the biological structure that the benchmark intends to retain. State which samples and states are in scope.
Confirm the source, sample definitions, processing groups, conditions, and annotation process. Identify missing or ambiguous metadata before analysis.
Tabulate conditions and states by batch and sample. Flag exact confounding, sparse connecting samples, and states observed in only one batch.
Report donors, specimens, or other biological replicates separately from cells and technical replicates. Note pairing and unequal replication.
Specify which conclusions are supported, which are conditional on annotations or sampling, and which biological questions remain unidentifiable.
No numerical threshold for the number of cells or samples makes a dataset universally adequate. Suitability depends on the benchmark target, state rarity, sample heterogeneity, and whether the design has the necessary overlap. A compact dataset with clear sample-level structure can be more informative for a particular question than a much larger atlas with uncertain provenance or extensive confounding.
Diagnostics, Alternatives, and Limits
Several warning signs should prompt a narrower interpretation or a different dataset: batch labels align exactly with condition; relevant states appear in only one batch; annotations change resolution across samples; sample identifiers are missing; or a large cell total is driven by one or two biological units. Also inspect whether exclusions, quality filters, or label assignments were applied differently by batch. These checks do not prove that a dataset is unusable, but they reveal where benchmark conclusions may depend on unverified assumptions.
Real datasets offer realistic technical and biological complexity, but their biological truth is usually incomplete. Labels can be uncertain, states can be missed, and measured differences may combine biology with processing. Curated reference datasets may provide consistent annotations, yet can underrepresent novel or condition-specific states. A controlled experiment with balanced batches and replicated biological units can strengthen interpretation, but it still does not reveal every cell's complete biological identity. These are complementary design choices, not guarantees of truth.
A key limitation is that absence from the observed data is not proof of biological absence. A state may be rare, lost during processing, or missed by sampling. Likewise, an annotated shared state is not necessarily biologically identical across batches. Dataset documentation and sample-level overlap support careful claims; they do not turn imperfect labels into an error-free ground truth.
Check Your Understanding
Answer each question using the design principles above. The solutions follow the questions.
- Two conditions are each processed in a separate batch, with five donors per condition. Can the condition effect be separated from the batch effect from this design alone?
- A cell type occurs in 90% of cells in one sample and in no other sample. Does that establish replication of the cell type?
- What information should be reported when a dataset has many cells but only two biological donors?
- Why is a reference-based cell label not automatically complete biological ground truth?
Solutions. (1) No. If condition and batch indicators coincide, their effects are aliased; additional donors within the same confounded groups do not separate them. (2) No. The cell type is represented in one sample, so the observation does not establish replication across independent biological units. (3) Report the two-donor replication count separately from the cell count, describe the sample and batch structure, and limit generalization about between-donor variability. (4) The reference may omit relevant states or encode uncertain labels, so agreement with it measures consistency with that reference rather than complete knowledge of each cell's biology.