Why Diagnose Batch Dependence?
A cell-type annotation method can produce different predicted labels or confidence scores in different batches. That pattern may reflect technical dependence: for example, a batch-specific processing effect could alter measured expression and thereby influence predictions. But a raw comparison between batches can also differ simply because the batches contain different proportions of cell types, whose predictions or confidence naturally differ. The central task is therefore to ask whether predictions vary by batch within comparable cell types, rather than merely comparing batch-wide totals.
This is a diagnostic of the prediction process, not by itself a test of whether the labels are correct. When reliable reference labels are available, they can define the cell types used for adjustment. When only predicted labels are available, conditioning on those labels can be circular: the same predictions are being used both to define the groups and to assess their behavior. In that case, describe the analysis as exploratory and consider whether a separately supported, broader grouping is available.
Composition Can Mimic a Batch Effect
Let \(B\) denote batch, \(T\) cell type, and \(\hat{Y}\) the predicted label. The overall proportion receiving a particular predicted label \(k\) in batch \(b\) is a weighted average of the within-type proportions:
The weights \(P(T=t\mid B=b)\) are the batch’s cell-type composition. Thus, the overall rate can differ across batches even when every within-type prediction rate is identical. A useful descriptive comparison uses a common set of weights \(w_t\), such as the pooled composition, for each batch:
Standardization answers a specific question: what would the batch-specific prediction rate be if each batch had the same chosen cell-type mix? The weights should be reported, and types absent from a batch cannot be estimated there without additional assumptions.
Worked Example: A Raw Difference Explained by Composition
Suppose two hypothetical batches contain a common type and a rare type. The target label is the label expected for the common type. In each batch, 90% of common-type cells and 20% of rare-type cells receive that label. Batch A is 90% common and 10% rare; Batch B is 10% common and 90% rare.
Raw rates. For Batch A, the target-label rate is \(0.90(0.90)+0.10(0.20)=0.83\), or 83%. For Batch B, it is \(0.10(0.90)+0.90(0.20)=0.27\), or 27%. The raw difference is 56 percentage points.
Standardized rates. Give each type a common weight of 0.50. For either batch, the standardized rate is \(0.50(0.90)+0.50(0.20)=0.55\), or 55%. There is no within-type difference in these data; the raw contrast arises entirely from the different compositions.
This calculation is descriptive. It does not establish that composition is the only explanation in a real data set, and it depends on having comparable, correctly defined cell types in both batches.
Testing Conditional Differences in Labels
For two batches and a binary outcome, such as whether a cell receives a specified label, a logistic regression can model the probability of that label using batch and cell type. For multiple predicted labels, a multinomial regression models the probability of each category. A likelihood-ratio test compares a model containing cell type with one that also contains batch. This tests whether batch adds information about the predicted label conditional on the included cell types. If batch effects may differ by type, include batch-by-type interactions and test the batch terms and interactions jointly.
These tests rely on an adequate model, independent observational units for the stated standard errors, and enough observations to estimate the relevant parameters. With cell-level data, cells from the same donor or biological sample are usually correlated. Treating every cell as independent can make standard errors too small and p-values misleading. Use a method that reflects the design, such as sample-level summaries, a mixed-effects model, or cluster-robust inference when appropriate and supported by enough independent clusters. If there are only a few biological samples, a cell-level p-value does not create additional biological replication.
Worked Example: A Conditional Label Test and Its Unit of Analysis
In a hypothetical comparison within one trusted cell type, 50 independent observational units in Batch A yield 36 target-label predictions and 14 other predictions. In Batch B, 24 of 50 receive the target label and 26 receive another label. For this calculation, suppose the units really are independent; if these were cells from a small number of samples, this assumption would not be justified.
Under the null hypothesis that prediction and batch are independent, the pooled target-label proportion is \(60/100=0.60\), and the pooled other-label proportion is \(40/100=0.40\). Each batch has 50 units, so the expected counts in each batch are 30 target and 20 other. All expected counts exceed 5.
The Pearson chi-square statistic is the sum of squared observed-minus-expected differences divided by expected counts:
There is one degree of freedom, so the two-sided chi-square p-value is approximately 0.0143. At \(\alpha=0.05\), this is evidence of an association between batch and predicted label within this cell type, under the independence assumption. The observed target-label rates are 72% and 48%, a difference of 24 percentage points. If the 100 units were cells clustered within donors or samples, the calculation would not provide a valid cell-independent test; inference should instead reflect the sample structure.
For several cell types, the same logic extends through a regression model or a stratified analysis. A single pooled test can hide opposite within-type patterns, so inspect type-specific rates and, when scientifically relevant, assess interaction. Sparse categories can cause unstable estimates or unreliable large-sample tests; combining categories is defensible only when scientifically meaningful, not simply because it improves a p-value.
Testing Differences in Confidence
Confidence may be a maximum predicted probability, a margin, or another defined score. State exactly which score is being analyzed and its scale. A linear model can compare mean scores across batches while adjusting for cell type, provided the conditional mean is adequately modeled and residual-based inference is reasonable. Because scores are bounded and often skewed, inspect residuals and fitted values; alternatives include a suitable generalized model or comparisons of sample-level summaries. The method should also account for repeated cells within samples. Do not treat a small change in a confidence score as evidence of lower accuracy unless labels are validated against an appropriate reference.
Worked Example: Comparing Confidence Using Sample-Level Means
Suppose a hypothetical study has three independent biological samples per batch. For the same cell type, the sample-level mean confidence scores are 0.70, 0.72, and 0.74 in Batch A, and 0.80, 0.82, and 0.84 in Batch B. Assume the samples are independent, the score summaries are comparable, and a pooled-variance two-sample t procedure is reasonable for these sample-level means.
The means are 0.72 and 0.82, so the estimated difference (Batch B minus Batch A) is 0.10, or 10 percentage points on the probability scale. In each batch the sample variance is 0.0004: for Batch A, the squared deviations from 0.72 sum to \(0.0004+0+0.0004=0.0008\), and division by \(3-1\) gives 0.0004. Batch B has the same variance.
The pooled variance is \(s_p^2=[(3-1)(0.0004)+(3-1)(0.0004)]/(3+3-2)=0.0004\), so \(s_p=0.0200\). The standard error of the difference is \(0.0200\sqrt{1/3+1/3}=0.01633\), and the test statistic is \(t=0.10/0.01633=6.124\), with \(df=4\). The two-sided p-value is approximately 0.0036. A 95% t interval is \(0.10\pm 2.776(0.01633)\), or approximately 0.0547 to 0.1453.
Under the stated assumptions, the data support a difference in mean sample-level confidence for this cell type; the estimated difference is 10 percentage points, with a 95% interval from about 5.47 to 14.53 percentage points. The small number of independent samples makes the interval and test sensitive to assumptions. This result does not show that batch caused the difference or that either batch is less accurate.
Assumptions, Diagnostics, and Alternatives
Before interpreting a test, check that cell-type definitions are comparable across batches, that the modeled batch and type categories are represented in the data, and that the inferential unit matches the sampling design. Plot within-type predicted-label proportions and confidence distributions by batch, and show sample-level variation where possible. Large discrepancies between cell-level and sample-level summaries can signal that a few samples dominate the apparent result.
A useful diagnostic is to compare raw and composition-standardized rates. A large raw difference that shrinks after standardization is consistent with a composition explanation; a remaining within-type difference warrants further investigation. It is still an association, not proof of a technical cause. Batch may be confounded with donor, condition, tissue, or another variable. If every sample from one condition was processed in one batch and every sample from another condition in a different batch, the data cannot separate those effects without additional design information.
Alternative approaches answer related but distinct questions. A chi-square test is convenient for a single categorical comparison but does not automatically handle multiple cell types or sample clustering. Logistic or multinomial regression adjusts for measured covariates and can represent interactions, but depends on model adequacy and sufficient data. Standardization gives interpretable adjusted rates but is primarily descriptive unless uncertainty is calculated using a valid sampling model. For confidence, a regression or sample-level comparison targets a score difference, whereas a label test targets category probabilities. None of these methods alone establishes that predictions are accurate or identifies the biological or technical mechanism.
A Practical Analysis Sequence
Specify the predicted label or confidence score, the batch variable, and the source of cell-type groupings used for adjustment.
Tabulate cell-type counts by batch and identify absent types, sparse categories, and imbalanced sample representation.
Display batch-specific rates or confidence summaries by type, alongside raw and composition-standardized summaries.
Use a categorical or score model with cell type and batch, including interactions when warranted, and account for biological-sample clustering.
Report effect sizes and uncertainty in context, note confounding and model limitations, and describe the result as association unless the design supports causal attribution.
Check Your Understanding
Answer each question, then compare with the solutions.
- Why can overall target-label rates differ across batches even when within-type rates are identical?
- A raw batch contrast is 20 percentage points but the rates standardized to a common cell-type mix are equal. What does this suggest, and what does it not prove?
- Why can a chi-square test treating thousands of cells as independent give an overly small p-value?
- What question does a batch-by-cell-type interaction address?
- A batch coefficient is statistically significant in a confidence model. Does that establish that predictions in that batch are less accurate?
Solutions. (1) The batches can have different cell-type weights, and overall rates are weighted averages of the within-type rates. (2) The contrast is consistent with composition accounting for the raw difference in the observed groups; it does not prove that composition is the only explanation or establish a causal mechanism. (3) Cells from the same sample are correlated, so treating them as independent exaggerates the effective sample size and can underestimate uncertainty. (4) It tests whether the batch association varies across cell types rather than being adequately summarized by a common batch effect. (5) No. It establishes an adjusted association in the modeled score, not accuracy; accuracy requires comparison with suitable reference labels, and significance alone does not establish causation.