Batch-Condition Confounding and Identifiability

Why Batch-Condition Overlap Matters

Suppose samples from one biological condition were all processed in one batch and samples from another condition were all processed in a second batch. If their measured expression differs, the difference may reflect biology, batch, or both. The data alone cannot tell which explanation is correct. This is a problem of identifiability: whether the design contains enough distinct comparisons to determine the parameter or contrast of interest.

The central question is not simply whether batch and condition are associated. It is whether the observed sample layout supports the particular biological comparison you want to make. Some designs make condition effects inseparable from batch effects. Others have incomplete overlap but still identify some or all condition contrasts under an appropriate model.

Model and Definitions

Consider one gene at a time, with \(n\) samples. Let \(y\) be the vector of measured expression values, \(X\) the design matrix describing the samples’ batches and conditions, and \(\beta\) the corresponding coefficient vector. A common starting point is the additive linear model

For the model to describe the mean expression, assume \(E(\varepsilon\mid X)=0\). For standard least-squares uncertainty calculations, one commonly also assumes independent errors with common variance; other error structures require appropriate uncertainty calculations. The rank and estimability checks below concern the design matrix itself and do not require normally distributed errors.

For a two-batch, two-condition example, use an intercept, an indicator for batch B, and an indicator for the treatment condition. The coefficient of the treatment indicator is the treatment-versus-control contrast in this additive model. The model assumes that the batch effect is the same in both conditions: there is no batch-by-condition interaction.

The full coefficient vector need not be identifiable for every contrast to be estimable. For example, a design can fail to separate all condition and batch effects while still supporting a comparison between two conditions observed within the same batch. Therefore, assess the contrast you need, not only whether the entire model has full rank.

Worked Example: Complete Confounding

Checking Estimability

A precise test for estimability uses the null space of \(X\), the set of vectors \(v\) for which \(Xv=0\). If \(\beta\) produces a given mean vector, then \(\beta+v\) produces the same mean vector whenever \(v\) is in this null space. Consequently, the contrast \(c^\mathsf{T}\beta\) is estimable exactly when adding any such \(v\) leaves the contrast unchanged.

This gives a short justification for the criterion. If \(c^\mathsf{T}v=0\) for every \(v\) in the null space, then \(c^\mathsf{T}(\beta+v)=c^\mathsf{T}\beta\), so the contrast does not change across coefficient vectors that fit the same means. Conversely, if some null-space vector has \(c^\mathsf{T}v\ne0\), then \(\beta\) and \(\beta+v\) fit the same means but give different contrast values. That contrast cannot be determined from the design.

For the complete-confounding example, \(v=(0,1,-1)^\mathsf{T}\) satisfies \(Xv=0\). The treatment contrast has \(c=(0,0,1)^\mathsf{T}\), so \(c^\mathsf{T}v=-1\ne0\). It is not estimable. The combined batch-plus-treatment contrast, with \(c=(0,1,1)^\mathsf{T}\), has \(c^\mathsf{T}v=0\), and is estimable.

Worked Example: Partial Overlap Under an Additive Model

Overlap, Missing Cells, and Which Contrasts Survive

For categorical batch and condition variables, a useful visual diagnostic is a table of sample counts for every batch-condition combination. A zero count marks a missing cell. The pattern of nonzero cells indicates which comparisons are directly connected by the design.

For an additive model with batch and condition main effects, represent each batch and each condition as a node in a graph. Draw an edge when samples are observed in that batch-condition combination. If this graph is connected, all batch and condition main effects are identifiable up to the chosen reference coding. If it has disconnected components, effects that compare components can be confounded. This graph criterion applies to the additive main-effects model; adding interaction terms requires additional information about the cells.

Diagnostics and Practical Choices

Before fitting a batch-adjusted model, check the sample layout directly. Tabulate sample counts by batch and condition, verify that levels are coded consistently, and inspect the rank of the exact design matrix intended for the analysis. If software reports aliased or redundant coefficients, treat that as a design finding rather than a computational nuisance. Changing the reference category or using a different full-rank coding can change coefficient labels, but it cannot create information or make a non-estimable contrast estimable.

An overlap table is often easier to interpret than a single correlation between batch and condition. A correlation may flag association, but it does not tell you precisely which contrasts the design identifies, particularly when there are several levels. Likewise, plots such as principal component plots can reveal structure in expression data, but they cannot resolve perfect confounding: the same observed separation can be attributed to batch or biology.

The main alternatives are to analyze an estimable within-batch contrast, restrict the scientific question to comparisons supported by overlap, or report that the desired contrast is not identifiable from these samples. Omitting batch does not solve confounding; it simply attributes the combined difference to whichever terms remain in the model. If an independent source of information or additional samples can connect the groups, that may support the missing comparison, but a reparameterization of the existing data cannot.

Common Mistakes and Limitations

• Calling an observed difference a condition effect. If condition and batch change together with no overlap, the difference combines their effects. Report the ambiguity rather than assigning it to biology.
• Assuming a full-rank model is automatically appropriate. Full rank establishes identifiability for the specified columns, not that the additive model is scientifically correct. In particular, an unobserved cell can leave an interaction assumption untestable.
• Treating every missing cell as complete confounding. Some incomplete designs retain enough connected comparisons to estimate main effects. Check the rank and the contrast of interest rather than relying only on the number of empty cells.
• Thinking a different coding fixes aliasing. Coding can clarify or simplify the parameterization, but it leaves the information in \(X\) unchanged.
• Confusing estimability with precision. An estimable contrast can still have a large standard error when overlap is sparse. Rank answers whether it can be identified; it does not guarantee a precise estimate.

The key limitation is structural: an unobserved comparison cannot be recovered from these data without additional assumptions or information. Design-based identifiability should therefore be checked before interpreting batch-adjusted biological contrasts.