Why Perfect Confounding Matters
In a multi-omics study, a measured difference between cohorts may reflect biology, technical processing, or both. An additive model can represent these contributions, but a model formula alone cannot guarantee that the data distinguish them. If every sample from one cohort is processed in one batch and every sample from the other cohort is processed in another, the cohort and batch indicators may be identical. Then their separate effects cannot be recovered from the observed measurements.
The issue is not merely that estimates might be imprecise. Under perfect confounding, different values of the cohort and batch coefficients can produce exactly the same fitted mean for every observed sample. Consequently, the data cannot choose between those coefficient values, even with arbitrarily precise measurements on the same confounded design.
Model and Definition of Identifiability
For one omics feature, let \(Y_i\) be the measured value for sample \(i\), in the feature's measurement units. Let \(C_i\) equal 1 for a designated cohort and 0 otherwise, and let \(B_i\) equal 1 for a designated batch and 0 otherwise. Consider the additive model
Here, \(\mu\) is the mean for the reference cohort in the reference batch, \(\beta\) is the cohort-associated shift, and \(\gamma\) is the batch-associated shift. The error \(\varepsilon_i\) represents remaining sample-level variation. For the usual mean-based interpretation, assume \(E(\varepsilon_i\mid C_i,B_i)=0\). For the likelihood comparisons below, one may additionally assume independent errors with a common distribution whose location parameter is identified by its mean; normal errors with common variance are a familiar example. The identifiability conclusion itself follows from the model's means and does not require normality.
This is a property of the model together with the observed design, not a statement about whether a particular dataset is large or whether an estimation algorithm is sophisticated. More observations with the same confounded pattern may improve estimation of observable group means, but they do not create information that separates the two effects.
Deriving the Non-Identifiability
Suppose \(C_i=B_i\) for every observed sample. Substituting \(B_i=C_i\) into the model gives
Only the sum \(\beta+\gamma\) appears in this expression. To see the consequence directly, for any real number \(t\), define \(\beta'=\beta+t\) and \(\gamma'=\gamma-t\). Then
Thus every such pair \((\beta',\gamma')\) gives the same mean for every observed sample. Under the same error model, it also gives the same distribution of the observations and the same likelihood. No estimator can use these observations to determine which allocation of the combined shift to cohort and batch is correct.
There is an important condition when describing what is identifiable. If the observed samples include both \(C_i=B_i=0\) and \(C_i=B_i=1\), the two group means identify \(\mu\) and \(\beta+\gamma\): the first group has mean \(\mu\), and the difference between the second and first group means is \(\beta+\gamma\). If all samples have indicator 0, the mean is just \(\mu\); neither effect contributes to the observed means. If all samples have indicator 1, the observed mean is \(\mu+\beta+\gamma\), so the combined shift cannot be separated from \(\mu\).
Worked Example: Two Effects, One Observable Shift
Suppose the indicator values are identical for all samples, with both indicator groups represented. The hypothetical feature measurements are 5, 7, and 6 units for the samples with \(C_i=B_i=0\), and 10, 11, and 9 units for the samples with \(C_i=B_i=1\). Find the observed group means and determine which effects these data identify.
The reference-group mean is
The group with both indicators equal to 1 has mean
Under the stated model, these means estimate \(\mu\) and \(\mu+\beta+\gamma\), respectively. Their difference is
so the observed data support an estimated combined cohort-and-batch shift of 4 units. They do not say how much of that shift is due to cohort. For example, \(\beta=1.5\) and \(\gamma=2.5\), or \(\beta=4\) and \(\gamma=0\), both give \(\beta+\gamma=4\) and the same fitted group means. These are alternative model allocations, not conclusions the data can distinguish.
Worked Example: When Only the Indicator-1 Group Is Observed
Suppose a hypothetical study contains four samples, all with \(C_i=B_i=1\), and their feature measurements are 12, 13, 11, and 12 units. What combination of model parameters is informed by the observed mean?
The sample mean is
For each observed sample, the model mean is \(\mu+\beta+\gamma\). Thus this group mean informs that whole sum, not \(\beta+\gamma\) by itself. For instance, \((\mu,\beta+\gamma)=(10,2)\) and \((11,1)\) both give a fitted mean of 12 units. The data do not separate the reference mean from the combined shift, and they also do not separate the cohort and batch contributions within that shift.
By contrast, if all observed samples had \(C_i=B_i=0\), their mean would inform \(\mu\) alone. Since both effects would multiply zero for every sample, those observations would provide no information about \(\beta\) or \(\gamma\).
What Additional Design Information Can Do
The way out is not a more elaborate adjustment applied to exactly the same confounded observations. Separation requires information in which cohort and batch do not always change together, or an independently justified source of information that constrains one of their effects. For example, measuring samples from the same cohort across batches, or including multiple cohorts within a batch, can provide comparisons that vary one indicator while holding the other fixed.
Worked Example: Added Comparisons Separate the Shifts
Consider a hypothetical follow-up design with measurements in all four cohort-and-batch combinations. For simplicity, suppose the measurements in each cell are as follows, in feature units: \((C=0,B=0)\): 5 and 7; \((C=1,B=0)\): 7 and 9; \((C=0,B=1)\): 6 and 8; \((C=1,B=1)\): 8 and 10. Calculate the cell means and use the model to identify the additive effects.
The four means are
In the reference cell, the model mean is \(\mu\), so \(\mu=6\) units. Comparing cohort 1 with cohort 0 while holding batch at 0 gives
Comparing batch 1 with batch 0 while holding cohort at 0 gives
The final cell also agrees with the additive model: \(\mu+\beta+\gamma=6+2+1=9\) units, its observed mean. Unlike the perfectly confounded example, these comparisons vary cohort without changing batch and vary batch without changing cohort. That is the design information needed to distinguish the two shifts under this model.
In practice, additional information might come from a study design with overlapping cohort and batch assignments, or from external calibration data that genuinely constrain a technical effect. Any such constraint must be justified by the measurement process or study design. Merely assigning a convenient value to a coefficient does not make that value an empirical finding.
Diagnostics, Alternatives, and Limitations
A direct diagnostic is to tabulate the observed sample counts for each cohort-by-batch combination. If only the \((0,0)\) and \((1,1)\) combinations occur, then the two indicators are identical in the observed data. In that case, a model fit may show an aliased coefficient, omit one of the indicators, report a non-unique solution, or return results that depend on software conventions. These are possible symptoms, not a requirement that every program display the problem in the same way. The central check is whether the observed design contains comparisons that separate the indicators.
| Approach | What it can provide | What it cannot provide under perfect confounding |
|---|---|---|
| Fit both additive indicators | Fitted means and, if both indicator groups occur, the combined shift | A unique cohort coefficient and batch coefficient |
| Omit one indicator | A single coefficient describing the observed group difference under that specification | Evidence that the retained coefficient is purely biological or purely technical |
| Impose an external constraint | A decomposition conditional on that constraint | A decomposition learned from the confounded observations alone |
| Obtain additional design or calibration data | Comparisons that may separate cohort and batch contributions | A guaranteed answer if the added information does not actually distinguish the effects |
A common mistake is to say, “The adjusted model reports a cohort coefficient, so the batch effect has been removed and the remaining coefficient is biological.” The correction is that a coefficient reported after dropping or constraining a confounded term depends on that modeling choice. Without separating design information or a defensible external constraint, the observed difference cannot be attributed uniquely to either source.
Perfect confounding is a structural limitation, not a diagnosis that every cohort difference is technical or that no useful analysis is possible. One may still describe the observed group means, estimate their combined difference when both indicator groups are represented, or make predictions for the observed design. But attributing that combined difference to separate biological and technical causes requires information not contained in the perfectly confounded measurements. This conclusion is specific to the stated additive model; a different model cannot manufacture the missing comparison merely by being more complex.
Check Your Understanding
Answer each question, then compare with the solutions below.
- In the additive model, why do \((\beta,\gamma)=(2,3)\) and \((4,1)\) produce the same fitted means when \(C_i=B_i\) for every sample?
- Both indicator groups are observed, and their means are 8 and 13 units for indicator 0 and indicator 1, respectively. What is identified about the model means and shifts?
- All observed samples have \(C_i=B_i=1\), and their mean is 15 units. Is \(\beta+\gamma\) identified separately from \(\mu\)? Explain.
- A software package drops the batch indicator and reports a cohort coefficient of 5 units for a perfectly confounded design. Can this coefficient be interpreted as a uniquely identified biological effect? Why or why not?
Solutions. 1. Both pairs sum to 5, so the model mean \(\mu+(\beta+\gamma)C_i\) is unchanged for every sample. 2. The reference mean is estimated as 8 units, and the combined shift is estimated as \(13-8=5\) units; the separate cohort and batch effects are not identified. 3. No. The observed mean informs \(\mu+\beta+\gamma=15\) units, not the sum separately from \(\mu\). 4. No. Dropping batch assigns the observed combined difference to the retained cohort term by model specification; the confounded data do not uniquely support that attribution.