The Law Of Total Variance For Counts

Why Separate the Two Parts of Count Variability?

For a gene, a count recorded in one cell can differ from a count in another for more than one reason. Even among cells regarded as comparable, the observed count may vary around its conditional average. Also, the conditional average itself may differ across cell states, library exposures, or other recorded characteristics. The law of total variance gives an exact accounting identity for these two kinds of variation.

This identity does not, by itself, say what biological or technical mechanism produced either component. Its interpretation depends on the variable we condition on and on how well that variable represents the relevant grouping of cells. It is a way to organize variance, not a claim that the components are separate causal mechanisms.

In single-cell applications, \(Y\) can be the observed count for one gene in a randomly selected cell from a defined population of cells. \(Z\) might indicate a cell type, a biological state, or a library-size group. The first term averages the count variance within values of \(Z\), weighting each value by its probability in the population. The second term measures how much the expected count changes across those values. Both terms have units of squared counts, and their sum is the marginal variance of \(Y\).

Deriving the Decomposition

The identity follows by adding and subtracting the conditional mean. Define \(m(Z)=E(Y\mid Z)\) and the conditional residual \(\varepsilon=Y-m(Z)\). Then \(Y-E(Y)=\varepsilon+\{m(Z)-E(Y)\}\). Squaring and taking expectations yields

The cross term is zero. In fact, \(m(Z)-E(Y)\) is determined by \(Z\), while the conditional residual has \(E(\varepsilon\mid Z)=0\). Therefore,

Also, \(E(\varepsilon^2)=E[\operatorname{Var}(Y\mid Z)]\), because the conditional residual has conditional mean zero, and \(E[(m(Z)-E(Y))^2]=\operatorname{Var}(E[Y\mid Z])\). Substitution gives the stated identity. No Poisson, negative binomial, or other specific count distribution is required. The assumptions are that the variables and conditional moments are defined and that \(Y\) has a finite second moment; the conditioning variable may be discrete or continuous.

Applying the Identity to Gene Counts

For a single gene, suppose a cell is sampled from a target population and \(Z\) records a characteristic available for conditioning. For each possible value \(z\), the conditional mean \(E(Y\mid Z=z)\) describes the average count among cells with that characteristic, while \(\operatorname{Var}(Y\mid Z=z)\) describes their within-group count variability. The identity says that the population-wide variance can be recovered by averaging the within-group variances and adding the variance among the group means.

The Conditioning Variable Changes the Split

The total variance belongs to the marginal distribution of \(Y\); the two components depend on what information is placed in \(Z\). A finer conditioning variable can move variation from the expected conditional variance into variation among conditional means. The sum remains unchanged. This matters when deciding whether a component represents variability within a scientifically meaningful group or variability left unexplained by the chosen grouping.

Estimation, Diagnostics, and Common Pitfalls

The law is an exact population identity. In data, its terms must be estimated, and estimates can be unstable when groups are small or when the conditional mean varies in ways not captured by \(Z\). A practical calculation should make the target cell population and conditioning variables explicit. It should also check whether each group has enough observed cells to estimate its mean and variance usefully, and whether the grouping leaves important systematic differences in counts within groups.

For discrete groups with probabilities \(p_z\), the population terms can be written as

In a sample, replacing probabilities and conditional moments by estimates does not guarantee exact equality for every chosen estimator. For example, sample variances commonly use denominator \(n-1\), whereas a direct decomposition of observed squared deviations may use denominator \(n\). When reporting an empirical decomposition, state the estimator and denominator, and check numerically that the reported components use a consistent convention.

• Using one group’s variance instead of an average. The first term averages conditional variances using the distribution of \(Z\); it is not simply the variance in a convenient or largest group.
• Forgetting the variance of the conditional means. Averaging within-group variances alone omits between-group differences and generally does not equal the marginal variance.
• Treating the split as intrinsic. The components depend on the conditioning variable. Changing the grouping can change each component even though their sum stays fixed.
• Claiming a mechanism from the identity. A large component is not, by itself, proof of a particular biological or technical cause. The identity is descriptive and does not establish causation.

A direct marginal variance calculation is useful when the only question is how variable counts are overall. The total-variance decomposition answers a different question: how much of that variance is average variability conditional on the selected information, and how much reflects differences in conditional means? Stratified summaries can help display those differences, but they should be interpreted alongside group sizes and the definition of the target population. The earlier material on sampling noise and biological variation motivates separating kinds of variation; this identity provides a precise accounting framework without assigning causes to the terms.

The main limitation is that the identity does not choose an appropriate \(Z\), supply reliable estimates from sparse data, or identify a count distribution. It also does not imply that conditioning removes measurement noise or that the conditional variance has the same value in all groups. Those are additional modeling or scientific questions. The useful diagnostic is to verify both the arithmetic and the interpretation: the weighted conditional variances plus the variance of the conditional means must equal the marginal variance under the same population definition.