Decomposing Count Variance Across Levels

Why Decompose the Variance?

A count GLMM can have Poisson variation after conditioning on its random effects and still show greater-than-Poisson variation when those effects are averaged over. The reason is that observations with different latent effects have different conditional means. The marginal variance therefore combines variation within a specified latent setting with variation in the mean across latent settings.

This decomposition gives a precise interpretation to two sources of variation. It does not, by itself, establish that the model fits well or determine which model should be used for extra-Poisson variation. Those conclusions require further assessment. Here the target is the variance of one count response after averaging over the random effects, and how that variance can be divided into interpretable components.

Definition: Let $Y$ be a count response and $B$ denote all random effects relevant to it. Write $\lambda(B)=E(Y\mid B)$ for its conditional mean. If $Y\mid B\sim\operatorname{Poisson}(\lambda(B))$, then the marginal variance decomposition is $$ \operatorname{Var}(Y)=E\{\lambda(B)\}+\operatorname{Var}\{\lambda(B)\}. $$ The first term is the expected conditional Poisson variance. The second is variation in the conditional mean induced by the random effects.

Deriving the Basic Decomposition

Assume the conditional count model is Poisson: given the random effects $B$ and the observed covariates, $Y$ has mean and variance $\lambda(B)$. The law of total variance states that for any random variable $Y$ with finite variance, $\operatorname{Var}(Y)=E\{\operatorname{Var}(Y\mid B)\}+\operatorname{Var}\{E(Y\mid B)\}$. Substituting the Poisson conditional mean and variance gives the result.

\begin{aligned} \operatorname{Var}(Y) &=E\{\operatorname{Var}(Y\mid B)\} +\operatorname{Var}\{E(Y\mid B)\}\\ &=E\{\lambda(B)\}+\operatorname{Var}\{\lambda(B)\}. \end{aligned}

Since $E(Y)=E\{E(Y\mid B)\}=E\{\lambda(B)\}$, the marginal mean is the first term. Thus the marginal variance is at least the marginal mean under these assumptions, because $\operatorname{Var}\{\lambda(B)\}\geq 0$. Equality holds exactly when the conditional mean is constant across the random effects (up to events of probability zero). This is a statement about marginal variance; conditionally, the Poisson model still has variance equal to its mean.

For a log-link GLMM, a common specification is $\lambda(B)=\exp(\mathbf{x}^{T}\boldsymbol{\beta}+\mathbf{z}^{T}B)$. The fixed covariates are held at their specified values while taking expectations over the random effects. If $B$ has a multivariate normal distribution with mean zero and covariance matrix $G$, then $\mathbf{z}^{T}B$ is normal with variance $v=\mathbf{z}^{T}G\mathbf{z}$. Consequently, $E\{\lambda(B)\}=\exp(\mathbf{x}^{T}\boldsymbol{\beta}+v/2)$ and $\operatorname{Var}\{\lambda(B)\}=E\{\lambda(B)\}^{2}(\exp(v)-1)$. This calculation depends on the normal random-effects distribution and the log link; the total-variance identity itself does not.

Worked Calculations

Worked Example: A Normal Random Intercept

Suppose a hypothetical count response has conditional distribution $Y_j\mid b_j\sim\operatorname{Poisson}(4\exp(b_j))$, where $b_j\sim N(0,0.25)$. There are no other random effects. Find the marginal mean and each component of the marginal variance for an observation.

The assumptions are that the conditional distribution is Poisson, the log mean is $\log(4)+b_j$, and the random intercept is normal with variance $0.25$. The lognormal moment formulas apply with $v=0.25$. Thus

E(Y_j)=E(\lambda_j)=4\exp(0.25/2)=4e^{0.125}\approx4.5326.

The expected conditional Poisson variance is therefore $4.5326$ count-squared. The random-intercept contribution is

\begin{aligned} \operatorname{Var}(\lambda_j) &=(4e^{0.125})^2(e^{0.25}-1)\\ &\approx 20.5444067(0.2840254)\\ &\approx 5.8351\text{ count-squared}. \end{aligned}

Adding the unrounded contributions gives $\operatorname{Var}(Y_j)\approx 4.5326+5.8351=10.3677$ count-squared. The marginal variance exceeds the marginal mean because observations from groups with different intercepts have different conditional rates. The random-intercept term is not additional Poisson noise within a group; it is variation in the rate between groups.

Worked Example: Checking the Two Components Numerically

Consider a hypothetical lognormal conditional mean $\lambda= \exp(c+B)$, where $B\sim N(0,\tau^2)$. Suppose the parameters are chosen so that $E(\lambda)=2.1992$ counts and $\operatorname{Var}(\lambda)=1.01226$ count-squared. Equivalently, take $\tau^2=\log\{1+1.01226/(2.1992)^2\}$ and $c=\log(2.1992)-\tau^2/2$. Given $B$, let $Y$ be Poisson with mean $\lambda$.

The conditional Poisson component is $E\{\operatorname{Var}(Y\mid B)\}=E(\lambda)=2.1992$ count-squared. The random-effect component is $\operatorname{Var}\{\lambda\}=1.01226$ count-squared by the specified lognormal moments. Therefore

\operatorname{Var}(Y)=2.1992+1.01226=3.21146\text{ count-squared}\approx3.211\text{ count-squared}.

The marginal mean is $2.1992$ counts, so the marginal variance is larger by $1.01226$ count-squared. If the displayed components are rounded separately to three decimal places, they are $2.199$ and $1.012$; those displayed values sum to $3.211$, consistent with the unrounded total rounded to three decimal places.

Separating Contributions Across Levels

With multiple levels of random effects, the same identity can be applied repeatedly. Suppose $U$ is a higher-level effect and $V$ is a lower-level effect, and the conditional model is Poisson with mean $\lambda(U,V)$. First condition on $U$, then expand the conditional variance given $U$. The resulting terms are the expected conditional Poisson variance, variation in the mean over $V$ within $U$, and variation in the mean averaged over $V$ across $U$.

\begin{aligned} \operatorname{Var}(Y) &=E\{\operatorname{Var}(Y\mid U)\}+\operatorname{Var}\{E(Y\mid U)\}\\ &=E\{E[\operatorname{Var}(Y\mid U,V)\mid U]\}\\ &\quad+E\{\operatorname{Var}[E(Y\mid U,V)\mid U]\} +\operatorname{Var}\{E(Y\mid U)\}. \end{aligned}

Because the conditional distribution is Poisson, $\operatorname{Var}(Y\mid U,V)=E(Y\mid U,V)=\lambda(U,V)$. The three terms therefore have clear roles. The first is the expected Poisson variation; the second is lower-level variation in the conditional mean after fixing $U$; and the third is higher-level variation in the mean after averaging over $V$. Each term is nonnegative. The identity does not require $U$ and $V$ to be independent, but their joint distribution must be used in the conditional expectations. If the order of conditioning is changed, the allocation among level-specific terms can change; the total variance cannot.

Worked Example: Two Latent Levels

For a hypothetical count response, let the higher-level rate multiplier $M$ equal 1 or 3 with probability $1/2$ each. Let a lower-level multiplier $A$ equal 0.5 or 1.5 with probability $1/2$ each, independently of $M$. Given both multipliers, suppose $Y\sim\operatorname{Poisson}(MA)$. Identify the three variance contributions.

The conditional Poisson component is $E(MA)=E(M)E(A)=2(1)=2$ count-squared. Within a fixed higher-level value $M$, the conditional mean is $MA$. Since $E(A)=1$ and $\operatorname{Var}(A)=0.25$, its variance over the lower-level effect is $M^2(0.25)$. Also, $E(M^2)=(1^2+3^2)/2=5$, so the expected lower-level contribution is $5(0.25)=1.25$ count-squared.

After averaging over $A$, the conditional mean given $M$ is $E(MA\mid M)=M$. Since $M$ is 1 or 3 with equal probabilities, its mean is 2 and its variance is $\{(1-2)^2+(3-2)^2\}/2=1$ count-squared. The higher-level contribution is therefore 1 count-squared, and the total is $2+1.25+1=4.25$ count-squared. The marginal mean is 2 counts; the extra 2.25 count-squared consists of the two random-effect contributions.

What the Decomposition Does and Does Not Say

The decomposition is useful because it distinguishes conditional Poisson variation from heterogeneity in rates across latent settings. In a random-intercept model, for example, the second term quantifies how much variation in the conditional mean is induced by differences among groups. In a multi-level model, repeated conditioning can organize that heterogeneity by level. These are model-based variance components for the specified covariates and random-effects distribution, not direct summaries of every possible source of observed variation.

The assumptions need attention. The basic formula requires a finite marginal variance and a correctly specified conditional mean and variance. For a Poisson GLMM, the conditional variance equals the conditional mean. If the conditional response distribution has a different variance, the first component must instead be calculated as $E\{\operatorname{Var}(Y\mid B)\}$; it is not generally $E\{\lambda(B)\}$. A random-effects distribution also matters: the identity remains true for other distributions, but the lognormal moment shortcut used above may not.

This tutorial concerns the marginal variance of one response. For a sum of responses within a group, the variance also includes covariances between responses. Shared random effects can make those responses dependent marginally even when they are conditionally independent. Thus multiplying a single-observation variance by the group size is generally not justified without accounting for those covariances.

Specify what is conditioned on.
Write the conditional mean and variance given the relevant random effects and covariates.

Apply total variance.
Separate expected conditional variance from the variance of the conditional mean.

Expand by level if needed.
Use iterated conditional expectations to separate within-level and between-level mean variation.

Check the interpretation.
Verify that components sum to the marginal variance and that the conditioning assumptions match the model.

Diagnostics, Alternatives, and Common Pitfalls

A variance decomposition is a consequence of a specified model, not a diagnostic test of that model. Useful checks include comparing observed count variability with model-based predictions, examining whether residual patterns remain by group or fitted mean, and using simulated data from the fitted model to assess whether its distribution captures the observed counts. Systematic excess variation within groups can indicate that conditional Poisson variation is inadequate; systematic differences between groups can indicate that the random-effects structure or its distribution needs review. Neither pattern alone identifies a unique replacement model.

A key alternative is to change the conditional count distribution to allow variance beyond the Poisson mean. That differs conceptually from adding random effects: random effects represent latent heterogeneity structured across units or levels, while a more flexible conditional distribution changes the variation remaining after conditioning on those effects. The two ideas can also be combined, but their contributions and assumptions must then be stated separately. Choosing between them is a modeling question, not something settled by the total-variance identity alone.

Mistake: calling the marginal variance Poisson. Conditional equidispersion does not imply marginal equidispersion. Correctly, the random-effects contribution adds $\operatorname{Var}\{\lambda(B)\}$ to the expected conditional variance.
Mistake: treating a variance component as a direct observation-level error variance. The variance of the conditional mean describes heterogeneity induced by latent effects, not additional Poisson variation within a fixed latent setting.
Mistake: ignoring shared effects when analyzing totals. For a group sum, marginal covariances between responses may contribute. Compute the variance of the sum with its covariance terms rather than simply multiplying the single-response variance.
Mistake: attributing every extra-Poisson pattern to random effects. The decomposition is conditional on the model being appropriate. Check the conditional distribution, link, covariates, and random-effects structure before interpreting its components.

Key takeaway: For a conditionally Poisson GLMM, marginal variance equals the expected conditional Poisson variance plus variation in the conditional mean across random effects. Iterated conditioning can allocate that mean variation across levels, provided the conditioning structure and assumptions are made explicit.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If $Y\mid B$ is Poisson with mean $\lambda(B)$, write its marginal variance as two components and identify each one.
A hypothetical model has $E\{\lambda(B)\}=3.4$ counts and $\operatorname{Var}\{\lambda(B)\}=0.8$ count-squared. What are the marginal mean and variance?
Why can two responses be conditionally independent given a shared group effect yet dependent marginally?
In the two-level decomposition, what does the term $\operatorname{Var}\{E(Y\mid U)\}$ represent?

Solutions. (1) $\operatorname{Var}(Y)=E\{\lambda(B)\}+\operatorname{Var}\{\lambda(B)\}$; these are the expected conditional Poisson variance and the variance of the conditional mean. (2) The marginal mean is $3.4$ counts and the variance is $3.4+0.8=4.2$ count-squared. (3) Both responses depend on the same random group effect, so averaging over that shared effect induces marginal association even if their conditional covariance is zero. (4) It is between-$U$ variation in the response mean after averaging over lower-level effects; it is the higher-level contribution under this conditioning order.