Poisson Equidispersion Conditional on Random Effects

Why the Conditioning Matters

A Poisson GLMM describes count responses after accounting for observed predictors and latent group or subject effects. Its familiar variance-equals-mean property applies to the response distribution conditional on the random effects. That qualification matters: different values of the random effects imply different conditional means, and variation among those means can make the overall, marginal response variance larger than its marginal mean.

The benchmark is a statement about the assumed conditional distribution, not a rule that every observed count or every small sample must have its sample variance equal to its sample mean. To use it correctly, specify the conditional model first, calculate its mean at the relevant covariate and random-effect values, and compare variation to that conditional benchmark.

The Conditional Poisson GLMM

Let $Y_{ij}$ be a count for observation $i$ in group $j$, let $x_{ij}$ denote its fixed-effect predictors, and let $b$ collect the random effects in the model. A common Poisson GLMM with a log link is

Y_{ij}\mid b \sim \operatorname{Poisson}(\mu_{ij}), \qquad \log(\mu_{ij})=x_{ij}^{\mathsf T}\beta+z_{ij}^{\mathsf T}b+\log(E_{ij}),

where $\beta$ is the fixed-effect coefficient vector, $z_{ij}$ is the random-effect design vector, and $E_{ij}$ is an exposure when a count is measured over unequal observation times or opportunities. If exposure is not needed, omit $\log(E_{ij})$. The conditional mean is therefore $\mu_{ij}=\exp(x_{ij}^{\mathsf T}\beta+z_{ij}^{\mathsf T}b+\log(E_{ij}))$, which is positive.

Definition: A response is conditionally equidispersed under a Poisson GLMM when, for each observation and given its predictors and random effects, its conditional variance equals its conditional mean: $$ E(Y_{ij}\mid b)=\operatorname{Var}(Y_{ij}\mid b)=\mu_{ij}. $$

This result requires the conditional response distribution to be Poisson. The response must be a nonnegative integer count, and the model must specify a valid conditional mean. In the usual GLMM formulation, responses are also assumed conditionally independent given the random effects and predictors. The distribution assigned to $b$ is needed to complete the random-effects model, but the variance-equals-mean result itself holds for any fixed value of $b$; it does not depend on the random effects being normally distributed.

Why a Poisson Response Has Equal Conditional Mean and Variance

For a Poisson random variable $Y$ with parameter $\mu>0$, the probability mass function is $P(Y=y)=e^{-\mu}\mu^y/y!$ for $y=0,1,2,\ldots$. Fix the predictors and random effects, so that the conditional parameter is $\mu_{ij}$. The following calculation establishes the benchmark for that fixed conditional mean.

\begin{aligned} E(Y\mid b) &=\sum_{y=1}^{\infty}y e^{-\mu}\frac{\mu^y}{y!}\\ &=e^{-\mu}\mu\sum_{y=1}^{\infty}\frac{\mu^{y-1}}{(y-1)!} =\mu. \end{aligned}

For the variance, first calculate the second factorial moment. For $y\geq2$, $y(y-1)/y!=1/(y-2)!$. Thus

\begin{aligned} E\{Y(Y-1)\mid b\} &=e^{-\mu}\mu^2\sum_{y=2}^{\infty}\frac{\mu^{y-2}}{(y-2)!}\\ &=\mu^2. \end{aligned}

Since $Y^2=Y(Y-1)+Y$, it follows that $E(Y^2\mid b)=\mu^2+\mu$. Subtracting the squared conditional mean gives

\operatorname{Var}(Y\mid b) =E(Y^2\mid b)-\{E(Y\mid b)\}^2 =(\mu^2+\mu)-\mu^2 =\mu.

Replacing $\mu$ with the model-specific $\mu_{ij}$ proves conditional equidispersion in the Poisson GLMM. The calculation applies separately at every covariate pattern and random-effect value; it does not say that all observations share one common mean or variance.

Worked Example: Calculating the Conditional Benchmark

Suppose a hypothetical count model records events per 30-day period. For one person, let the fixed part of the linear predictor be $\log(2)+0.4x$, with $x=1$, and let the person's random intercept be $b=-0.1$. Assume the exposure is one 30-day period. Calculate the conditional mean, variance, and probability of observing zero events.

The conditional linear predictor is $\eta=\log(2)+0.4(1)-0.1=\log(2)+0.3$. Therefore

\mu=\exp\{\log(2)+0.3\}=2e^{0.3}\approx2.700.

Conditional on this predictor and random effect, the model says $Y\mid b\sim\operatorname{Poisson}(2.700)$. Hence $E(Y\mid b)=2.700$ events per 30-day period, and $\operatorname{Var}(Y\mid b)=2.700$ events-squared. The conditional standard deviation is $\sqrt{2.700}\approx1.643$ events. The probability of zero events is

P(Y=0\mid b)=e^{-2.700}\frac{2.700^0}{0!}=e^{-2.700}\approx0.06721.

This is a model-based conditional calculation. It does not imply that the realized count must be close to 2.700, or that one observation can reveal its own variance.

Worked Example: Different Groups, Different Conditional Means

Consider a hypothetical random-intercept model for counts per fixed observation period, with $\log(\mu_{ij})=\log(4)+b_j$. For two groups, suppose the random intercepts are $b_1=-0.5$ and $b_2=0.5$. Calculate each group's conditional mean and variance.

For group 1,

\mu_1=\exp\{\log(4)-0.5\}=4e^{-0.5}\approx2.426, \qquad \operatorname{Var}(Y\mid b_1)=2.426.

For group 2,

\mu_2=\exp\{\log(4)+0.5\}=4e^{0.5}\approx6.595, \qquad \operatorname{Var}(Y\mid b_2)=6.595.

Both groups satisfy conditional equidispersion, but they do not have the same conditional mean or variance. Comparing the two groups' raw counts as if they had a common Poisson mean would ignore the group effects. The equality applies within each specified conditional distribution.

Conditional and Marginal Variation Are Not the Same

A random effect is not observed in the same way as a measured predictor. If the random effects are integrated out, the resulting marginal distribution combines the Poisson response variation with variation in the conditional means. The law of total variance gives the concise identity

\operatorname{Var}(Y_{ij}) =E\{\operatorname{Var}(Y_{ij}\mid b)\} +\operatorname{Var}\{E(Y_{ij}\mid b)\} =E(\mu_{ij})+\operatorname{Var}(\mu_{ij}).

The first term is the average conditional Poisson variance; the second reflects variation in the conditional mean across random-effect values. Thus the marginal variance can exceed the marginal mean even when every conditional distribution is exactly Poisson. This distinction is essential when interpreting observed variation: a marginal comparison alone does not determine whether conditional equidispersion holds.

Worked Example: A Sample Variance Is Not the Model Variance

Suppose five hypothetical, conditionally independent counts are observed under the same covariates and the same fixed random-effect value, giving a common conditional mean $\mu=2$. The counts are $0,1,2,3,4$. Calculate the sample mean and sample variance, then interpret their relationship to the Poisson benchmark.

The sample mean is

\bar{x}=\frac{0+1+2+3+4}{5}=\frac{10}{5}=2.

The squared deviations from 2 sum to $4+1+0+1+4=10$. Using the sample-variance denominator $n-1=4$,

s^2=\frac{10}{4}=2.500.

The model's conditional variance is 2, while this small sample's variance is 2.500. That difference does not by itself refute the Poisson assumption: a sample variance fluctuates from sample to sample, and five observations provide little information about a variance. The calculation does show why the equality is a population-model property rather than an exact equality required of every observed sample.

Checking the Benchmark in Practice

Conditional equidispersion is a model assumption to assess, not an identity that can be verified from a single count. Useful checks ask whether observed responses behave plausibly under the fitted conditional Poisson distributions. Conditional Pearson residuals are one summary:

r_{ij}=\frac{y_{ij}-\widehat{\mu}_{ij}}{\sqrt{\widehat{\mu}_{ij}}},

where $\widehat{\mu}_{ij}$ is the fitted conditional mean. The denominator uses the Poisson conditional standard deviation. Look for systematic residual patterns across fitted means, predictors, groups, or observation order, rather than expecting each residual to be small. Unusually frequent large residuals or persistent patterns can indicate that the conditional model is not representing the data adequately.

A practical complication is that $b$ is latent. In a fitted GLMM, conditional means often use estimated random effects, so the resulting residuals are not based on known true effects. Estimates are uncertain and typically shrunk toward the population mean. Simulation-based checks that reproduce the fitting and prediction procedure can help account for this complexity. Also inspect whether the assumed mean structure, exposure, and grouping structure are credible; a poor mean model can produce apparent variance problems.

Do not compare the raw variance of counts from observations with substantially different fitted conditional means to one single mean. Even under a correct model, those observations do not share one Poisson distribution. Likewise, an excess of zeros or a few large counts is a reason to investigate the fit, not proof of one particular alternative model. The pattern and design context matter.

How This Benchmark Relates to Other Models

A Poisson GLM uses the same conditional Poisson mean-variance relationship but does not include latent random effects to represent group-level heterogeneity. A Poisson GLMM retains the conditional Poisson response while allowing its mean to vary with random effects. The distinction is not that one model has a Poisson variance and the other does not; it is that the GLMM conditions on an additional latent component.

A marginal analysis instead describes the response after averaging over the random-effects distribution. Its variance need not equal its mean, as the total-variance identity shows. These conditional and marginal descriptions answer different questions and should not be substituted for one another. Models that explicitly allow additional conditional count variation are another possible approach when the Poisson conditional benchmark is inadequate; choosing among them requires examining the type and structure of the mismatch rather than treating any variance excess as a diagnosis.

Common Mistakes and Limitations

Wrong: “The sample variance is larger than the sample mean, so the conditional Poisson GLMM is false.” Correction: Sample quantities fluctuate, and heterogeneous fitted means make a raw comparison especially misleading. Assess variation relative to the conditional model and its uncertainty.
Wrong: “The GLMM has equal marginal mean and variance because it is Poisson.” Correction: The equality is conditional on the random effects. Integrating over varying conditional means can increase the marginal variance.
Wrong: “Every group must have the same Poisson variance.” Correction: Each group's conditional variance equals its own conditional mean, which can differ across groups.
Wrong: “A fitted random effect is known exactly, so residuals based on it are definitive.” Correction: Random effects are latent and estimated with uncertainty; account for that when interpreting conditional diagnostics.

The variance-equals-mean condition is specific to the conditional Poisson response distribution. It does not establish that the chosen fixed effects, random-effects structure, or exposure specification is correct. Nor does it guarantee that a finite data set will look perfectly Poisson. It provides a precise benchmark against which conditional model behavior can be evaluated.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For a conditional Poisson mean of 1.5, what is the conditional variance and what is the conditional standard deviation?
A model has $\log(\mu)=\log(1.5)+0.2$ at a specified random-effect value. Calculate the conditional mean and variance.
At a fixed conditional mean of 0.8, calculate the conditional probability of zero counts.
Why does conditional equidispersion not guarantee equality of the marginal mean and variance?

Solutions. (1) The variance is $1.5$ counts-squared and the standard deviation is $\sqrt{1.5}\approx1.225$ counts. (2) $\mu=\exp\{\log(1.5)+0.2\}=1.5e^{0.2}\approx1.832$, so the conditional variance is also approximately $1.832$ counts-squared. (3) $P(Y=0\mid b)=e^{-0.8}0.8^0/0!=e^{-0.8}\approx0.4493$. (4) After averaging over random effects, the law of total variance gives $\operatorname{Var}(Y)=E(\mu)+\operatorname{Var}(\mu)$; variation among conditional means adds a nonnegative term that is absent from the marginal mean.