Observation-Level Random Effects Versus Negative Binomial Models

Two Ways to Represent Extra-Poisson Variation

A Poisson model assumes that, conditional on its rate, a count has variance equal to its mean. In count data, variation can exceed that level even after measured predictors and group-level random effects have been included. Two common representations are an observation-level random effect, which gives each count its own latent multiplicative effect, and a negative-binomial Type 2 model, whose rate is mixed by a gamma distribution. The central comparison is subtle: the models can have exactly the same marginal mean and variance, yet assign different probabilities to individual count values.

The observation-level effect is not a shared cluster effect. Its latent value applies to one observation only, so it can inflate that observation's variance without, by itself, creating covariance between two observations in the same cluster. The negative-binomial distribution likewise describes extra variation in a count; it does not automatically create dependence among multiple counts. Shared random effects are needed to represent dependence arising from common groups.

In the usual log-link GLMM parameterization, the conditional rate is often written \(\lambda_i\exp(u_i)\), where \(\lambda_i=\exp(x_i^\mathsf{T}\beta+\text{offset}_i)\) is the rate at \(u_i=0\). Here \(\lambda_i\) is not the marginal mean: integrating over the normal effect gives marginal mean \(\lambda_i\exp(\sigma^2/2)\). Equivalently, use \(m_i=\lambda_i\exp(\sigma^2/2)\) and the mean-one multiplier \(W_i\) above. This distinction matters when comparing coefficients or predicted means across models.

Deriving the Matching Mean and Variance

For a Poisson mixture, the law of total expectation and the law of total variance give \(E(Y_i)=E(\Lambda_i)\) and \(\operatorname{Var}(Y_i)=E(\Lambda_i)+\operatorname{Var}(\Lambda_i)\), where \(\Lambda_i\) is the random Poisson rate. For the mean-one lognormal multiplier, the normal moment-generating function gives \(E(W_i^r)=\exp\{(r^2-r)\sigma^2/2\}\). In particular, \(E(W_i)=1\) and \(\operatorname{Var}(W_i)=\exp(\sigma^2)-1\). Therefore:

The NB2 variance is \(m_i+\alpha m_i^2\). Consequently, the models have matching first two moments when \(\alpha=\exp(\sigma^2)-1\), or, in the reverse direction, \(\sigma^2=\log(1+\alpha)\). This is a moment match, not a claim that the two mixture distributions are identical.

Same Variance Does Not Mean Same Distribution

A useful way to expose the difference is to examine factorial moments. For a Poisson count conditional on rate \(\Lambda_i\), the \(r\)th factorial moment is \(E\{Y_i(Y_i-1)\cdots(Y_i-r+1)\mid\Lambda_i\}=\Lambda_i^r\). Averaging over the mixing distribution therefore gives \(E\{(Y_i)_r\}=E(\Lambda_i^r)\). For the mean-one lognormal multiplier, this becomes \(m_i^r\exp\{(r^2-r)\sigma^2/2\}\).

For NB2, the equivalent rate mixture is \(\Lambda_i=m_iG_i\), where \(G_i\) has a gamma distribution with shape \(1/\alpha\), scale \(\alpha\), mean 1, and variance \(\alpha\). Its \(r\)th moment is \(\prod_{j=0}^{r-1}(1+j\alpha)\). The second factorial moments agree under the variance-matching conversion. The third generally does not:

For positive \(\alpha\), these expressions differ. The lognormal mixing distribution has a different shape from the gamma mixing distribution, so the induced count distributions can differ in their upper tails and at zero even when their means and variances match. The lognormal mixture generally has no simple closed-form probability mass function; its probabilities are obtained by integrating the conditional Poisson probabilities over the normal effect. The NB2 model has a closed-form negative-binomial probability mass function.

What the Observation-Level Effect Can—and Cannot—Explain

An observation-level random effect is appropriate when extra-Poisson heterogeneity is plausibly specific to individual observations and a lognormal latent rate is a defensible representation. It may be used alongside group-level random effects: the group effect represents shared variation, while the observation-level effect represents additional count-specific variation. Because each observation has its own independent latent effect, it does not create extra covariance between distinct observations once shared effects are accounted for.

Model Checks, Alternatives, and Limitations

A comparison should begin with the sampling and dependence structure, not just with which model produces a larger variance estimate. State whether counts are conditionally independent given the modeled random effects, identify which effects are shared, and specify the conditional mean and mixing distribution. For the observation-level model, the normal effects are assumed independent across observations and normally distributed on the log-rate scale. For the gamma-mixture interpretation of NB2, the rate multiplier has a gamma distribution with the stated mean and variance. Both retain a Poisson sampling distribution conditional on the latent rate.

Useful checks include comparing observed and model-implied count frequencies, especially zero and large counts, and examining whether residual variation remains across fitted-mean ranges or within groups. Such checks should be interpreted against predictions from the fitted model, preferably using simulation that reproduces its grouping and design. A variance match at each mean cannot diagnose a wrong mixing distribution by itself. Signs of trouble include systematic excesses of zeros or large counts relative to simulated data, residual association among observations in the same group after shared effects are included, or strong sensitivity of conclusions to the chosen mixing assumption.

The principal alternatives answer different modeling needs. An NB2 model uses gamma mixing and a convenient closed-form count distribution; an observation-level random effect uses normal mixing on the log-rate scale and is naturally expressed as a GLMM random effect. A Poisson model is simpler but imposes conditional equidispersion. A model with shared random intercepts or slopes addresses dependence associated with common groups; it may be combined with either overdispersion representation when justified. None is automatically correct because it fits the observed variance. The choice should reflect the scientific and sampling structure, distributional fit, and the purpose of the analysis.

A practical limitation of an observation-level effect is that it adds a latent parameter for each observation; it should not be interpreted as a set of reliably estimated individual effects when there is only one count per observation. Its role is to specify a distribution for residual heterogeneity. Conversely, the NB2 dispersion parameter imposes gamma mixing, which may not describe the data's higher moments. A numerical fit or a matched variance cannot remove these assumptions.

Common Pitfall

Wrong reasoning: “The observation-level model and NB2 give the same mean and variance, so they are interchangeable.” Correction: Their mixing distributions differ. The third factorial moments and zero probabilities can differ even under an exact first-two-moment match. Choose and assess a model using the distributional and dependence assumptions relevant to the data, not only its variance formula.