Observation-Level Random Effects for Overdispersion

Why Add a Random Effect for Each Observation?

A Poisson GLMM can be equidispersed conditional on its random effects and still show extra-Poisson variation when those effects are averaged over. An observation-level random effect (OLRE) gives each count its own latent intercept. It represents unmeasured heterogeneity among observations: two observations with the same recorded predictors can have different underlying rates.

The construction is useful when a conditional Poisson model appears too restrictive, but its interpretation depends on a strong distributional assumption. The added effect is not a measured predictor and does not identify the particular cause of heterogeneity. It is a model for residual variation at the observation level.

Definition: For observations $i=1,\ldots,n$, an observation-level random-intercept Poisson GLMM is $$ Y_i\mid b_i \sim \operatorname{Poisson}(\lambda_i),\qquad \log(\lambda_i)=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}+\log(t_i)+b_i, \qquad b_i\overset{\mathrm{iid}}{\sim}N(0,\sigma_{\mathrm{OL}}^2). $$ Here $Y_i$ is a count, $\mathbf{x}_i$ is the fixed-effect design vector, $\boldsymbol{\beta}$ is the fixed-effect coefficient vector, $t_i$ is a known exposure (omit $\log(t_i)$ when no offset is needed), and each $b_i$ belongs to exactly one observation. Conditional on the random effects and predictors, counts are independent and have Poisson mean and variance $\lambda_i$. The random effects are assumed independent of the predictors.

The model has two levels of interpretation. Conditional on $b_i$, the mean is $\lambda_i$, and the Poisson variance is also $\lambda_i$. Marginally, after accounting for the distribution of $b_i$, the mean and variance need not be equal. The coefficient $\beta_k$ describes the conditional log rate ratio for a one-unit increase in predictor $x_k$, holding the other predictors and the latent effect fixed. With a common random-effect distribution across predictor values, the corresponding marginal rate ratio is also $\exp(\beta_k)$.

How the Latent Effect Adds Variation

Write $\mu_{0i}=\exp\{\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}+\log(t_i)\}$, the rate-scaled mean when $b_i=0$. The actual conditional mean is $\lambda_i=\mu_{0i}\exp(b_i)$. Thus the random intercept multiplies the mean by a positive, observation-specific factor. A positive $b_i$ raises the rate; a negative $b_i$ lowers it.

The normal random effect implies $E[\exp(b_i)]=\exp(\sigma_{\mathrm{OL}}^2/2)$. Applying the law of total expectation and the law of total variance gives

E(Y_i)=\mu_{0i}\exp(\sigma_{\mathrm{OL}}^2/2)=m_i,\qquad \operatorname{Var}(Y_i)=E(\lambda_i)+\operatorname{Var}(\lambda_i) =m_i+m_i^2\{\exp(\sigma_{\mathrm{OL}}^2)-1\}.

The second term in the variance is nonnegative and is positive whenever $\sigma_{\mathrm{OL}}^2>0$. It is the extra variation attributable to heterogeneity in the latent means. This brief calculation explains the mechanism; the lognormal-Poisson mixture and its moments can be examined in more depth separately.

Because each observation has an independent $b_i$, this construction increases each observation’s marginal variance without introducing shared random-effect covariance between distinct observations. Conditional independence alone is not the full story: the marginal distribution of each count is a mixture of Poisson distributions with different means.

Worked Example: Computing the Marginal Mean and Variance

Suppose a hypothetical count model has $\mu_{0i}=2.0$ events per observation and an estimated observation-level variance $\sigma_{\mathrm{OL}}^2=0.25$. Calculate the marginal mean and variance.

The marginal mean is $$ m_i=2.0\exp(0.25/2)=2.0\exp(0.125)\approx 2.266. $$ The count therefore has an average of about 2.266 events per observation across the assumed distribution of latent effects, rather than 2.0.

The extra-variance component is $$ m_i^2\{\exp(0.25)-1\} \approx (2.266)^2(0.2840)\approx 1.459. $$ Consequently, the marginal variance is approximately $2.266+1.459=3.725$ events squared per observation. This exceeds the marginal mean because the latent rates vary from observation to observation. The conditional variance for a particular observation remains equal to its own conditional mean $\lambda_i$.

Constructing and Interpreting the Model

An OLRE is included as a random intercept whose grouping factor has a different level for every row of the analysis data. It is not the same as a random intercept for a site, person, or other unit that contributes multiple observations. For a cluster random intercept, observations in the same cluster share a latent shift; for an OLRE, no two observations share the shift.

The model is fitted by integrating over the distribution of the $b_i$ values as part of the GLMM likelihood. The estimated $\sigma_{\mathrm{OL}}^2$ describes the fitted spread of latent log rates. It is not a direct measurement of a known source of heterogeneity, nor does it mean that each observation has a separately interpretable, reliably estimated effect. For a new observation, the appropriate model-based description is that its latent effect is drawn from the assumed distribution.

Worked Example: Including Exposure and a Binary Predictor

In a hypothetical analysis, the conditional model is $$ \log(\lambda_i)=\log(t_i)+\beta_0+\beta_1x_i+b_i, \qquad \beta_0=\log(1.2),\quad \beta_1=\log(0.75),\quad \sigma_{\mathrm{OL}}^2=0.09. $$ Here $x_i=0$ or $1$, and $t_i$ is exposure measured in identical time units. Find the marginal means for an observation with $t_i=2$ in each predictor group.

For $x_i=0$, the mean at $b_i=0$ is $$ \mu_{0i}=2(1.2)=2.4. $$ Averaging over the random-effect distribution gives $$ m_i=2.4\exp(0.09/2)=2.4\exp(0.045)\approx 2.510 $$ expected events per observation.

For $x_i=1$, the conditional mean at $b_i=0$ is $2(1.2)(0.75)=1.8$. Its marginal mean is $$ 1.8\exp(0.045)\approx 1.883 $$ expected events per observation. The conditional rate ratio comparing $x_i=1$ with $x_i=0$ is $\exp(\beta_1)=0.75$. Since the same random-effect variance applies to both groups, the marginal rate ratio is also $1.883/2.510\approx 0.750$. The offset makes the rate scale with exposure; the OLRE adds observation-specific heterogeneity around that rate.

Worked Example: Interpreting Latent Rate Multipliers

Consider a hypothetical model with $\mu_{0i}=2.0$ and $\sigma_{\mathrm{OL}}=0.30$, so that $\sigma_{\mathrm{OL}}^2=0.09$. Illustrate how a latent effect one standard deviation above or below zero changes the conditional mean.

At $b_i=0.30$, the multiplier is $\exp(0.30)\approx1.350$, giving conditional mean $2.0(1.350)\approx2.700$ events. At $b_i=-0.30$, the multiplier is $\exp(-0.30)\approx0.7408$, giving conditional mean $2.0(0.7408)\approx1.482$ events. These are illustrations of the model’s multiplicative effect, not estimates of two particular observations’ true latent rates. A standard deviation is not a bound: the normal distribution assigns probability to values beyond $\pm0.30$ as well.

Assumptions, Diagnostics, and Failure Modes

The OLRE model is most defensible when the response is a count, the conditional Poisson mean is sensibly specified, and remaining excess variation is plausibly observation-specific. Its key assumptions are conditional Poisson sampling, normally distributed independent observation-level intercepts with a common variance, and independence of those effects from the predictors. The chosen predictors, functional forms, and exposure offsets must also be appropriate. Adding an OLRE does not make these assumptions true by itself.

Check the fitted model using diagnostics appropriate for count GLMMs, including residual patterns against fitted values and predictors, influential observations, and simulation-based checks of the distribution of counts. Examine whether residual overdispersion remains and whether the fitted random-effect variance is stable and estimable. A variance estimate near zero suggests little evidence for this additional component under the model; convergence warnings or strong sensitivity to modelling choices call for caution. It is also useful to inspect whether a few extreme counts dominate the fit.

A particularly important failure mode is dependence among observations. If rows from the same subject or site share an unmodelled influence, independent $b_i$ values cannot represent that shared dependence. A separate cluster-level random effect or another dependence structure may be needed. Likewise, an OLRE cannot correct a wrongly specified mean function, omitted predictors, incorrect exposure, or a data process that is not well represented by a normal distribution of log rates.

Approach	What it represents	Key distinction
Observation-level random intercept	Independent latent heterogeneity for each count	Raises marginal variance without a shared observation-level effect between rows
Cluster random intercept	Latent heterogeneity shared within a cluster	Can represent dependence among observations in the same cluster
Negative binomial model	Extra-Poisson variation through a different count-distribution formulation	It is not automatically equivalent to a lognormal random-effect mixture

The alternatives answer related but not identical modelling questions. Choose the random-effect structure to match the sampling and dependence structure, not solely to reduce a dispersion statistic. The OLRE’s distributional assumption may be consequential, and with one count per random-effect level, the latent effects are integrated out rather than learned as stable individual-specific quantities. Comparing alternatives and their implications requires attention to fit, assumptions, and the inferential target.

Common Mistake

Wrong reasoning: “The OLRE makes the conditional Poisson variance larger than the conditional mean, so it fixes conditional overdispersion.” The conditional distribution remains Poisson, so $\operatorname{Var}(Y_i\mid b_i)=E(Y_i\mid b_i)=\lambda_i$. The extra variance appears after averaging across the distribution of $b_i$, through variation in $\lambda_i$. Also, because OLREs are independent across observations, they do not stand in for a shared cluster effect.

Key takeaway: An OLRE adds a normally distributed, observation-specific shift to the conditional log mean. Conditional counts remain Poisson; marginally, the varying latent means produce extra-Poisson variance. The model addresses independent observation-level heterogeneity, not every form of misspecification or dependence.

Check Your Understanding

Answer each question, then compare with the solutions below.

If $\sigma_{\mathrm{OL}}^2=0$, how does the OLRE model differ from the ordinary conditional Poisson model?
With $\mu_{0i}=1.5$ and $\sigma_{\mathrm{OL}}^2=0.16$, calculate the marginal mean and the added variance.
Why does an independent OLRE for every row not represent a shared site effect?
Suppose a predictor’s coefficient is $\log(0.8)$, and the random-effect variance is the same across predictor groups. What is the marginal rate ratio for a one-unit increase in that predictor?

Solutions. (1) The random effect is zero with probability one when its variance is zero, so $\lambda_i=\mu_{0i}$, and the marginal count is Poisson with mean and variance $\mu_{0i}$. (2) The marginal mean is $1.5\exp(0.08)\approx1.6249$. The added variance is $1.6249^2\{\exp(0.16)-1\}\approx1.6249^2(0.1735)\approx0.4581$; the total marginal variance is approximately $1.6249+0.4581=2.0831$. (3) Each row has its own independent effect, so no pair of rows shares a latent shift; a site effect must be common to rows from that site. (4) The conditional rate ratio is $\exp\{\log(0.8)\}=0.8$, and it is also the marginal rate ratio when the random-effect distribution is common across predictor values.