Combining Negative Binomial Variation With Random Intercepts

Why Combine These Two Sources of Variation?

Repeated counts from the same group can vary for two different reasons. Counts may be more variable than a Poisson distribution allows even after a group's underlying rate is held fixed. At the same time, groups may have persistently different rates. A negative binomial GLMM with a group-level random intercept represents both features: negative binomial variation within groups conditional on their intercepts, and heterogeneity between groups through those intercepts.

The distinction matters because the two mechanisms have different implications. Conditional negative binomial dispersion increases the variance of individual counts; a shared random intercept also makes observations from the same group statistically dependent. Treating one mechanism as if it were the other can leave dependence unexplained or attribute too much variation to the wrong model component.

Model and Assumptions

Definition: For groups $j=1,\ldots,J$ and observations $i$ within group $j$, a negative binomial type 2 random-intercept model is $$ Y_{ij}\mid b_j \sim \operatorname{NB2}(\mu_{ij},\kappa), \qquad \log(\mu_{ij}) = \log(E_{ij}) + \mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+b_j, \qquad b_j\sim N(0,\sigma_b^2). $$ Here $Y_{ij}$ is a nonnegative count, $E_{ij}$ is a known exposure when needed (use $E_{ij}=1$ if there is no exposure offset), $\mathbf{x}_{ij}$ is the covariate vector, and $\boldsymbol{\beta}$ is the fixed-effect coefficient vector. The conditional mean is $\mu_{ij}$, and the NB2 conditional variance is $\mu_{ij}+\kappa\mu_{ij}^2$, with $\kappa>0$. Equivalently, its negative binomial size parameter is $1/\kappa$. The random intercepts are independent across groups, and observations are conditionally independent given all group intercepts.

The model assumes that group-specific departures from the fixed predictor are additive on the log-mean scale. Thus, increasing $b_j$ multiplies every mean in group $j$ by $e^{b_j}$. The normal distribution for $b_j$ has mean zero, so the fixed predictor describes the log rate at the reference intercept $b_j=0$, not the mean rate after averaging over groups. The covariates and offsets are treated as given, and the random intercepts are assumed independent of them. These assumptions should fit the sampling structure: observations sharing a group must genuinely share the same modeled group effect.

The parameter $\kappa$ controls conditional overdispersion: at a fixed $b_j$, the variance exceeds the mean by $\kappa\mu_{ij}^2$. The parameter $\sigma_b^2$ controls the spread of the group intercepts. If $\sigma_b^2=0$, the group-level heterogeneity disappears; if $\kappa$ tends to zero, the conditional distribution approaches Poisson. Neither parameter is a substitute for the other.

From Conditional to Marginal Variation

To derive marginal moments, fix an observation's covariates and write $a_{ij}=E_{ij}\exp(\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta})$. Its conditional mean is $\lambda_{ij}(b_j)=a_{ij}e^{b_j}$. For $b_j\sim N(0,\sigma_b^2)$, the lognormal moment identities give $E(e^{b_j})=e^{\sigma_b^2/2}$ and $E(e^{2b_j})=e^{2\sigma_b^2}$. Let $m_{ij}=E(Y_{ij})$ denote the mean averaged over the random intercept.

m_{ij}=a_{ij}e^{\sigma_b^2/2},\qquad E\{\lambda_{ij}(b_j)^2\}=a_{ij}^2e^{2\sigma_b^2} =m_{ij}^2e^{\sigma_b^2}.

The law of total variance states that the marginal variance is the mean conditional variance plus the variance of the conditional mean. Applying it here gives

\begin{aligned} \operatorname{Var}(Y_{ij}) &=E\{\operatorname{Var}(Y_{ij}\mid b_j)\} +\operatorname{Var}\{E(Y_{ij}\mid b_j)\}\\ &=m_{ij}+\kappa m_{ij}^2e^{\sigma_b^2} +m_{ij}^2(e^{\sigma_b^2}-1)\\ &=m_{ij}+m_{ij}^2\{(1+\kappa)e^{\sigma_b^2}-1\}. \end{aligned}

This separates the mechanisms. In the second line, $\kappa m_{ij}^2e^{\sigma_b^2}$ is the expected conditional negative binomial excess variance. The term $m_{ij}^2(e^{\sigma_b^2}-1)$ is the variance of the group-specific conditional mean. Both contribute to the marginal variance, but only the second arises from variation in the shared intercept.

The random intercept also induces dependence within a group. For two distinct observations $i$ and $k$ in group $j$, conditional independence implies that their conditional covariance is zero. Their marginal covariance comes from the shared random intercept:

\begin{aligned} \operatorname{Cov}(Y_{ij},Y_{kj}) &=\operatorname{Cov}\{\lambda_{ij}(b_j),\lambda_{kj}(b_j)\}\\ &=m_{ij}m_{kj}(e^{\sigma_b^2}-1). \end{aligned}

The conditional dispersion parameter $\kappa$ does not appear in this covariance: under the model's conditional-independence assumption, NB sampling variation for one observation does not directly covary with NB sampling variation for another. The random intercept creates their shared variation. In general the marginal distribution of a count is not itself NB2, and the marginal variance is not obtained by simply replacing the conditional mean in the NB2 variance formula.

Worked Example: Conditional Dispersion at a Given Group Intercept

Suppose a hypothetical count has exposure $E_{ij}=1$, fixed linear predictor $\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}=\log(1.5)$, group intercept $b_j=0.40$, and NB2 dispersion $\kappa=0.20$. Calculate its conditional mean and variance. These calculations condition on the stated group intercept; they do not average over the random-intercept distribution.

The conditional mean is

\mu_{ij}=1.5e^{0.40}=2.237737\text{ counts}.

Using that unrounded mean in the NB2 variance formula gives

\begin{aligned} \operatorname{Var}(Y_{ij}\mid b_j) &=\mu_{ij}+\kappa\mu_{ij}^2\\ &=2.237737+0.20(2.237737^2)\\ &\approx 3.239230\text{ counts}^2. \end{aligned}

The conditional variance is about $3.239$ counts squared, greater than the conditional mean of about $2.238$ counts. This difference reflects NB2 dispersion at the specified intercept. It is not the variance across groups, because no averaging over $b_j$ has been done.

Worked Example: Marginal Variance With Both Components

Consider a hypothetical observation with $a_{ij}=2$, $\kappa=0.25$, and $\sigma_b^2=0.36$. Here $a_{ij}$ is the mean when $b_j=0$. Find its mean and variance averaged over the group-intercept distribution.

First, the marginal mean is

m_{ij}=2e^{0.36/2}=2e^{0.18}\approx2.394435\text{ counts}.

The marginal variance formula gives

\begin{aligned} \operatorname{Var}(Y_{ij}) &=m_{ij}+m_{ij}^2\{(1+\kappa)e^{\sigma_b^2}-1\}\\ &=2.394435+(2.394435)^2\{1.25e^{0.36}-1\}\\ &\approx 6.933\text{ counts}^2. \end{aligned}

For clarity, the two added variance contributions are approximately $2.054$ counts squared from expected conditional NB2 excess variance, and $2.484$ counts squared from variation in the conditional mean across groups. Together with the mean component $2.394$, they give approximately $6.933$. The computation illustrates why using only the conditional variance at $b_j=0$ would miss substantial marginal variation.

Worked Example: Dependence Between Two Counts in One Group

Suppose two hypothetical observations in the same group have $a_{1j}=1.2$ and $a_{2j}=2.0$, with $\sigma_b^2=0.25$ and $\kappa=0.30$. Calculate their marginal covariance and approximate correlation. Conditional independence is assumed, and both observations share the same random intercept.

Their marginal means are $m_{1j}=1.2e^{0.125}\approx1.359778$ counts and $m_{2j}=2.0e^{0.125}\approx2.266297$ counts. The shared intercept gives

\operatorname{Cov}(Y_{1j},Y_{2j}) =m_{1j}m_{2j}(e^{0.25}-1) \approx(1.359778)(2.266297)(0.284025) \approx0.8753\text{ counts}^2.

To calculate the correlation, first calculate each marginal variance:

\begin{aligned} \operatorname{Var}(Y_{1j})&\approx1.359778+(1.359778)^2\{1.30e^{0.25}-1\} \approx2.597,\\ \operatorname{Var}(Y_{2j})&\approx2.266297+(2.266297)^2\{1.30e^{0.25}-1\} \approx5.704. \end{aligned}

Thus the marginal correlation is approximately $0.8753/\sqrt{(2.597)(5.704)}\approx0.227$. The positive covariance comes from the shared group intercept. The value of $\kappa$ increases each marginal variance and therefore affects the correlation's denominator, but it does not create the covariance in the numerator.

Interpreting and Checking the Model

A fixed-effect coefficient describes a conditional rate contrast: holding the random intercept and other predictors fixed, a one-unit increase in a covariate with coefficient $\beta_r$ multiplies the conditional mean by $e^{\beta_r}$. The random-intercept variance instead describes how much group-specific rates vary around the fixed predictor. The marginal mean includes the lognormal multiplier $e^{\sigma_b^2/2}$; consequently, it is not generally equal to the mean at $b_j=0$.

A useful diagnostic asks whether the model reproduces both the count variability and the dependence pattern. Compare observed and model-implied variability across groups and within groups, and inspect whether residual patterns remain associated with group membership or covariates. Look for groups with systematic excess counts or too many zeros relative to fitted predictions, as well as a poor fit to the upper tail. These patterns can signal a misspecified mean structure, unsuitable conditional count distribution, incorrect grouping, or a random-intercept distribution that does not represent the group heterogeneity well. A fitted dispersion parameter or random-intercept variance alone cannot establish that the model is adequate.

The main alternatives answer narrower questions. A Poisson random-intercept model allows group-specific rates and hence within-group dependence, but assumes conditional equidispersion. An NB2 model without a random intercept allows conditional overdispersion but does not, by itself, represent shared group-level dependence. The combined model is appropriate when both features are supported by the design and data. It is not automatically preferable: its additional parameters require enough groups and within-group information to estimate reliably, and strong confounding between $\kappa$ and $\sigma_b^2$ can make their separate contributions uncertain.

Common Mistake and Limitations

Common pitfall: “The counts are overdispersed, so the negative binomial dispersion parameter accounts for all extra variation; a group random intercept is unnecessary.” This reasoning overlooks the source of dependence. Conditional NB2 dispersion increases an observation's variance, whereas a shared random intercept makes counts in the same group covary. A model can match an overall variance and still fail to capture that within-group covariance pattern.

The decomposition is model-based, not a guarantee that the data have exactly these two sources of variation. The assumed log link, NB2 variance relationship, normal random-intercept distribution, and conditional independence may all be imperfect. Also, the marginal variance depends on the covariate-specific mean, so one constant marginal variance or intraclass correlation generally does not describe every observation pair. Use the model only when its assumptions are defensible, and interpret uncertainty in $\kappa$ and $\sigma_b^2$ rather than treating estimated components as known.

State the conditioning.
Write the NB2 mean and variance conditional on the group intercept before discussing marginal count variation.

Average over groups when needed.
Use the random-intercept distribution and the law of total variance to obtain marginal means and variances.

Separate variance from dependence.
Attribute conditional NB2 excess variance to $\kappa$, and shared within-group covariance to $\sigma_b^2$.

Check Your Understanding

Try each question, then compare your reasoning with the brief solutions.

If $\sigma_b^2=0$ but $\kappa>0$, what source of extra variation remains, and what happens to covariance between distinct observations in a group?
For a count with $a=3$, $\sigma_b^2=0.20$, and $\kappa=0.10$, write its marginal mean and variance in terms of these values.
Why is the conditional NB2 dispersion parameter absent from the marginal covariance of two distinct observations sharing a random intercept?
A Poisson random-intercept model leaves clear conditional overdispersion, while an NB2 model without a random intercept leaves group-patterned residuals. What model feature addresses both issues?

Solutions. (1) Conditional NB2 excess variation remains because $\kappa>0$. With $\sigma_b^2=0$, the shared intercept is constant, so the model-implied covariance between distinct conditionally independent observations is zero. (2) The marginal mean is $m=3e^{0.10}$ counts. The variance is $m+m^2\{1.10e^{0.20}-1\}$ counts squared. (3) Conditional NB2 sampling variation is independent across observations under the model; their covariance arises from the shared random mean, so it depends on $\sigma_b^2$, not $\kappa$. (4) Use an NB2 GLMM with a group-level random intercept, provided the distributional and grouping assumptions are reasonable.