Why the Variance Form Matters
For a count response, two models can assign the same expected count to an observation and still make quite different predictions about how widely counts vary around that expectation. Negative-binomial models allow the conditional variance to exceed the conditional mean, but the label “negative binomial” does not by itself specify how that extra variation grows as the mean changes. NB1 and NB2 make different choices.
That distinction matters in a generalized linear mixed model (GLMM): the variance form describes variation in the response conditional on the random effects. It is not automatically a description of the total variation across all observations. The mean–variance relationship affects the fitted likelihood, uncertainty, and predictive spread, so interpreting the form is part of understanding the model rather than just a software setting.
Both forms reduce to the Poisson conditional variance, \(\operatorname{Var}(Y_{ij}\mid b_i,\mathbf{x}_{ij})=\mu_{ij}\), when their dispersion parameter is zero. For positive dispersion, both allow conditional variance greater than the mean. Their difference is the way that excess variance scales as \(\mu_{ij}\) grows.
Reading the Two Mean–Variance Relationships
For NB1, the variance-to-mean ratio is constant: $$ \frac{\operatorname{Var}(Y_{ij}\mid b_i,\mathbf{x}_{ij})}{\mu_{ij}}=1+\alpha_1. $$ Thus, if the conditional mean doubles, the conditional variance doubles too. The variance is a straight line through zero as a function of the mean.
For NB2, the variance-to-mean ratio increases with the mean: $$ \frac{\operatorname{Var}(Y_{ij}\mid b_i,\mathbf{x}_{ij})}{\mu_{ij}}=1+\alpha_2\mu_{ij}. $$ The variance has a linear term and a quadratic term in the mean. When \(\mu_{ij}\) is small, the linear term may dominate; as the mean grows, the quadratic contribution can become substantial. The same value of \(\alpha\) therefore does not imply the same variance under NB1 and NB2.
| Feature | NB1 | NB2 |
|---|---|---|
| Conditional variance | \(\mu(1+\alpha_1)\) | \(\mu+\alpha_2\mu^2\) |
| Variance-to-mean ratio | \(1+\alpha_1\), constant in \(\mu\) | \(1+\alpha_2\mu\), increases with \(\mu\) |
| Growth as the mean increases | Linear | Includes a quadratic term |
| Poisson limit | \(\alpha_1=0\) | \(\alpha_2=0\) |
These are conditional relationships. For example, a log-link model might specify \(\log(\mu_{ij})=\mathbf{x}_{ij}^{\mathsf T}\boldsymbol{\beta}+\mathbf{z}_{ij}^{\mathsf T}b_i\). Once \(b_i\) and the covariates are given, the variance function determines the response variation around that conditional mean. The random effects can add further variation when considering observations without conditioning on \(b_i\).
How the Forms Arise From a Negative-Binomial Distribution
One standard negative-binomial parameterization describes \(Y\) as the number of failures before \(r\) successes, where each trial has success probability \(p\). With this convention, $$ E(Y)=\frac{r(1-p)}{p}, \qquad \operatorname{Var}(Y)=\frac{r(1-p)}{p^2}. $$ To express this distribution using a specified mean \(\mu\), choose \(r\) and \(p\) as functions of \(\mu\) and a dispersion parameter.
For either form, choose \(p=r/(r+\mu)\). Then \(r(1-p)/p=\mu\), so the negative-binomial distribution has the desired mean.
Since \(1-p=\mu/(r+\mu)\), the variance becomes \(\mu+\mu^2/r\). This identity connects the negative-binomial size parameter \(r\) to the mean–variance relationship.
For NB2, set \(r=1/\alpha_2\), giving \(\mu+\alpha_2\mu^2\). For NB1, set \(r=\mu/\alpha_1\), giving \(\mu+\alpha_1\mu=\mu(1+\alpha_1)\).
This derivation also shows why the two forms are distinct parameterizations of negative-binomial distributions. In NB2, the size \(r\) is constant across conditional means when \(\alpha_2\) is constant. In NB1, \(r\) changes with the conditional mean. If the dispersion is zero, the negative-binomial model has the Poisson limit; the displayed size expressions are then understood as limits as \(r\) tends to infinity.
Worked Example: Translating Dispersion Into Negative-Binomial Parameters
Suppose a hypothetical observation has conditional mean \(\mu=4\). Compare the negative-binomial parameters and conditional variances under NB1 and NB2 if both dispersion parameters happen to equal \(0.5\).
NB1. The size is \(r=\mu/\alpha_1=4/0.5=8\). The corresponding probability is \(p=r/(r+\mu)=8/(8+4)=2/3\). The mean is \(8(1/3)/(2/3)=4\), and the variance is $$ \frac{r(1-p)}{p^2} =\frac{8(1/3)}{(2/3)^2} =\frac{8/3}{4/9} =6. $$ The NB1 variance formula gives the same result: \(4(1+0.5)=6\).
NB2. The size is \(r=1/\alpha_2=1/0.5=2\). Now \(p=r/(r+\mu)=2/(2+4)=1/3\). The mean is \(2(2/3)/(1/3)=4\), and the variance is $$ \frac{r(1-p)}{p^2} =\frac{2(2/3)}{(1/3)^2} =\frac{4/3}{1/9} =12. $$ The NB2 variance formula also gives \(4+0.5(4^2)=12\).
The two models have the same conditional mean and the same numerical dispersion value, but different conditional variances: 6 counts-squared under NB1 and 12 counts-squared under NB2. Dispersion numbers should therefore be interpreted together with their variance form.
Comparing Variances Across Conditional Means
A useful comparison holds the two dispersion parameters fixed and examines the variance functions over a range of means. If one uses the same numerical value \(\alpha\) for illustration, NB1 and NB2 have equal variances when $$ \mu(1+\alpha)=\mu+\alpha\mu^2. $$ For \(\mu>0\), subtracting \(\mu\) from both sides gives \(\alpha\mu=\alpha\mu^2\). If \(\alpha>0\), equality occurs at \(\mu=1\). This equality is a consequence of assigning the same numerical dispersion to both forms; it is not a general equivalence between NB1 and NB2.
Worked Example: Comparing NB1 and NB2 at Three Means
Suppose a hypothetical GLMM has conditional means of 1, 3, and 8 for three covariate settings. Compare NB1 with \(\alpha_1=0.6\) and NB2 with \(\alpha_2=0.25\). These are different dispersion values, so the comparison is about the specified models, not about equal parameter values.
At \(\mu=1\), NB1 gives \(1(1+0.6)=1.6\). NB2 gives \(1+0.25(1^2)=1.25\).
At \(\mu=3\), NB1 gives \(3(1+0.6)=4.8\). NB2 gives \(3+0.25(3^2)=3+2.25=5.25\).
At \(\mu=8\), NB1 gives \(8(1+0.6)=12.8\). NB2 gives \(8+0.25(8^2)=8+16=24\).
Here NB1 has the larger variance at mean 1, while NB2 has the larger variance at means 3 and 8. The results follow from the chosen parameters and means; neither form is invariably more variable at every possible mean.
Using Mean–Variance Patterns as a Diagnostic
The variance forms suggest a diagnostic question: after accounting for the fitted conditional means and random effects, does the remaining count variability look more consistent with a roughly constant variance-to-mean ratio or with a ratio that grows with the mean? A simple illustration can use groups of hypothetical observations with similar fitted means. In practice, such grouped summaries are exploratory: estimated variances can be noisy, especially when groups are small, and the random-effects structure must also be considered.
Worked Example: A Pattern Consistent With NB2
Imagine three hypothetical sets of observations with fitted conditional means 2, 4, and 6. Suppose their estimated conditional variances are 4, 12, and 36. Compare the implied NB2 dispersion values and the NB1 variance-to-mean ratios.
For NB2, rearrange \(\operatorname{Var}(Y\mid b)=\mu+\alpha_2\mu^2\) to get \(\alpha_2=(\operatorname{Var}(Y\mid b)-\mu)/\mu^2\). At mean 2 this is \((4-2)/2^2=2/4=0.5\). At mean 4 it is \((12-4)/4^2=8/16=0.5\). At mean 6 it is \((36-6)/6^2=30/36\approx0.833\). The first two estimates agree at 0.5, while the third is higher; the three summaries do not follow a perfectly constant NB2 dispersion pattern.
For NB1, the implied dispersion is \(\operatorname{Var}(Y\mid b)/\mu-1\). The three values are \(4/2-1=1\), \(12/4-1=2\), and \(36/6-1=5\). They increase substantially with the mean rather than remaining constant. These summaries suggest that NB1’s constant variance-to-mean ratio would not describe this pattern well. They do not, by themselves, establish that NB2 is adequate: sampling noise, covariates, and random effects still need to be assessed in the fitted model.
A GLMM’s marginal variance includes both the conditional response variation and variation in the conditional mean induced by random effects. The law of total variance gives $$ \operatorname{Var}(Y_{ij}) =E\{\operatorname{Var}(Y_{ij}\mid b_i,\mathbf{x}_{ij})\} +\operatorname{Var}\{E(Y_{ij}\mid b_i,\mathbf{x}_{ij})\}. $$ The first term averages the NB1 or NB2 conditional variance over the random-effects distribution; the second captures variation in the conditional mean. Consequently, a raw variance-to-mean ratio calculated across groups need not match either conditional variance function. Shared random effects can also induce dependence among outcomes in the same group.
Model Checking, Alternatives, and Limitations
A variance form should be assessed as part of a complete conditional model, not selected by looking only at the overall sample variance. Useful checks include comparing observed and simulated count distributions, examining residual patterns across fitted conditional means, and checking whether the model systematically under- or over-predicts variability at low or high means. In a GLMM, checks should also consider whether the group-level structure is adequate; an unsuitable random-effects specification can distort apparent evidence about the response variance.
The Poisson model is a natural alternative when conditional variance equal to the conditional mean is plausible. NB1 and NB2 both allow extra conditional variation, but impose different ways for it to grow. A quasi-Poisson approach specifies a mean–variance relationship for estimation without using a full negative-binomial probability distribution; it is not the same likelihood as either NB1 or NB2. The choice should be guided by the response process, fit, and the intended inference, rather than by a rule that one form is always preferable.
The variance functions are not complete descriptions of all distributional behavior. They do not guarantee that the model captures the observed frequency of every count, and matching a variance pattern does not ensure that the conditional mean or random-effects assumptions are correct. The negative-binomial distribution also imposes a particular shape beyond its first two moments. In finite samples, diagnostics may not clearly distinguish the forms, especially over a narrow range of conditional means.
Common Mistakes
- Calling the dispersion parameters interchangeable. A value of 0.5 means a constant 50% increase in the mean under NB1, but contributes \(0.5\mu^2\) to the variance under NB2. Always name the variance form with the parameter.
- Claiming NB2 is always more variable. The ordering depends on the mean and the two dispersion values. Compare the actual variance functions at the means of interest.
- Using marginal variability to identify the conditional form. Random effects add variation in conditional means. Check the conditional model and group structure rather than treating a raw overall variance as the response dispersion.
- Interpreting excess variance as proof of one model. Overdispersion relative to Poisson does not uniquely select NB1 or NB2. Diagnostics and the fitted model’s assumptions remain relevant.
Check Your Understanding
Answer each question, then compare your reasoning with the brief solutions.
- Under NB1 with \(\alpha_1=0.4\), what is the conditional variance-to-mean ratio?
- Under NB2 with \(\alpha_2=0.2\) and conditional mean \(\mu=5\), calculate the conditional variance.
- For NB1 and NB2 with the same positive numerical dispersion value, at which positive conditional mean are their variance functions equal?
- Why might the overall variance across observations in a GLMM fail to reveal whether its conditional response model is NB1 or NB2?
Solutions. 1. The ratio is \(1+0.4=1.4\). 2. The variance is \(5+0.2(5^2)=5+5=10\). 3. Equating \(\mu(1+\alpha)\) and \(\mu+\alpha\mu^2\), then dividing by \(\alpha\mu>0\), gives \(\mu=1\). 4. The overall variance includes both the expected conditional response variance and the variance of the conditional mean across random effects, so it is not just the NB1 or NB2 conditional variance function.