Matching NB1 And NB2 To Count Variance Patterns

Why distinguish two negative binomial patterns?

A count response can vary more than a Poisson model permits, but “extra variation” does not specify how that variation grows with the mean. NB1 and NB2 are two negative binomial parameterizations that make different predictions as the expected count changes. The distinction matters when observations have substantially different fitted means: the models can agree at low means yet imply quite different variability at high means.

The earlier material on the negative binomial as a Poisson overdispersion family establishes why a negative binomial model can allow variance beyond the Poisson value. Here the focus is narrower: use the shape of the variance-mean relationship to assess whether NB1 or NB2 is a better representation. Neither parameterization should be selected from a single variance estimate or treated as a universal description of count data.

The extra variance above Poisson is \(\alpha_1\mu\) for NB1 and \(\alpha_2\mu^2\) for NB2. Thus NB1 has a constant variance-to-mean ratio, \(1+\alpha_1\), while NB2 has a constant excess-variance-to-squared-mean ratio, \((\operatorname{Var}(Y\mid\cdot)-\mu)/\mu^2=\alpha_2\). Equivalently, the NB2 variance-to-mean ratio is \(1+\alpha_2\mu\), which increases with the mean. These are different patterns, not simply two names for the same fitted dispersion.

Distributional assumptions and a justification

Both models assign probability to nonnegative integer counts. One common negative binomial parameterization uses a mean \(\mu\) and a positive size parameter \(r\), with probability mass function

For NB2, \(r=1/\alpha_2\) is constant across observations when \(\alpha_2>0\). For NB1, \(r=\mu/\alpha_1\), so the size changes with the mean. This parameterization makes both variance formulas transparent. It also identifies an assumption that can be overlooked: each model imposes a particular relationship between the mean and dispersion, rather than giving every observation an unrelated variance.

A short derivation comes from a Poisson-Gamma mixture. Suppose, conditional on a latent rate \(\Lambda\), that \(Y\) is Poisson with mean \(\Lambda\). Let \(\Lambda\) have a Gamma distribution with shape \(r\) and rate \(r/\mu\), so \(E(\Lambda)=\mu\) and \(\operatorname{Var}(\Lambda)=\mu^2/r\). The law of total variance gives

Setting \(r=1/\alpha_2\) yields the NB2 variance, \(\mu+\alpha_2\mu^2\). Setting \(r=\mu/\alpha_1\) yields the NB1 variance, \(\mu+\alpha_1\mu\). The derivation shows why the patterns differ: NB2 uses a constant Gamma shape, whereas NB1’s shape varies with the mean. The mixture is a useful justification for the variance forms; it does not establish that a particular data-generating process is truly a Poisson-Gamma mixture.

For these formulas to guide selection, the mean model must be reasonably specified, the response must be a count, and observations must be independent at the level being modeled—or their dependence must be represented by the model. In a GLMM, the formulas describe variance conditional on the predictors and random effects. Do not compare them directly with marginal variances that also include variation across random effects. The dispersion parameter is assumed constant on its stated scale unless the model explicitly allows it to vary.

Reading variance patterns across means

A practical comparison asks how the excess variance changes as the fitted mean changes. If replicate observations are available at similar fitted means, estimate a variance within those groups and compare it with the mean. A pattern in which \((s^2-\mu)/\mu\) is roughly stable is compatible with NB1; a pattern in which \((s^2-\mu)/\mu^2\) is roughly stable is compatible with NB2. Here \(s^2\) is a within-group sample variance, used as a noisy estimate of the relevant conditional variance.

The comparison is meaningful only when mean differences have been handled. A raw variance computed across observations with different expected counts includes variation due to their different means; it is not a direct estimate of the conditional variance at one mean. In practice, fitted means or carefully chosen groups with similar covariate profiles can help organize diagnostics. When there is only one count at each distinct mean, there is no direct within-mean sample variance; model-based residual diagnostics and likelihood comparisons become more important.

Exposure, model comparison, and diagnostics

Exposure can help distinguish the patterns, but it does not settle the choice by itself. Suppose a rate is approximately constant and the expected count is \(\mu=e\lambda\), where \(e\) is exposure and \(\lambda\) is the rate per exposure unit. Under NB1, \(\operatorname{Var}(Y)/\mu=1+\alpha_1\) is constant as exposure changes. Under NB2, \(\operatorname{Var}(Y)/\mu=1+\alpha_2 e\lambda\) increases with exposure. This comparison assumes the rate and dispersion structure are otherwise appropriate; changes in exposure can also coincide with changes in the underlying population or mean model.

Variance summaries are useful diagnostics, not a substitute for fitting the full response model. Compare NB1 and NB2 with the same observations and mean-model structure, and check that both fits converged and yield sensible fitted values. AIC compares fitted likelihood models while penalizing the number of estimated parameters; for a model with \(k\) estimated parameters and maximized log likelihood \(\ell\), \(\mathrm{AIC}=-2\ell+2k\). AIC is useful for comparing these non-nested alternatives when their likelihoods are computed for the same response data under comparable likelihood definitions. It does not identify a true model or explain the source of overdispersion.

A useful diagnostic sequence is to inspect whether the fitted mean structure is credible, examine variability across ranges of fitted means, and then check model-based residual patterns and likelihood-based comparisons. Patterns that change systematically with fitted means can indicate that the chosen variance function is inadequate. Extreme counts or an excess of zeros can also produce poor fit, but neither feature automatically establishes NB1 or NB2; investigate the fitted distribution and the response process rather than selecting a parameterization from a label such as “overdispersed.”

Common pitfalls and limits

• Using the overall sample variance. Counts with different expected means can have a large pooled variance even if the conditional variance function is correct. Compare variability at similar fitted means or use conditional model diagnostics.
• Choosing from one mean level. At a single \(\mu\), both models can match the same variance by choosing different dispersion parameters. The distinction is the pattern across a range of means.
• Treating noisy ratios as decisive. Ratios such as \((s^2-\mu)/\mu^2\) can be unstable, especially with few replicates or small means. Use them as descriptive evidence and check uncertainty and fitted-model behavior.
• Ignoring dependence or mean-model errors. Unmodeled clustering, omitted predictors, or a poor exposure specification can look like extra dispersion. A dispersion choice cannot repair those problems automatically.
• Reading a preferred variance function as a mechanism. A better NB1 or NB2 fit describes the observed conditional variance pattern; it does not prove a particular biological or sampling mechanism.

The main alternative is the other negative binomial parameterization; the Poisson model is also a useful reference when the variance is close to the mean. NB1 and NB2 are limited to their respective variance shapes and may both fit poorly. They are not remedies for every kind of distributional misspecification, and AIC can only rank the candidate models actually considered. The practical conclusion should therefore combine a plausible mean model, variance-pattern diagnostics, and comparable model fits.