The Poisson Limit and Boundary Tests for Overdispersion

From Negative-Binomial Size to the Poisson Boundary

The negative-binomial model allows count variability beyond the Poisson variance. Its size parameter $r$ controls how much extra variability is permitted: as $r$ grows, the variance approaches the mean. This limiting relationship is more than a useful approximation. It identifies the Poisson model as a boundary case of the negative-binomial family, which has direct consequences for likelihood-ratio tests of overdispersion.

It is convenient to use the reciprocal size, $\alpha=1/r$, rather than $r$ itself. Finite negative-binomial sizes correspond to $\alpha>0$, while the Poisson model corresponds to $\alpha=0$. Thus, a test for overdispersion is a test of a nonnegative parameter at the edge of its allowed parameter space—not an ordinary test of a parameter that can vary on both sides of an interior null value.

Definition: In the mean–size parameterization, a negative-binomial random variable $Y$ with mean $\mu>0$ and size $r>0$ has probability mass function $$ P(Y=y)=\frac{\Gamma(y+r)}{\Gamma(r)y!} \left(\frac{r}{r+\mu}\right)^r \left(\frac{\mu}{r+\mu}\right)^y,\qquad y=0,1,2,\ldots. $$ Its variance is $\operatorname{Var}(Y)=\mu+\mu^2/r$. Writing $\alpha=1/r$, this becomes $\operatorname{Var}(Y)=\mu+\alpha\mu^2$, with the Poisson model at $\alpha=0$.

The Poisson boundary is understood as the limit $r\to\infty$, or equivalently $\alpha\downarrow0$. It is not a negative-binomial distribution with a finite size $r$. In a regression model, the same idea applies observation by observation: the fitted means may vary with covariates, but the common dispersion parameter $\alpha$ approaches zero.

Deriving the Poisson Limit

For a fixed nonnegative integer $y$, the mass function can be written using the rising product $r(r+1)\cdots(r+y-1)$. Separating out powers of $r$ makes each part of the limit visible.

Theorem: For fixed $\mu>0$ and each $y=0,1,2,\ldots$, the negative-binomial probability $P(Y=y)$ converges as $r\to\infty$ to the probability that a Poisson random variable with mean $\mu$ equals $y$. Consequently, $Y$ converges in distribution to $\operatorname{Poisson}(\mu)$.

Proof. For integer $y\geq 0$, the ratio of gamma functions is $\Gamma(y+r)/\Gamma(r)=\prod_{j=0}^{y-1}(r+j)$, with an empty product of one when $y=0$. Hence

P(Y=y)=\frac{\mu^y}{y!} \left\{\prod_{j=0}^{y-1}\left(1+\frac{j}{r}\right)\right\} \left(1+\frac{\mu}{r}\right)^{-y} \left(1+\frac{\mu}{r}\right)^{-r}.

Here $y$ and $\mu$ are fixed. The finite product tends to one, and $(1+\mu/r)^{-y}$ tends to one. Also, $(1+\mu/r)^r\to e^\mu$, so the final factor tends to $e^{-\mu}$. Therefore

\lim_{r\to\infty}P(Y=y)=e^{-\mu}\frac{\mu^y}{y!},

the Poisson mass at $y$. To pass from these point probabilities to convergence in distribution, let $F_r$ denote the distribution of $Y$. For any real $x$ that is not an integer, $P(Y\leq x)$ is either zero if $x<0$, or the finite sum of the masses for $y=0,\ldots,\lfloor x\rfloor$. Each term in that finite sum converges to the corresponding Poisson mass. Thus the distribution functions converge at every continuity point of the Poisson distribution function, which proves convergence in distribution. $\square$

Worked Example: Checking the Limit at Three Counts

Take $\mu=2$. For $y=0$, the negative-binomial mass is $(1+2/r)^{-r}$, which tends to $e^{-2}$. For $y=1$, the mass is

P(Y=1)=2\left(1+\frac{2}{r}\right)^{-r-1} \longrightarrow 2e^{-2}.

For $y=2$, the mass is

P(Y=2)=\frac{2^2}{2!}\left(1+\frac{1}{r}\right) \left(1+\frac{2}{r}\right)^{-r-2} \longrightarrow 2e^{-2}.

These are exactly the Poisson probabilities for counts 0, 1, and 2 when the mean is 2: $e^{-2}$, $2e^{-2}$, and $2e^{-2}$. The argument applies to every fixed nonnegative count, not only these three.

What the Boundary Means for Variability

At a fixed mean $\mu$, the negative-binomial variance exceeds the Poisson variance by $\mu^2/r=\alpha\mu^2$. Increasing the size reduces this extra term. In a regression setting with means $\mu_i$, the NB2 variance is $\mu_i+\alpha\mu_i^2$; setting $\alpha=0$ removes the extra variance for every observation.

Worked Example: Translating Size into Extra Variance

Suppose a negative-binomial count has mean $\mu=4$ and size $r=8$. Then $\alpha=1/8$, and

\operatorname{Var}(Y)=4+\frac{4^2}{8}=4+2=6.

The Poisson model with the same mean has variance 4, so the negative-binomial model adds 2 units of variance. If the size increases to $r=80$, then

\operatorname{Var}(Y)=4+\frac{16}{80}=4.2.

The excess variance is now 0.2. In the limit $r\to\infty$, it is zero. This variance comparison illustrates the limiting relationship, while the mass-function argument above establishes the distributional limit itself.

Why the Likelihood-Ratio Reference Law Changes

Consider independent counts with negative-binomial means $\mu_i(\beta)$ and a shared dispersion parameter $\alpha\geq0$. The null hypothesis of Poisson dispersion is $H_0:\alpha=0$; the one-sided alternative is $H_1:\alpha>0$. Let $\ell(\beta,\alpha)$ be the log-likelihood, and define the likelihood-ratio statistic

T=2\left\{\sup_{\beta,\,\alpha\geq0}\ell(\beta,\alpha) -\sup_{\beta}\ell(\beta,0)\right\}.

Wilks’ usual chi-square result applies to regular tests in which the null parameter lies in the interior of the parameter space. Here $\alpha=0$ is the boundary: values below zero are not permitted in the negative-binomial model. The fitted improvement can move away from the null only in the positive direction. Under appropriate regularity conditions, this one-sided constraint gives a mixture reference distribution rather than an ordinary $\chi^2_1$ distribution.

Theorem: Suppose the null model is identifiable, the nuisance parameter $\beta$ is an interior parameter, and the observations are independent. Assume the number of regression parameters is fixed, the information for $\beta$ and for $\alpha$ after accounting for $\beta$ is nonsingular, and the log-likelihood has a valid local quadratic expansion at $\alpha=0$ from the right. Also assume the maximum-likelihood estimators are consistent and have the usual local asymptotic behavior under the null. Then, for testing $H_0:\alpha=0$ against $\alpha>0$, $$ T\xrightarrow{d}\frac{1}{2}\delta_0+\frac{1}{2}\chi^2_1, $$ where $\delta_0$ is a point mass at zero.

Derivation. The local quadratic expansion after profiling out the nuisance parameter can be written, for local alternatives $\alpha=h/\sqrt{n}$, $h\geq0$, as

\ell_p(h/\sqrt{n})-\ell_p(0) =h\Delta_n-\frac{1}{2}h^2 I_{\mathrm{eff}}+o_p(1), \qquad \frac{\Delta_n}{\sqrt{I_{\mathrm{eff}}}}\xrightarrow{d}N(0,1),

where $\ell_p$ is the profile log-likelihood and $I_{\mathrm{eff}}>0$ is the efficient information for $\alpha$. The stated regularity conditions ensure that this expansion applies over bounded local values of $h$, and that the maximizing local parameter is the relevant maximizer asymptotically. For the quadratic expression, the unconstrained maximizing value is $\Delta_n/I_{\mathrm{eff}}$; imposing $h\geq0$ gives

\hat h=\max\left(0,\frac{\Delta_n}{I_{\mathrm{eff}}}\right).

If $\Delta_n\leq0$, the constrained maximum is at $h=0$, and twice the log-likelihood improvement is zero. If $\Delta_n>0$, twice the improvement is $\Delta_n^2/I_{\mathrm{eff}}$. With $Z_n=\Delta_n/\sqrt{I_{\mathrm{eff}}}$, the statistic therefore has the same limiting form as $\max(0,Z)^2$, where $Z\sim N(0,1)$. The event $Z\leq0$ has probability $1/2$ and gives zero; conditional on $Z>0$, squaring gives the positive part of a $\chi^2_1$ distribution, with the same squared-value distribution as $Z^2$. This gives the stated mixture. $\square$

Worked Example: A Boundary-Aware P-Value

Suppose a fitted model gives $T=2.7055$ for testing Poisson dispersion against negative-binomial overdispersion. This value is approximately the 90th percentile of $\chi^2_1$. Since $T>0$, the mixture-reference p-value is

P(T_{\mathrm{lim}}\geq 2.7055) =\frac{1}{2}P(\chi^2_1\geq2.7055) \approx\frac{1}{2}(0.10)=0.05.

Using an ordinary $\chi^2_1$ reference would instead give a p-value of about 0.10. The difference is not a change to the observed statistic; it comes from the null distribution’s point mass at zero. Equivalently, the 5% upper-tail critical value for the mixture is the 90th percentile of $\chi^2_1$, approximately 2.7055, rather than the 95th percentile, approximately 3.841.

Conditions, Interpretation, and Common Pitfalls

The mixture result is an asymptotic statement, not a universal finite-sample law. Its use depends on the model and conditions behind the local likelihood expansion. In particular, the nuisance mean model must be identifiable under the Poisson null, and the dispersion direction must contain nonzero information after the mean parameters have been accounted for. If parameters are unidentified under the null, if the model is otherwise irregular, or if the sample is too small for the approximation to be credible, the simple mixture need not describe the statistic well.

For a positive observed statistic $t$, the mixture’s upper-tail probability is one half the $\chi^2_1$ upper-tail probability at $t$. At $t=0$, the mixture has probability $1/2$ at zero; its tail probability using $P(T_{\mathrm{lim}}\geq0)$ is 1. The half-weight is part of the reference distribution, not a rule to halve every p-value regardless of the observed statistic.

Pitfall: treating infinite size as an interior null. The size $r$ cannot take the value infinity as an ordinary finite parameter, and the reciprocal $\alpha=1/r$ is constrained to be nonnegative. Applying Wilks’ interior-parameter rule mechanically and using $\chi^2_1$ ignores this boundary.
Correction: Parameterize dispersion by $\alpha$, recognize $H_0:\alpha=0$ as a boundary test, and use the half-point-mass, half-$\chi^2_1$ limit only when its regularity conditions are appropriate.
Pitfall: assuming the mixture is exact for any data set. The theorem describes a large-sample null distribution. It does not guarantee an accurate finite-sample approximation, particularly when information about dispersion is weak.

Key takeaway: The negative-binomial family approaches Poisson as $\alpha=1/r\downarrow0$. Because that Poisson value lies on the boundary $\alpha\geq0$, a regular one-parameter likelihood-ratio test of overdispersion has a half-$\delta_0$, half-$\chi^2_1$ asymptotic null distribution—not the ordinary $\chi^2_1$ law.

Check Your Understanding

Use the definitions and results above to answer the following.

For fixed mean $\mu$, what happens to the negative-binomial variance as $r\to\infty$, and what is the corresponding limit in terms of $\alpha$?
Which factor in the negative-binomial mass function produces the $e^{-\mu}$ term in the Poisson limit?
Why is $\alpha=0$ a boundary value rather than an interior value for the overdispersion test?
Under the mixture reference law, how should the upper-tail probability for a positive likelihood-ratio statistic be calculated?
Name one condition whose failure can make the simple half-$\delta_0$, half-$\chi^2_1$ approximation inappropriate.