Assessing Fit Under Negative-Binomial Variance

Why Assess Fit After Fitting a Negative-Binomial Model?

A negative-binomial model allows count variability to exceed the Poisson variance, but that flexibility does not guarantee that its variance function describes the data. A fitted model can still understate variability at some mean levels, overstate it at others, or miss systematic features of the count distribution. Assessment therefore asks whether the observed departures from fitted means are compatible with the particular negative-binomial variance model, not simply whether an overdispersion parameter was estimated.

The diagnostics below use fitted means $\hat\mu_i$ and a fitted size parameter $\hat\kappa$. For the NB2 mean-size parameterization, the conditional variance is $\mu_i+\mu_i^2/\kappa$. The same diagnostic logic applies to other negative-binomial variance parameterizations, but the variance used in the residual and the deviance formula must match the model actually fitted.

Definition: For an observed count $y_i$, fitted mean $\hat\mu_i$, and fitted conditional variance $\widehat V_i$, the Pearson residual is $$ r_{Pi}=\frac{y_i-\hat\mu_i}{\sqrt{\widehat V_i}}. $$ Under an NB2 model, $\widehat V_i=\hat\mu_i+\hat\mu_i^2/\hat\kappa$. A Pearson residual expresses the observed count's difference from its fitted mean in units of the model's fitted standard deviation.

Residual Patterns Under NB2

A residual plot is useful when its axes are tied to the model's assumptions. Plot Pearson residuals against fitted means, important predictors, exposure, or observation order when that order has substantive meaning. If the conditional mean and variance are adequately described, residuals should not show a persistent trend, curvature, changing spread, or clusters associated with a predictor. Because counts are discrete, residuals will not form a smooth, perfectly symmetric cloud.

A residual trend is a warning, not a diagnosis by itself. A pattern against a predictor may indicate a poorly specified mean relationship rather than a variance failure. A spread that grows systematically with fitted means is more directly relevant to the variance function. Isolated large residuals can flag observations for investigation, but do not alone establish that the negative-binomial family is wrong.

Worked Example: Reading Pearson Residuals Across Fitted Means

Suppose an NB2 model has fitted size $\hat\kappa=2$. For four observations, fitted means are $1,2,4,8$, and observed counts are $0,1,6,17$. Calculate the Pearson residuals.

For $\hat\mu=1$, the fitted variance is $1+1^2/2=1.5$, so $r_P=(0-1)/\sqrt{1.5}\approx-0.816$.

For $\hat\mu=2$, the fitted variance is $2+2^2/2=4$, so $r_P=(1-2)/2=-0.500$.

For $\hat\mu=4$, the fitted variance is $4+4^2/2=12$, so $r_P=(6-4)/\sqrt{12}\approx0.577$.

For $\hat\mu=8$, the fitted variance is $8+8^2/2=40$, so $r_P=(17-8)/\sqrt{40}\approx1.423$.

These four residuals move from negative to positive as the fitted mean increases. Four observations cannot establish a trend, but a similar pattern across many observations would prompt a check of the mean specification and, separately, whether the variance function is adequate. The increasing fitted variance has already been accounted for in each residual's denominator.

Deviance Residuals and Their Meaning

Pearson residuals standardize the difference from the mean directly. Deviance residuals instead measure each observation's contribution to the likelihood contrast between the fitted model and a saturated model, which gives every observation its own fitted mean. For NB2, with size held fixed at $\kappa$, the signed deviance residual is

r_{Di}=\operatorname{sign}(y_i-\hat\mu_i) \sqrt{2\left[ y_i\log\left(\frac{y_i}{\hat\mu_i}\right) -(y_i+\hat\kappa)\log\left(\frac{y_i+\hat\kappa}{\hat\mu_i+\hat\kappa}\right) \right]}.

When $y_i=0$, the first term $y_i\log(y_i/\hat\mu_i)$ is defined by its continuous limit as zero. The deviance is the sum of squared deviance residuals. These residuals and the Pearson residuals can highlight different observations; neither should be treated as a universal standard-normal score.

Result: The NB2 deviance contribution inside the square root is nonnegative and is zero exactly when $y_i=\hat\mu_i$. To see why, hold $\kappa>0$ fixed and write the log likelihood for one observation, omitting terms independent of the candidate mean $m>0$: $$ \ell(m)=\text{constant}+\ y_i\log m-(y_i+\kappa)\log(m+\kappa). $$ Its derivative is $$ \ell'(m)=\frac{y_i}{m}-\frac{y_i+\kappa}{m+\kappa} =\frac{\kappa(y_i-m)}{m(m+\kappa)}. $$ For $y_i>0$, this derivative is positive for $m<y_i$ and negative for $m>y_i$, so the likelihood is maximized at $m=y_i$. For $y_i=0$, the derivative is negative for every $m>0$, and the likelihood is maximized at the boundary $m=0$. Thus the saturated mean maximizes the likelihood, and twice the log-likelihood difference between the saturated mean and a fitted mean is nonnegative, with equality only when the means agree. Substitution of the NB2 likelihood gives the displayed deviance contribution. $\square$

Worked Example: An NB2 Deviance Residual

For an observation with $y=6$, fitted mean $\hat\mu=4$, and size $\hat\kappa=2$, the residual is positive because $6>4$. Its squared value is

r_D^2=2\left[6\log(6/4)-8\log(8/6)\right] =2\left[6(0.405465)-8(0.287682)\right] \approx 0.26265.

Therefore $r_D\approx+\sqrt{0.26265}\approx0.5125$. For comparison, the Pearson residual is $(6-4)/\sqrt{4+4^2/2}=2/\sqrt{12}\approx0.5774$. The values need not match: the two residuals quantify discrepancy in different ways.

Observed and Fitted Count Frequencies

Residual plots examine individual departures. A frequency comparison instead asks whether the fitted model predicts roughly the observed number of zeros, ones, twos, and larger counts. For a fitted model with independent counts, let $p_{ij}$ be the fitted probability that observation $i$ equals $j$. The expected number of observations with count $j$ is

E_j=\sum_{i=1}^{n}p_{ij}.

This calculation allows different fitted means across observations. Comparing a pooled observed frequency table with probabilities evaluated at the average fitted mean is generally not equivalent: the mixture of observation-specific distributions need not be the distribution at the average mean.

Result: If the counts are conditionally independent and their fitted probabilities are treated as fixed, the observed frequency $O_j$ of count $j$ has expectation $E_j=\sum_i p_{ij}$ and variance $\sum_i p_{ij}(1-p_{ij})$. Indeed, write $I_{ij}=1$ when observation $i$ equals $j$, and $0$ otherwise. Then $O_j=\sum_i I_{ij}$, $E[I_{ij}]=p_{ij}$, and $\operatorname{Var}(I_{ij})=p_{ij}(1-p_{ij})$. Independence makes the covariances zero, so expectations and variances sum as stated. In practice, estimated model parameters add uncertainty, so this variance is not a complete calibration for a fitted-model check. $\square$

Worked Example: Comparing Count Frequencies With NB2 Expectations

Suppose 100 observations share fitted mean $\hat\mu=2$ and size $\hat\kappa=2$. In this case the NB2 probability of count $y$ is $P(Y=y)={y+1\choose y}(1/2)^{y+2}$, since $\kappa/(\kappa+\mu)=1/2$.

The fitted probabilities for counts 0, 1, 2, and 3 are respectively $1/4$, $2/8=1/4$, $3/16$, and $4/32=1/8$. The remaining probability for counts 4 or more is $1-(1/4+1/4+3/16+1/8)=3/16$. Expected frequencies are therefore 25, 25, 18.75, 12.5, and 18.75.

Suppose the observed frequencies in those same categories are 20, 20, 20, 15, and 25. The model underpredicts the combined frequency of counts 3 or more by $40-31.25=8.75$ observations, while predicting more zeros and ones than observed. This is a pattern worth investigating: the fitted model may not capture the tail behavior well. It is not, by itself, proof of a failure, especially because fitted parameters were estimated from the data.

Dispersion Summaries: Useful, but Not Automatic Tests

A common summary is the Pearson statistic $X^2=\sum_i r_{Pi}^2$, often divided by a residual degrees-of-freedom count such as $n-p$, where $p$ is the number of fitted regression coefficients. A value substantially above one can signal more residual variation than expected under the fitted variance; a value substantially below one can signal less. The residual deviance divided by $n-p$ is also reported in practice.

These ratios are diagnostic summaries, not exact universal tests. Their reference behavior depends on the design, fitted parameters, sample size, and distribution of the means. In particular, the negative-binomial size parameter is estimated, counts are discrete, and sparse expected frequencies can make a simple chi-square approximation unreliable. A ratio near one does not establish that the distributional shape or conditional mean is correct, and a ratio away from one does not identify which component failed.

Worked Example: Interpreting a Pearson Dispersion Ratio

A fitted NB2 model has size $\hat\kappa=2$, five fitted means $1,2,3,4,5$, and observed counts $0,0,0,8,10$. Suppose the model has $p=2$ fitted regression coefficients, so the conventional residual degrees of freedom are $5-2=3$. The NB2 variances are $1.5,4,7.5,12,17.5$.

The Pearson residuals are $-1/\sqrt{1.5}\approx-0.8165$, $-2/\sqrt4=-1$, $-3/\sqrt{7.5}\approx-1.0954$, $4/\sqrt{12}\approx1.1547$, and $5/\sqrt{17.5}\approx1.1952$. Their squares sum to approximately $0.6667+1+1.2+1.3333+1.4286=5.6286$.

Thus $X^2/(n-p)=5.6286/3\approx1.876$. This is a prompt to examine residual patterns and fitted frequencies; it is not a formal conclusion that the variance model is inadequate. With only five observations, the ratio is especially unstable.

Practical Assessment and Common Pitfall

A disciplined assessment combines diagnostics rather than relying on one number. Check Pearson and deviance residuals against fitted means and predictors; inspect observed versus fitted count frequencies, using each observation's own fitted distribution when means vary; and report a dispersion ratio with its limitations. When a consequential conclusion depends on the diagnostic, a parametric bootstrap can provide a more tailored reference: simulate counts from the fitted negative-binomial model, refit the model in each simulated data set, and compare the observed diagnostic with its simulated distribution. Refitting matters because it reflects parameter estimation in the diagnostic's calibration.

Common mistake: “The Pearson dispersion ratio is close to one, so the negative-binomial model fits.”
Correction: A single aggregate ratio can conceal trends, tail mismatch, excess zeros, or a misspecified mean. Treat it as one summary alongside residual patterns and count-frequency comparisons.
Common mistake: Compare observed pooled frequencies with NB probabilities calculated at the average fitted mean.
Correction: Sum the observation-specific fitted probabilities for each count category. Heterogeneous fitted means create a mixture of distributions.

Key takeaway: A negative-binomial variance model is credible only to the extent that its standardized residuals, predicted count frequencies, and dispersion summaries show no important systematic disagreement with the data. No single diagnostic certifies fit.

Check Your Understanding

Use these questions to check whether you can distinguish residual calculations from conclusions about model adequacy.

Under NB2, what fitted variance belongs in the denominator of a Pearson residual?
Why can a residual trend against a predictor indicate a mean-model problem rather than a variance-model problem?
For varying fitted means, how should the expected number of observations with count $j$ be calculated?
Why is a Pearson dispersion ratio near one not proof that a negative-binomial model fits well?
What additional step in a parametric bootstrap reflects the fact that model parameters were estimated?