Why Count Imputation Needs a Clustered Model
Counts such as hospital admissions, infection episodes, or service visits are nonnegative integers. Their distributions are often right-skewed, and their variability can be much larger than their mean. When records are nested in clinics, schools, or other groups, records from the same group may also be more alike than records from different groups. An imputation model that ignores either feature can generate implausible values or understate uncertainty.
For a missing count, the conditional imputation model should represent the count's mean, its distribution around that mean, and the dependence induced by the cluster. In MICE, this model is fitted within the chained-equations procedure using the other variables as predictors; missing counts are then drawn from its predictive distribution. The model is not just a device for predicting a conditional mean: its distribution determines the range and frequency of the imputed counts.
These specifications assume that counts are recorded on a meaningful nonnegative-integer scale, the conditional mean is adequately described by the chosen predictors and link, and the cluster structure is represented well enough for the intended imputation. As with other MICE conditional models, the observed information should support imputing the missing values under the missingness assumptions used for the analysis. Including important analysis variables and predictors of missingness can help preserve relevant relationships. Neither a Poisson nor an NB model repairs a badly specified mean structure, omitted clustering, or a structurally impossible outcome.
Specify the Conditional Mean and the Cluster Structure
A common starting point is a log-linear mean model with a cluster-specific random intercept:
Here \(\beta_0\) is an intercept, \(\beta\) contains predictor effects, and \(u_j\) allows the baseline count rate to vary across clusters. Conditional on the predictors and cluster effect, this Poisson model assumes \(\operatorname{Var}(Y_{ij}\mid X_{ij},u_j)=\mu_{ij}\). It also assumes conditional independence of records within a cluster once the modeled shared effect and predictors are accounted for. A Poisson model with a cluster effect can therefore accommodate dependence and variation between clusters without requiring extra-Poisson variation at the individual level conditional on \(u_j\).
If the count is accumulated over different exposure times, include the log exposure as an offset. An offset has coefficient fixed at one, so doubling exposure doubles the expected count when other predictors and the cluster effect are held constant. For positive exposure \(E_{ij}\), the mean specification becomes \(\log(\mu_{ij})=\log(E_{ij})+\beta_0+X_{ij}^{T}\beta+u_j\). This is appropriate when the count is a rate accumulated over exposure, not when exposure is an ordinary predictor with an estimated association.
If counts remain more variable than the conditional Poisson model allows, an NB2 model can replace the conditional Poisson distribution:
The mean model can retain the same log link and cluster effect. When \(\alpha=0\), the NB2 variance reduces to the Poisson variance; larger \(\alpha\) allows a wider conditional distribution. The random intercept and the NB dispersion parameter address different features: the random intercept represents shared cluster-level variation, while \(\alpha\) represents additional variation among counts conditional on the modeled cluster effect and predictors. Using both can be reasonable when the data support both, but it should not be an automatic response to any apparent overdispersion.
Why Cluster Heterogeneity Can Look Like Overdispersion
A cluster effect can produce a marginal variance greater than the marginal mean even when counts are conditionally Poisson. This distinction matters: an ordinary Poisson regression that omits clusters can show extra-Poisson variation because it has omitted shared heterogeneity. The following calculation makes that mechanism explicit.
Suppose, for fixed predictors, \(\mu_{ij}=\lambda\exp(u_j)\), where \(\lambda>0\) and \(u_j\sim N(0,\tau^2)\). Conditional on \(u_j\), the count is Poisson. The expectation of \(\exp(u_j)\) is \(\exp(\tau^2/2)\), and the variance of \(\exp(u_j)\) is \(\exp(\tau^2)\{\exp(\tau^2)-1\}\). Applying the law of total variance gives:
The second term is nonnegative and is positive when \(\tau^2>0\). Thus the marginal count variance exceeds its mean because cluster-specific means vary, even though each conditional distribution is Poisson. This is a justification for checking whether clustering is adequately represented before attributing all extra variation to an NB dispersion parameter.
Worked Example: Specifying an Imputation Model for Clinic Counts
Suppose a hypothetical data set records the number of emergency visits in a year for patients nested in clinics. One patient has 2 visits, 1 year of observation, age 60 years, and a missing value for a binary smoking indicator. The visit count is observed for many other patients, and the smoking indicator is included as a predictor in the count model. For illustration, suppose the fitted count model has \(\beta_0=-0.20\), an age coefficient of \(0.01\) per year, a smoking coefficient of \(0.30\), and a cluster effect \(u_j=0.10\) for this patient's clinic. Take the current value of the smoking indicator to be 1, as might occur during an iteration of the chained equations.
Because each patient has 1 year of observation, the offset is \(\log(1)=0\). The model's linear predictor and mean are:
This mean is in expected visits per patient-year. Under the conditional Poisson model, the conditional variance is \(2.226\) visits squared. Under an NB2 model with the same mean and \(\alpha=0.40\), it is \(2.226+(0.40)(2.226^2)=4.208\) visits squared. Both models have the same conditional expected count here, but the NB2 model allows a substantially wider range of counts. For the missing smoking value, the count model would use the current smoking value in the chain; in turn, the imputation model for smoking can use the count and other predictors. The count draw is not obtained by rounding the mean: it is drawn from the chosen count distribution.
How Overdispersion Changes Predictive Imputations
For a fixed conditional mean, a Poisson distribution concentrates probability more tightly than an NB2 distribution with positive \(\alpha\). The NB2 model does not simply add a constant amount to every imputed count. It changes the predictive distribution: counts near the mean can become less common, while both very small and large counts can become more common. The realized imputation is still an integer draw, not the conditional mean or variance.
Worked Example: Comparing Poisson and Negative-Binomial Predictive Spread
Consider a hypothetical missing count with conditional mean \(\mu=4\). Compare a Poisson model with an NB2 model having \(\alpha=0.50\). The Poisson variance is \(4\). The NB2 variance is:
For this NB2 parameterization, the size is \(r=1/\alpha=2\), and the success-probability parameter is \(p=r/(r+\mu)=2/(2+4)=1/3\). Its zero-count probability is \(P(Y=0)=p^r=(1/3)^2=1/9=0.1111\). The Poisson zero-count probability is \(P(Y=0)=\exp(-4)=0.01832\). Thus, under these specified models, a zero is much more probable in the NB2 predictive distribution than in the Poisson distribution, despite both having mean 4.
This comparison does not mean that an NB2 model always predicts a smaller count for a particular person. It means that across repeated draws at this same conditional mean, its distribution is more dispersed. In MICE, the chosen distribution affects the variability and tail behavior of the completed data sets, and therefore can affect estimates and uncertainty calculated from them.
Overdispersion should be assessed conditional on the fitted mean structure and clustering. A useful diagnostic is the pattern of conditional Pearson residuals, \(r_{ij}=(Y_{ij}-\hat{\mu}_{ij})/\sqrt{\hat{\mu}_{ij}}\), where \(\hat{\mu}_{ij}\) is the fitted conditional mean. Large residual magnitudes, too many extreme counts, or residual patterns associated with predictors can flag a poor fit. Also compare observed and fitted count distributions, including zero frequencies and upper-tail counts. In clustered data, inspect patterns by cluster: systematic differences between cluster means suggest missing cluster structure or a poor cluster-effect specification, not automatically a need for NB dispersion.
Worked Example: Screening Conditional Overdispersion
Suppose a hypothetical Poisson model with a cluster intercept has fitted conditional means of 1, 1, 2, 2, 3, and 3 for six observed counts, respectively. The corresponding counts are 0, 2, 0, 4, 1, and 5. The sum of squared conditional Pearson residuals is:
The residuals include several counts far from their fitted means, so the pattern warrants investigation. This small calculation is a screening summary, not a formal decision that the NB2 model is correct: the sample is tiny, and the sum's reference distribution and degrees of freedom depend on the fitted model. Next inspect whether the extreme residuals come from particular clusters, whether the mean model is missing a nonlinear predictor effect, and whether fitted and observed zeros or upper tails differ. If clustering and mean structure look adequate but conditional spread remains too large, fit and assess an NB2 alternative.
Choosing Between the Models and Recognizing Failure
A clustered Poisson model is a defensible starting point when counts are conditionally close to equidispersed after accounting for predictors, exposure, and cluster structure. Prefer a clustered NB2 model when diagnostics show persistent conditional extra-Poisson variation and the NB2 distribution gives a plausible fit. A Poisson model without cluster structure is generally not an equivalent alternative when records within groups are dependent. Conversely, using NB2 alone does not necessarily represent the shared dependence of observations in the same cluster.
A model comparison should consider more than one dispersion statistic. Examine conditional residuals, observed versus fitted count frequencies, the number of zeros and large counts, and patterns across clusters and predictors. A reduction in residual spread after adding a cluster effect may indicate that clustering, rather than within-cluster NB dispersion, explained the apparent overdispersion. If estimated \(\alpha\) is near zero, the NB2 model approaches Poisson; if \(\alpha\) is large or unstable, check sparse information, omitted predictors, and model convergence before relying on its imputations.
Important limitations remain. A negative-binomial distribution is not a generic remedy for every count-data problem. Excess zeros caused by a distinct structural process, a bounded count, truncation, or a hurdle between zero and positive counts may require a different model. A log-linear mean can also be wrong even when the variance distribution seems flexible. Imputed counts should be checked for support and plausibility, and the imputation model should include variables needed to preserve the relationships relevant to the analysis. Missing-not-at-random mechanisms are not resolved simply by choosing an NB model.
A common mistake is to see a sample variance larger than the sample mean and immediately select an unclustered NB model. That comparison ignores predictor effects and cluster dependence. The correction is to assess dispersion conditional on a suitable mean and cluster structure; then use NB2 only when extra conditional variation remains. Another mistake is to impute the fitted mean itself, or to round it, because it is not an observed count. Instead, predictive imputation uses integer-valued draws from the specified distribution, with the model's uncertainty represented as required by multiple imputation.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A count is collected over 3 years of observation. What offset belongs in a log-linear count model, and what does it do to the mean?
- At a conditional mean of 5, calculate the NB2 variance when \(\alpha=0.20\). Compare it with the Poisson variance.
- Conditional residuals show extra spread in an unclustered Poisson model, but the largest counts are concentrated in a few clinics. What should be investigated before adding NB dispersion?
- Why can Poisson and NB2 imputations have the same conditional mean but different completed-data distributions?
Solutions. (1) Use \(\log(3)\) as an offset with coefficient fixed at one. It multiplies the expected count by 3 relative to one year of exposure, holding other terms fixed. (2) The NB2 variance is \(5+(0.20)(5^2)=5+5=10\), compared with Poisson variance 5; both are in count-squared units. (3) Fit and inspect a model that represents clinic-level variation, then check whether conditional overdispersion remains. Cluster concentration can reflect omitted dependence rather than NB variation. (4) Their means can be set to the same value, but NB2 with positive \(\alpha\) has variance \(\mu+\alpha\mu^2\), greater than the Poisson variance \(\mu\), so predictive draws have different spread and tail probabilities.