Conway-Maxwell-Poisson For Flexible Count Dispersion

Why Use a Flexible Count Distribution?

A Poisson model imposes a distinctive restriction: conditional on its mean, the count variance equals that mean. Some count data are more concentrated than this restriction allows; others vary more. A count distribution that can represent both patterns is useful when the response is still a nonnegative integer count, but the Poisson mean–variance equality is not credible.

The Conway–Maxwell–Poisson (CMP, also called COM-Poisson) family adds a dispersion parameter to a Poisson-shaped probability mass function. Its flexibility comes with an important interpretive cost: the CMP rate parameter is not, in general, the response mean. The mean must be obtained from the normalized distribution, usually by calculation or numerical methods.

The restriction \(\nu>0\) ensures that the defining series converges for every \(\lambda>0\). To see why, let \(w_y=\lambda^y/(y!)^\nu\) be the unnormalized weight at count \(y\). The ratio of successive weights is $$ \frac{w_{y+1}}{w_y}=\frac{\lambda}{(y+1)^\nu}, $$ which approaches zero as \(y\) increases when \(\nu>0\). Thus the weights eventually decrease sufficiently rapidly for their sum to be finite. Dividing each weight by that finite sum produces a probability distribution.

What the Dispersion Parameter Changes

When \(\nu=1\), the normalizing constant is \(Z(\lambda,1)=e^\lambda\). Therefore $$ P(Y=y)=\frac{\lambda^y}{y!e^\lambda}, $$ which is the Poisson probability mass function with mean and variance \(\lambda\). For \(\nu\ne1\), the factorial is raised to a different power, changing how quickly the weights for larger counts diminish. Values \(\nu>1\) penalize larger counts more strongly than the Poisson model does; values \(0<\nu<1\) penalize them less strongly. These changes can produce variance below or above the mean, respectively, although the exact mean and variance depend on both \(\lambda\) and \(\nu\).

The distinction between \(\lambda\) and the mean matters in applications. In a regression model, one may specify a linear predictor for \(\log(\lambda)\), but that does not make \(\log(\lambda)\) the log of the conditional mean except in the Poisson special case. Some software instead offers mean-parameterized CMP models. Their parameterization must be checked: it is not interchangeable with the rate parameterization above without a conversion.

These identities follow directly from the probability mass function. Write \(\theta=\log\lambda\), so each unnormalized weight is \(e^{y\theta}/(y!)^\nu\). Differentiating the convergent series for \(Z\) with respect to \(\theta\) gives $$ \frac{\partial Z}{\partial\theta} =\sum_{y=0}^{\infty}\frac{y e^{y\theta}}{(y!)^\nu}. $$ Dividing by \(Z\) gives \(E(Y)\), and hence \(\partial\log Z/\partial\theta=E(Y)\). Differentiating once more yields $$ \frac{\partial^2\log Z}{\partial\theta^2} =\frac{\sum_y y^2 e^{y\theta}/(y!)^\nu}{Z} -\left(\frac{\sum_y y e^{y\theta}/(y!)^\nu}{Z}\right)^2 =\operatorname{Var}(Y). $$ This also shows that the variance is not obtained simply by reading \(\lambda\) or \(\nu\) from the model specification.

Calculating Moments from the Weights

For particular parameter values, the normalizing constant and moments can be calculated by summing the weights. Define $$ S_0=\sum_{y=0}^{\infty}w_y,\qquad S_1=\sum_{y=0}^{\infty}y w_y,\qquad S_2=\sum_{y=0}^{\infty}y^2 w_y, \qquad w_y=\frac{\lambda^y}{(y!)^\nu}. $$ Then \(Z=S_0\), \(E(Y)=S_1/S_0\), and \(\operatorname{Var}(Y)=S_2/S_0-(S_1/S_0)^2\). In practice, the series is evaluated numerically with enough terms to make the omitted tail negligible. The successive-weight ratio above is useful for checking how quickly the terms shrink.

Using CMP for Count Responses

A CMP response model is appropriate when the observed response consists of nonnegative integer counts and a Poisson distribution’s conditional mean–variance restriction is inadequate. In a generalized linear mixed model, it can be used as the conditional response distribution. For example, given random effects \(b_i\), an observation \(Y_{ij}\) might have a CMP distribution with rate \(\lambda_{ij}\) and dispersion \(\nu\). The model must specify how the rate depends on covariates and, if present, random effects. Conditional independence of responses given those random effects is an additional modeling assumption when that factorization is used.

The distributional choice does not correct every possible mismatch. CMP still assigns probability only to nonnegative integer counts. It does not automatically account for excess zeros, omitted predictors, dependence left unexplained by the random effects, or changes in dispersion across observations. A common-\(\nu\) specification, for instance, assumes one dispersion parameter applies across the modeled observations unless the model explicitly allows otherwise.

The alternatives answer different modeling needs. A Poisson model is simpler and imposes equality of conditional mean and variance. Negative binomial models provide a probability likelihood with particular forms of overdispersion; they do not generally provide CMP’s underdispersion behavior. Quasi-Poisson specifies a mean–variance relationship for inference but is not a full probability likelihood. CMP is a full count likelihood with flexible dispersion, but its normalizing constant and mean calculations can make estimation more computationally demanding. The choice should reflect the response support, the mean–variance pattern, and the scientific model—not just which distribution is easiest to fit.

Diagnostics, Pitfalls, and Limits

After fitting a CMP model, assess whether its assumptions and fitted probabilities describe the data adequately. Compare observed and fitted count frequencies, including the zero frequency and the upper tail; examine residual patterns against fitted values and predictors; and check whether conclusions change under plausible alternative count distributions. In a mixed model, these checks must also be interpreted in light of the assumed random-effects structure. A flexible conditional response distribution cannot compensate for an unsuitable dependence model.

• Do not equate \(\lambda\) with the mean. That equality holds for the Poisson special case \(\nu=1\), not generally for CMP.
• Do not treat \(\nu\) as a direct variance multiplier. The variance depends on both parameters through the normalized distribution; calculate or estimate it.
• Do not infer that every unusual count pattern is dispersion. Excess zeros, dependence, or a misspecified mean structure can also create apparent variance mismatch.
• Check numerical convergence. A truncated series that omits material tail weight can distort \(Z\), the mean, the variance, and fitted probabilities.

CMP’s flexibility has limits. The exact mean is generally not available as a simple closed-form function of \(\lambda\) and \(\nu\), and numerical work may be necessary. Parameter estimates can also be difficult to interpret directly on a mean scale. For reporting, provide conditional means or predicted probabilities when relevant, as well as the parameterization used. Finally, variance agreement alone is not proof of a suitable probability model: the full count distribution and the model’s conditional-independence and random-effects assumptions still matter.