Poisson and Multinomial Canonical Representations

Why These Two Discrete Families Have Canonical Forms

The Poisson law describes a count whose possible values range over all nonnegative integers. The multinomial law describes several category counts constrained to sum to a fixed total. Both can be written as exponential families, but their sufficient statistics and parameter coordinates reflect these different support structures. In particular, the multinomial constraint creates negative covariances between distinct category counts.

We use the canonical form \(f_\eta(x)=h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}\), where \(h\) is a carrier, \(T\) is the canonical statistic, \(\eta\) is the natural parameter, and \(A\) is the log-partition function. The representations below specify the support as well as these components; that matters because a formula for the mass alone does not identify the family completely.

The Poisson Law in Canonical Coordinates

To obtain the expression, substitute \(\lambda=e^\eta\): the factors \(e^{-\lambda}\lambda^x\) become \(\exp\{-e^\eta+x\eta\}\). Every finite real \(\eta\) gives \(e^\eta>0\), so it corresponds to a positive Poisson mean. The limiting case \(\lambda=0\) is a degenerate distribution at zero, but it is not represented by a finite natural parameter; it would require \(\eta\) to tend to \(-\infty\).

Poisson Moments From the Canonical Representation

The log-partition function is \(A(\eta)=e^\eta\), so its first two derivatives are both \(e^\eta\). The usual log-partition identities therefore suggest that the mean and variance are \(e^\eta=\lambda\). These identities can also be verified directly, without relying on a formal interchange of differentiation and an infinite sum.

For the mean, use the probability mass function and shift the summation index:

For the second factorial moment, the same shift gives

Since \(X^2=X(X-1)+X\), it follows that \(E[X^2]=\lambda^2+\lambda\). Consequently,

Each series manipulation is valid because the terms are nonnegative and the exponential series sums to \(e^\lambda\), which is finite for every \(\lambda>0\). This establishes the moments directly and confirms the canonical derivative calculation.

The Multinomial Law With a Reference Category

Let \(m\geq 1\) be a fixed integer and let \(X_i\) count the observations in category \(i\), for \(i=1,\ldots,K\), where \(K\geq2\). The counts satisfy \(X_i\in\{0,1,2,\ldots\}\) and \(\sum_{i=1}^K X_i=m\). For probabilities \(p_i>0\) with \(\sum_{i=1}^K p_i=1\), their joint mass is \(m!\prod_i p_i^{x_i}/\prod_i x_i!\) on this support.

Choose category \(K\) as the reference category, and use \(X_1,\ldots,X_{K-1}\) as the canonical statistic. Define \(\eta_i=\log(p_i/p_K)\) for \(i=1,\ldots,K-1\). Since \(x_K=m-\sum_{i=1}^{K-1}x_i\), the probability can be rearranged as

The constraint on the probabilities implies \(p_K=(1+\sum_{i=1}^{K-1}e^{\eta_i})^{-1}\). Therefore the canonical form is as follows.

Every finite vector \(\eta\in\mathbb R^{K-1}\) gives strictly positive probabilities summing to one. Conversely, every strictly positive probability vector gives finite log-ratios \(\eta_i\). Thus the stated natural parameter space is exact for the interior multinomial model. Boundary probability vectors, where one or more categories have probability zero, are limits of this family rather than points with finite reference-category coordinates.

Expected Counts and Covariances

The log-partition function for the reference-category representation is a finite sum expression, so differentiating it involves no exchange of differentiation with an infinite series. Write \(D=1+\sum_{j=1}^{K-1}e^{\eta_j}\). Then

A second derivative gives

These derivatives give the means and covariance matrix of the canonical statistic. For completeness, the relevant identity follows by differentiating the finite normalizing sum. If a finite-support mass has the form \(h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}\), differentiating its sum, which equals one, yields \(\partial A/\partial\eta_i=E[T_i]\). Differentiating that mean once more gives \(\partial^2 A/\partial\eta_i\partial\eta_j=E[T_iT_j]-E[T_i]E[T_j]=\operatorname{Cov}(T_i,T_j)\). All sums here are finite, so the derivatives may be taken term by term.

For the full \(K\)-vector of counts, the mean and covariance formulas include the reference category too:

One direct derivation views the \(m\) observations as independent categorical trials. Let \(I_{ri}\) equal one when trial \(r\) falls in category \(i\), and zero otherwise. Then \(X_i=\sum_{r=1}^m I_{ri}\), \(E[I_{ri}]=p_i\), and \(\operatorname{Var}(I_{ri})=p_i(1-p_i)\). For distinct categories \(i\ne j\), a single trial cannot fall in both, so \(E[I_{ri}I_{rj}]=0\) and \(\operatorname{Cov}(I_{ri},I_{rj})=0-p_ip_j=-p_ip_j\). Independence across trials makes cross-trial covariances zero. Summing the within-trial variances and covariances gives the stated formulas.

Minimal and Redundant Multinomial Coordinates

The reference-category representation uses \(K-1\) coordinates, matching the fact that \(K\) probabilities constrained to sum to one have \(K-1\) free coordinates. It is minimal: the support of \(T=(X_1,\ldots,X_{K-1})\) includes the zero vector and each vector with one coordinate equal to \(m\) and the others zero. These points affinely span \(\mathbb R^{K-1}\), so no nonzero linear combination of the canonical statistics is constant throughout the support.

There is also a symmetric-looking representation using all \(K\) counts and \(\eta_i=\log p_i\). Its log-partition function is \(m\log(\sum_i e^{\eta_i})\). But adding the same constant to every \(\eta_i\) changes neither the probabilities nor the mass, because \(\sum_i X_i=m\). This \(K\)-coordinate form is therefore redundant; it should not be confused with the minimal reference-category form. For example, setting \(\eta_K=0\) recovers the reference coordinates \(\eta_i=\log(p_i/p_K)\).

Common Pitfall and Why the Structure Matters

A common mistake is to treat distinct multinomial counts as independent because each has a simple marginal count distribution. They are not independent when the total is fixed: an increase in one category leaves fewer observations available for the others. The correction is the negative covariance \(-mp_ip_j\), not zero. The covariance matrix must also have a zero direction corresponding to the total, since \(\sum_i X_i=m\) is nonrandom.

The canonical form makes these facts systematic. For a Poisson law, the single natural parameter controls the mean and variance through \(A(\eta)=e^\eta\). For a multinomial law, the gradient and Hessian of its log-partition function encode category means and dependence, while the reference-category coordinates avoid a redundant parameter. These representations are useful for likelihood calculations and for recognizing the mean and covariance structure without repeatedly expanding the probability mass function.