Multivariate Exponential-Family Examples

Canonical Form as a Common Language

A multinomial count vector, a collection of independent Poisson counts, and a multivariate normal sample look like different kinds of data. Yet each likelihood separates into a part involving the data and parameters together, a parameter-only normalizing term, and a parameter-free base measure. Writing that structure explicitly reveals both the natural parameter and the data summaries through which the likelihood depends on the observations.

Definition: A dominated model has a canonical exponential-family representation if its density or mass function can be written as $$ f_\eta(x)=h(x)\exp\{\eta^\mathsf{T}T(x)-A(\eta)\}. $$ Here $T(x)$ is a statistic, $\eta$ is the natural parameter, $h(x)$ is a parameter-free base measure, and $A(\eta)$ is the log-partition function. The parameter space must make the density or mass function integrate or sum to one. In these examples, the support does not depend on the parameter.

A statistic $T(X)$ is sufficient for a parameter when the conditional distribution of the data given $T(X)$ does not depend on that parameter. For the dominated models below, we identify sufficiency from the standard likelihood factorization criterion: if the likelihood is a product of a parameter-dependent function of $T(x)$ and a parameter-free function of the full data, then $T$ is sufficient. A statistic need not be a vector of minimal length to be sufficient.

Multinomial Counts and Reference-Category Log Odds

Let $C=(C_1,\ldots,C_K)$ have a multinomial distribution with fixed total $n$ and probabilities $p_1,\ldots,p_K>0$, where $\sum_{j=1}^K p_j=1$. Its mass function is $$ P(C=c)=\frac{n!}{\prod_{j=1}^K c_j!}\prod_{j=1}^K p_j^{c_j}, \qquad c_j\in\{0,1,\ldots\},\quad \sum_{j=1}^K c_j=n. $$ Choose category $K$ as the reference and define $\eta_j=\log(p_j/p_K)$ for $j=1,\ldots,K-1$. Solving these relations together with the sum-to-one constraint gives $p_K=(1+\sum_{j=1}^{K-1}e^{\eta_j})^{-1}$ and $p_j=e^{\eta_j}p_K$. Thus the likelihood can be expressed as $$ P(C=c)=\frac{n!}{\prod_{j=1}^K c_j!} \exp\left\{\sum_{j=1}^{K-1}c_j\eta_j -n\log\left(1+\sum_{j=1}^{K-1}e^{\eta_j}\right)\right\}. $$ The natural statistic is $(C_1,\ldots,C_{K-1})$; the log-partition function is $n\log(1+\sum_{j=1}^{K-1}e^{\eta_j})$. The omitted count is determined by $C_K=n-\sum_{j<K}C_j$. For ordered categorical observations, the count vector is sufficient: conditional on the counts, every ordering has the same probability, so the conditional distribution of the ordering does not depend on $p$.

Worked Example: Three Multinomial Categories

Suppose $n=10$, $K=3$, and the observed counts are $(4,3,3)$. Let the category probabilities be $(0.2,0.3,0.5)$, using category 3 as the reference.

The natural parameters are $\eta_1=\log(0.2/0.5)=\log(0.4)$ and $\eta_2=\log(0.3/0.5)=\log(0.6)$. The log-partition function is $A(\eta)=10\log(1+e^{\eta_1}+e^{\eta_2})=10\log 2$.

The base measure at this count vector is $h(c)=10!/(4!3!3!)=3628800/(24\cdot6\cdot6)=4200$. Therefore the canonical expression is $4200\exp\{4\log(0.4)+3\log(0.6)-10\log 2\}$. Since $0.4/2=0.2$ and $0.6/2=0.3$, this equals $4200(0.2)^4(0.3)^3(0.5)^3$, the original multinomial probability. The observed counts determine all parameter-dependent terms.

Independent Poisson Counts

Let $Y_1,\ldots,Y_m$ be independent with $Y_j\sim\operatorname{Poisson}(\lambda_j)$, where every $\lambda_j>0$. Their joint mass function is $$ P(Y=y)=\prod_{j=1}^m\frac{e^{-\lambda_j}\lambda_j^{y_j}}{y_j!} =\left(\prod_{j=1}^m\frac{1}{y_j!}\right) \exp\left\{\sum_{j=1}^m y_j\log\lambda_j-\sum_{j=1}^m\lambda_j\right\}. $$ Set $\eta_j=\log\lambda_j$. Then $T(Y)=(Y_1,\ldots,Y_m)$, $h(y)=\prod_j1/y_j!$, and $A(\eta)=\sum_j e^{\eta_j}$. The natural parameter space is all of $\mathbb R^m$, because each positive mean corresponds to one real log mean.

For a single observed vector, the full vector $Y$ is sufficient for the vector of means. More usefully, suppose there are $r$ independent replicate vectors $Y_i=(Y_{i1},\ldots,Y_{im})$, with the same component means in each replicate. The joint likelihood has parameter-dependent part $$ \exp\left\{\sum_{j=1}^m\left(\sum_{i=1}^rY_{ij}\right)\log\lambda_j-r\sum_{j=1}^m\lambda_j\right\}. $$ Consequently, the componentwise totals $S_j=\sum_{i=1}^rY_{ij}$ are sufficient. The log-partition function for the full replicate sample is $r\sum_j e^{\eta_j}$.

Worked Example: Totals from Independent Poisson Replicates

For three replicate units, suppose the observed counts for two independent components are $Y_1=(1,2,4)$ and $Y_2=(0,2,3)$. The component totals are $S_1=7$ and $S_2=5$. Take the component means to be $\lambda_1=2$ and $\lambda_2=1.5$.

The natural parameters are $\eta_1=\log 2$ and $\eta_2=\log 1.5$. For each replicate the log-partition function is $A(\eta)=e^{\eta_1}+e^{\eta_2}=3.5$, so for three replicates it contributes $3A=10.5$. The parameter-dependent part of the joint likelihood is therefore $\exp\{7\log 2+5\log 1.5-10.5\}$.

The base measure for the full array is $1/(1!2!4!0!2!3!)=1/(48\cdot12)=1/576$. Thus the complete likelihood is $(1/576)\exp\{7\log 2+5\log 1.5-10.5\}$. The replicate pattern affects the parameter-free base measure, but the means enter through the totals $7$ and $5$.

Multivariate Normal with Unknown Mean and Covariance

Let $X\in\mathbb R^d$ have distribution $N(\mu,\Sigma)$, where $\mu\in\mathbb R^d$ and $\Sigma$ is symmetric positive definite. Its density is $$ f(x)= (2\pi)^{-d/2}|\Sigma|^{-1/2} \exp\left\{-\frac12(x-\mu)^\mathsf{T}\Sigma^{-1}(x-\mu)\right\}. $$ Expanding the quadratic gives a term linear in $x$, a quadratic term in $x$, and a term depending only on the parameters. Define $\eta_1=\Sigma^{-1}\mu$ and the symmetric matrix $\eta_2=-\frac12\Sigma^{-1}$. With the trace inner product on symmetric matrices, $x^\mathsf{T}\eta_2x=\operatorname{tr}(\eta_2xx^\mathsf{T})$. The canonical form is $$ f(x)=\exp\left\{\eta_1^\mathsf{T}x+\operatorname{tr}(\eta_2xx^\mathsf{T})-A(\eta_1,\eta_2)\right\}, $$ where $h(x)=1$, $T(x)=(x,xx^\mathsf{T})$, and $$ A(\eta_1,\eta_2) =-\frac14\eta_1^\mathsf{T}\eta_2^{-1}\eta_1 +\frac d2\log\pi-\frac12\log|-\eta_2|, \qquad \eta_2\text{ negative definite}. $$ The domain restriction is essential: it ensures the quadratic exponential is integrable. Every $\eta_1\in\mathbb R^d$ and negative definite $\eta_2$ correspond to a valid $\mu,\Sigma$, via $\Sigma=(-2\eta_2)^{-1}$ and $\mu=\Sigma\eta_1$.

For $n$ independent observations $X_1,\ldots,X_n$ from this model, the natural statistics are $\sum_iX_i$ and $\sum_iX_iX_i^\mathsf{T}$, and the log-partition function is $nA(\eta_1,\eta_2)$. Indeed, the sample likelihood is $$ \exp\left\{\eta_1^\mathsf{T}\sum_{i=1}^nX_i+ \operatorname{tr}\left(\eta_2\sum_{i=1}^nX_iX_i^\mathsf{T}\right)-nA(\eta_1,\eta_2)\right\}. $$ It follows from the likelihood factorization criterion that these two sums together are sufficient for $(\mu,\Sigma)$. They retain the information in the sample mean and the unnormalized centered scatter matrix, since $\sum_i(X_i-\bar X)(X_i-\bar X)^\mathsf{T} =\sum_iX_iX_i^\mathsf{T}-n\bar X\bar X^\mathsf{T}$.

Worked Example: Normal Sufficient Statistics in Two Dimensions

Consider three observed vectors $x_1=(1,0)^\mathsf{T}$, $x_2=(0,1)^\mathsf{T}$, and $x_3=(1,1)^\mathsf{T}$. Their first sufficient statistic is $\sum_i x_i=(2,2)^\mathsf{T}$. Their second is $$ \sum_{i=1}^3x_ix_i^\mathsf{T} =\begin{pmatrix}1&0\\0&0\end{pmatrix} +\begin{pmatrix}0&0\\0&1\end{pmatrix} +\begin{pmatrix}1&1\\1&1\end{pmatrix} =\begin{pmatrix}2&1\\1&2\end{pmatrix}. $$

For illustration, take $\mu=(0,0)^\mathsf{T}$ and $\Sigma=I_2$. Then $\eta_1=(0,0)^\mathsf{T}$, $\eta_2=-I_2/2$, and $A=\log(2\pi)$. The sample log likelihood is $-\frac12\operatorname{tr}(\sum_i x_ix_i^\mathsf{T})-3\log(2\pi) =-2-3\log(2\pi)$. The trace is $2+2=4$, so the quadratic contribution is $-4/2=-2$, verifying the calculation directly from the three squared Euclidean lengths $1,1,2$.

Deriving the Normal Log-Partition Function

The normal log-partition expression can be checked directly by integrating the canonical kernel. This also verifies the sign and determinant terms, which are easy to confuse. Let $B=-\eta_2$, so $B$ is positive definite. Completing the square gives $$ x^\mathsf{T}\eta_2x+\eta_1^\mathsf{T}x =-\left(x-\frac12B^{-1}\eta_1\right)^\mathsf{T} B\left(x-\frac12B^{-1}\eta_1\right) +\frac14\eta_1^\mathsf{T}B^{-1}\eta_1. $$

Integrating over $\mathbb R^d$, translate by $\frac12B^{-1}\eta_1$ and diagonalize the positive definite matrix $B$. The resulting Gaussian integral is $\pi^{d/2}|B|^{-1/2}$. Hence $$ \int_{\mathbb R^d}\exp\{x^\mathsf{T}\eta_2x+\eta_1^\mathsf{T}x\}\,dx =\pi^{d/2}|-\eta_2|^{-1/2} \exp\left\{-\frac14\eta_1^\mathsf{T}\eta_2^{-1}\eta_1\right\} =\exp\{A(\eta_1,\eta_2)\}. $$ Taking the logarithm proves the stated $A$. Substituting $\eta_1=\Sigma^{-1}\mu$ and $\eta_2=-\frac12\Sigma^{-1}$ yields $A=\frac12\mu^\mathsf{T}\Sigma^{-1}\mu+\frac12\log|\Sigma|+\frac d2\log(2\pi)$, exactly the constant required by the normal density. $\square$

Common Parameterization Errors

Putting every multinomial count into the natural parameter vector. The counts sum to fixed $n$, so one is redundant. Using $K-1$ reference-category log odds gives a one-to-one parameterization of the interior of the probability simplex.
Dropping the Poisson log-partition term. The term $-\sum_j\lambda_j=-\sum_je^{\eta_j}$ is required to normalize the mass function; the log means alone do not define the likelihood.
Using covariance where precision is required in the normal form. The quadratic coefficient is $-\frac12\Sigma^{-1}$, not $-\frac12\Sigma$. Also, the matrix term uses the trace pairing, which correctly accounts for off-diagonal entries.
Confusing a sufficient statistic with the original parameter. A statistic summarizes the observed data. Natural parameters re-express the model parameters; they are not data summaries.

Key takeaway: In each model, rewrite the log likelihood and collect the data-dependent terms. Their coefficients identify the natural statistics; the parameter-only term that normalizes the model is the log-partition function. Check the base measure and parameter domain as well as the visible exponential terms.

Check Your Understanding

For each model, identify the canonical ingredients and explain what information the sufficient statistic retains.

In a multinomial model with $K$ categories and fixed total $n$, why can the first $K-1$ counts be used instead of all $K$ counts?
For independent Poisson variables with means $\lambda_j$, what are the natural parameters and log-partition function?
With $r$ replicated independent Poisson vectors sharing component means, which totals carry the parameter-dependent part of the likelihood?
For an iid multivariate normal sample with unknown mean and covariance, name the two sample statistics in its canonical likelihood.
Why must the normal natural-parameter matrix $\eta_2$ be negative definite?