Vector-Parameter Exponential-Family Extensions

From a Scalar Parameter to a Vector

A one-parameter exponential-family representation pairs a scalar natural parameter with a scalar statistic. Some models require more than one parameter-dependent feature of the observation in the exponent. The natural extension is to pair a vector of natural parameters with a vector of statistics, using their inner product. The essential bookkeeping is to keep the parameter vector, statistic vector, and log-partition function dimensionally consistent.

Definition: Let $y$ take values in a sample space $\mathcal{Y}$, let $\nu$ be a parameter-independent measure on that space, and let $h(y)$ be a nonnegative carrier function. A $k$-dimensional exponential-family representation has density, relative to $\nu$, $$ f_{\eta}(y)=h(y)\exp\{\eta^{\mathsf T}T(y)-A(\eta)\}, $$ where $\eta\in\mathbb{R}^k$ is the natural-parameter vector, $T(y)=(T_1(y),\ldots,T_k(y))^{\mathsf T}$ is the statistic vector, and $A(\eta)$ is the log-partition function. The natural-parameter space consists of values of $\eta$ for which $$ 0<\int_{\mathcal{Y}}h(y)\exp\{\eta^{\mathsf T}T(y)\}\,d\nu(y)<\infty. $$

When the integral is finite and positive, normalization determines the log-partition function:

A(\eta)=\log\left(\int_{\mathcal{Y}}h(y)\exp\{\eta^{\mathsf T}T(y)\}\,d\nu(y)\right).

The parameter vector and statistic vector must have the same dimension, so $\eta^{\mathsf T}T(y)=\sum_{j=1}^k\eta_jT_j(y)$ is a scalar. The carrier and measure account for factors that do not depend on $\eta$. As in the scalar form, a particular model can sometimes be represented in more than one way; the representation is not automatically unique or identifiable.

Reading and Checking a Vector Representation

To identify the parts of a proposed representation, first collect all terms involving the unknown parameters and the observation. Each parameter coefficient in the exponent contributes a component of $T(y)$; the corresponding parameter is a component of $\eta$. Terms depending only on the parameters belong in $-A(\eta)$, while terms depending only on the observation can be placed in the carrier. The integral or sum over the sample space must then equal one after normalization.

Worked Example: A Three-Outcome Model

Let the sample space be $\{a,b,c\}$, with counting measure and carrier $h(y)=1$. Set $T(a)=(0,0)^{\mathsf T}$, $T(b)=(1,0)^{\mathsf T}$, and $T(c)=(0,1)^{\mathsf T}$. Find the log-partition function and the probabilities when $\eta=(\log 2,0)^{\mathsf T}$.

The unnormalized weights at $a,b,c$ are respectively $1$, $\exp(\eta_1)$, and $\exp(\eta_2)$. Their sum is $1+\exp(\eta_1)+\exp(\eta_2)$, so

A(\eta)=\log\{1+\exp(\eta_1)+\exp(\eta_2)\}.

At the specified parameter, the three weights are $1,2,1$, and their sum is $4$. Thus $A(\eta)=\log 4$, and the normalized probabilities are

P(Y=a)=\frac14,\qquad P(Y=b)=\frac12,\qquad P(Y=c)=\frac14.

The calculation checks both the vector bookkeeping and normalization: each outcome's exponent is the inner product of the same two-dimensional vectors, and the probabilities sum to one.

The coordinates used for the statistic vector are a choice, provided the corresponding natural parameter is changed consistently. If $B$ is an invertible $k\times k$ matrix, setting $T'(y)=BT(y)$ and $\eta'=B^{-\mathsf T}\eta$ preserves the exponent, since $\eta'^{\mathsf T}T'(y)=\eta^{\mathsf T}T(y)$. This is a change of coordinates, not a change to the probability model.

Worked Example: Changing Statistic Coordinates

Use the three-outcome model above and replace $T$ by $T'=BT$, where $B$ is the diagonal matrix with diagonal entries $2$ and $3$. For $\eta=(\log 2,0)^{\mathsf T}$, find the new natural parameter and check the probabilities.

Here $B^{-\mathsf T}$ is diagonal with entries $1/2$ and $1/3$, so

\eta'=\left(\frac{\log 2}{2},0\right)^{\mathsf T}.

The transformed statistics at $a,b,c$ are $(0,0)^{\mathsf T}$, $(2,0)^{\mathsf T}$, and $(0,3)^{\mathsf T}$. Their exponentiated inner products with $\eta'$ are $1,2,1$. The new log-partition function at this parameter is therefore still $\log 4$, giving probabilities $1/4,1/2,1/4$, exactly as before. Using the transformed statistics but leaving $\eta$ unchanged would not preserve these weights.

What the Log-Partition Function Encodes

The log-partition function does more than supply the normalizing constant. Under suitable regularity conditions, its first derivatives give the means of the components of $T(Y)$, and its second derivatives give their covariances. The condition below is a convenient sufficient condition for the differentiation under the integral used in the proof; it is not a claim that every exponential-family representation satisfies it everywhere.

Theorem: Suppose $\eta_0$ lies in an open subset of the natural-parameter space. Assume that in a neighborhood $U$ of $\eta_0$, the integral defining $A(\eta)$ is finite and positive, and that for every compact subset $K$ of $U$, there is a $\nu$-integrable function $g_K(y)$ such that, for all $\eta\in K$, $$ h(y)\exp\{\eta^{\mathsf T}T(y)\}\bigl(1+\|T(y)\|^2\bigr)\le g_K(y) $$ for $\nu$-almost every $y$. Then $A$ has first and second derivatives at $\eta_0$, and $$ \nabla A(\eta_0)=E_{\eta_0}[T(Y)],\qquad \nabla^2 A(\eta_0)=\operatorname{Cov}_{\eta_0}(T(Y)). $$

Proof. Write $Z(\eta)=\int h(y)\exp\{\eta^{\mathsf T}T(y)\}\,d\nu(y)$, so $A(\eta)=\log Z(\eta)$. The stated domination bounds the integrands and their first and second parameter derivatives on compact neighborhoods. Dominated convergence therefore permits differentiating under the integral. Componentwise, for $i,j\in\{1,\ldots,k\}$,

\frac{\partial Z}{\partial\eta_i} =\int T_i(y)h(y)e^{\eta^{\mathsf T}T(y)}\,d\nu(y),\qquad \frac{\partial^2 Z}{\partial\eta_i\partial\eta_j} =\int T_i(y)T_j(y)h(y)e^{\eta^{\mathsf T}T(y)}\,d\nu(y).

By the density formula, these integrals divided by $Z(\eta)$ are respectively $E_\eta[T_i(Y)]$ and $E_\eta[T_i(Y)T_j(Y)]$. Differentiating $A=\log Z$ gives

\frac{\partial A}{\partial\eta_i}=\frac{1}{Z}\frac{\partial Z}{\partial\eta_i} =E_\eta[T_i(Y)],

and

\frac{\partial^2 A}{\partial\eta_i\partial\eta_j} =\frac{1}{Z}\frac{\partial^2 Z}{\partial\eta_i\partial\eta_j} -\frac{1}{Z^2}\frac{\partial Z}{\partial\eta_i}\frac{\partial Z}{\partial\eta_j} =E_\eta[T_i(Y)T_j(Y)]-E_\eta[T_i(Y)]E_\eta[T_j(Y)].

The last expression is $\operatorname{Cov}_\eta(T_i(Y),T_j(Y))$, which proves both identities. $\square$

In a finite-support model, the required differentiation is just differentiation of a finite sum. For continuous models, the domination condition is one way to ensure that the same calculation is valid rather than merely formal.

Normal Model with Unknown Mean and Variance

The normal density with unknown mean $\mu\in\mathbb{R}$ and variance $\sigma^2>0$ provides a standard example with two natural parameters. Starting from its usual density, expand the squared deviation to isolate the observation-dependent terms:

f(y\mid\mu,\sigma^2) =\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(y-\mu)^2}{2\sigma^2}\right\} =\exp\left\{\frac{\mu}{\sigma^2}y-\frac{1}{2\sigma^2}y^2 -\frac{\mu^2}{2\sigma^2}-\frac12\log(2\pi\sigma^2)\right\}.

Relative to Lebesgue measure, take $h(y)=1$ and $T(y)=(y,y^2)^{\mathsf T}$. The matching natural parameters and log-partition function are

\eta_1=\frac{\mu}{\sigma^2},\qquad \eta_2=-\frac{1}{2\sigma^2},\qquad A(\eta)=\frac{\mu^2}{2\sigma^2}+\frac12\log(2\pi\sigma^2).

For a formula entirely in natural parameters, solve $\sigma^2=-1/(2\eta_2)$ and $\mu=-\eta_1/(2\eta_2)$. This gives

A(\eta)=-\frac{\eta_1^2}{4\eta_2}+\frac12\log\left(-\frac{\pi}{\eta_2}\right), \qquad \eta_2<0,\ \eta_1\in\mathbb{R}.

The restriction $\eta_2<0$ is essential: it is exactly the condition that the quadratic term makes the integral over the real line finite. If $\eta_2=0$, the remaining exponential is not integrable over the whole line; if $\eta_2>0$, it grows in the tails. Thus the natural-parameter space is $\{(\eta_1,\eta_2):\eta_1\in\mathbb{R},\eta_2<0\}$.

Worked Example: Recovering a Normal Density from Natural Parameters

Suppose $Y\sim N(2,4)$, so the mean is $2$ and the variance is $4$. Identify its natural parameters and verify the density at $y=3$.

The parameters are $\eta_1=2/4=1/2$ and $\eta_2=-1/(2\cdot4)=-1/8$. At $y=3$, the exponent in the vector representation is

\eta_1y+\eta_2y^2-A(\eta) =\frac32-\frac98-\frac12-\frac12\log(8\pi) =-\frac18-\frac12\log(8\pi).

Exponentiating gives $f(3)=e^{-1/8}/\sqrt{8\pi}$. Directly from the normal density, the deviation from the mean is $3-2=1$, so

f(3)=\frac{1}{\sqrt{2\pi\cdot4}} \exp\left\{-\frac{(3-2)^2}{2\cdot4}\right\} =\frac{e^{-1/8}}{\sqrt{8\pi}},

which agrees. The inverse formulas also recover the original parameters: $-1/(2\eta_2)=4$ and $-\eta_1/(2\eta_2)=2$.

Common Mistake and Why the Vector Form Matters

Common mistake: seeing two parameters in a model and putting both into the natural-parameter vector without checking the terms they multiply. For the normal density, the exponent has one parameter-dependent coefficient multiplying $y$, and another multiplying $y^2$. The appropriate statistic is therefore $(y,y^2)^{\mathsf T}$, not $(y,1)^{\mathsf T}$. The parameter-only terms must also be collected into $-A(\eta)$; omitting them would leave a density that does not integrate to one.

The vector notation provides a common language for models in which several features of an observation contribute to the exponent. It also makes the role of normalization and the log-partition function explicit. When the regularity conditions of the theorem hold, that same function summarizes the first and second moments of the statistic vector. These facts make it important to distinguish carefully between the natural parameter, the statistic, and the normalizer rather than treating them as interchangeable parts of a formula.

Check Your Understanding

Use the definitions and calculations in this tutorial to answer the following.

In a $k$-dimensional exponential-family representation, what dimensions must $\eta$ and $T(y)$ have, and why is their inner product a scalar?
For the three-outcome example, calculate the probabilities when $\eta=(0,\log 3)^{\mathsf T}$.
Explain why an invertible change $T'=BT$ requires the corresponding change $\eta'=B^{-\mathsf T}\eta$ to preserve the model.
For the normal model, identify $T(y)$, $\eta$, and the condition defining the natural-parameter space.
Under the theorem's regularity conditions, what do the gradient and Hessian of $A$ represent?