Natural and Mean Parameterizations

From Natural Coordinates to Mean Coordinates

An exponential family can be described using a natural parameter, which enters linearly in the exponent, or using the expected value of its canonical statistic. The second description is often easier to interpret: a mean parameter says what value of the statistic the model expects. The connection between the two descriptions is governed by the log-partition function.

The key question is not merely whether a mean can be calculated from a natural parameter. It is whether the mean can also serve as a coordinate: can a nearby mean determine a unique nearby natural parameter? A nonsingular covariance matrix provides a precise local answer.

Definition: Consider a $d$-parameter exponential family with density or mass function, relative to a fixed dominating measure, $$ f_\eta(x)=h(x)\exp\{\eta^{\mathsf T}T(x)-A(\eta)\}, $$ where $\eta$ belongs to an open natural parameter space $\Omega\subseteq\mathbb R^d$, $T(x)=(T_1(x),\ldots,T_d(x))^{\mathsf T}$ is the canonical statistic, and $A$ is the log-partition function. The mean parameter is the vector $$ m(\eta)=E_\eta[T(X)]. $$ When $A$ is differentiable, $m(\eta)=\nabla A(\eta)$. The natural parameterization uses $\eta$; the mean parameterization uses $m$, wherever the map between them gives valid coordinates.

The notation $E_\eta$ emphasizes which member of the family supplies the expectation. The mean parameter is the expectation of the canonical statistic, not necessarily the expectation of the full observation $X$. For example, a family may have a vector statistic containing both $X$ and $X^2$.

For the results below, assume the family is regular on the open set $\Omega$, the log-partition function is twice continuously differentiable there, and the usual differentiation identities for the exponential family hold. In particular, $$ \nabla A(\eta)=E_\eta[T(X)], \qquad \nabla^2 A(\eta)=\operatorname{Cov}_\eta(T(X)). $$ Thus the derivative matrix of the mean map is the covariance matrix of the canonical statistic. These identities link coordinate changes directly to variation in the statistic under the model.

Local Invertibility

A function can be differentiable without having a differentiable inverse. For a vector-valued map, the relevant test is whether its Jacobian matrix is nonsingular. Here that Jacobian is $\nabla^2 A(\eta)$, so the test is whether the covariance matrix of $T(X)$ is nonsingular at the parameter value in question.

Theorem: Suppose $\Omega$ is open, $A$ is twice continuously differentiable on $\Omega$, and $m(\eta)=\nabla A(\eta)$. If $\operatorname{Cov}_{\eta_0}(T(X))$ is nonsingular at some $\eta_0\in\Omega$, then there are open neighborhoods $U$ of $\eta_0$ and $V$ of $m(\eta_0)$ such that $m$ maps $U$ one-to-one onto $V$, and its inverse $g:V\to U$ is continuously differentiable. At corresponding points, $$ Dg(m(\eta))=\{\nabla^2 A(\eta)\}^{-1} =\{\operatorname{Cov}_{\eta}(T(X))\}^{-1}. $$

Proof. The map $m=\nabla A$ is continuously differentiable because $A$ is twice continuously differentiable. Its Jacobian at $\eta_0$ is $$ Dm(\eta_0)=\nabla^2 A(\eta_0)=\operatorname{Cov}_{\eta_0}(T(X)). $$ By assumption this matrix is nonsingular. The inverse function theorem therefore supplies open neighborhoods $U$ of $\eta_0$ and $V$ of $m(\eta_0)$ on which $m$ is one-to-one and onto, with a continuously differentiable inverse $g$. To calculate the derivative of that inverse, use $m(g(u))=u$ for $u\in V$. The chain rule gives $$ Dm(g(u))Dg(u)=I_d, $$ where $I_d$ is the $d$-dimensional identity matrix. Since $Dm(g(u))=\nabla^2 A(g(u))$ is nonsingular on the chosen neighborhood, solving for $Dg(u)$ gives $$ Dg(u)=\{\nabla^2 A(g(u))\}^{-1}. $$ Substituting $u=m(\eta)$ proves the stated formula. $\square$

If the covariance matrix is positive definite, it is nonsingular, so the theorem applies. In a family with a redundant canonical statistic, however, some nonzero linear combination of the components may be constant with probability one. That creates zero variance in a direction and makes the covariance matrix singular. A mean vector in that redundant coordinate system need not determine all natural coordinates locally.

Worked One-Parameter Conversions

In one dimension, the mean map is a scalar function $m(\eta)=A'(\eta)$, and its derivative is $A''(\eta)=\operatorname{Var}_\eta(T(X))$. When this variance is positive, the local inverse has derivative $$ \frac{d\eta}{dm}=\frac{1}{A''(\eta)}. $$ The examples below show how this general relation appears in familiar families.

Worked Example: Bernoulli Success Probability

Let $X\sim\operatorname{Bernoulli}(p)$, with $0<p<1$. Its mass function can be written $$ P(X=x)=\exp\left\{x\log\frac{p}{1-p}+\log(1-p)\right\},\qquad x\in\{0,1\}. $$ Setting $\eta=\log\{p/(1-p)\}$, the log-partition function is $A(\eta)=\log(1+e^\eta)$. The mean parameter is $$ m=A'(\eta)=\frac{e^\eta}{1+e^\eta}=p. $$

Solving for the natural parameter gives $\eta=\log\{m/(1-m)\}$, with $0<m<1$. The covariance of the canonical statistic $T(X)=X$ is $$ A''(\eta)=\frac{e^\eta}{(1+e^\eta)^2}=p(1-p)=m(1-m). $$ Therefore the derivative of the inverse map is $d\eta/dm=1/\{m(1-m)\}$. At $m=0.7$, the natural parameter is $\log(7/3)$, the variance is $0.7(0.3)=0.21$, and the inverse derivative is $1/0.21=100/21$, approximately $4.762$. The positive derivative agrees with the fact that larger success probabilities require larger natural parameters.

Worked Example: Poisson Mean

Let $X\sim\operatorname{Poisson}(\lambda)$, with $\lambda>0$. Its mass function is $$ P(X=x)=\frac{e^{-\lambda}\lambda^x}{x!} =\frac{1}{x!}\exp\{x\log\lambda-\lambda\},\qquad x=0,1,2,\ldots. $$ Thus $T(X)=X$, $\eta=\log\lambda$, and $A(\eta)=e^\eta$. Differentiation gives the mean parameter $m=A'(\eta)=e^\eta=\lambda$. Hence $\eta=\log m$, for $m>0$.

The second derivative is $A''(\eta)=e^\eta=m$, which is the variance of $X$. It is positive for every finite $\eta$, so the mean map is locally invertible throughout the natural parameter space. Differentiating the inverse directly gives $d\eta/dm=1/m$, which agrees with $1/A''(\eta)=1/m$. For instance, at $\lambda=3$, the natural parameter is $\log 3$, the mean and variance are both $3$, and the inverse derivative is $1/3$.

Worked Example: Exponential Mean and the Sign of the Inverse Derivative

Let $X$ have an exponential distribution with rate $\lambda>0$, so its density is $\lambda e^{-\lambda x}$ for $x>0$. In canonical form, $T(X)=X$, $\eta=-\lambda<0$, and $$ f_\eta(x)=\exp\{\eta x-A(\eta)\},\qquad A(\eta)=-\log(-\eta),\quad \eta<0. $$ The mean parameter is $m=A'(\eta)=-1/\eta=1/\lambda$. Its range is $m>0$, and the inverse relation is $\eta=-1/m$.

Differentiating the log-partition function twice gives $A''(\eta)=1/\eta^2$, the variance of $X$. In mean coordinates this is $m^2$, so the derivative of the inverse must be $1/m^2$. Direct differentiation confirms the sign: $$ \frac{d\eta}{dm}=\frac{d}{dm}\left(-\frac{1}{m}\right)=\frac{1}{m^2}>0. $$ At $\lambda=2$, $\eta=-2$ and $m=1/2$. The variance is $1/4$, and $$ \left.\frac{d\eta}{dm}\right|_{m=1/2}=4 =\frac{1}{1/4} =\frac{1}{A''(-2)}. $$ As $\eta$ increases from $-2$ toward zero, the mean $-1/\eta$ increases. The positive inverse derivative is consistent with that direction of change.

Local Coordinates and Their Limits

The theorem is local: it guarantees a one-to-one correspondence in some neighborhood of a point when the covariance matrix is nonsingular there. It does not, by itself, establish that a proposed set of mean values is the image of the entire natural parameter space, or that every point in that proposed set corresponds to exactly one distribution.

A stronger uniqueness conclusion is available under a useful additional condition. Suppose $\Omega$ is convex and $\nabla^2 A(\eta)$ is positive definite at every point of $\Omega$. Then the gradient map is one-to-one on all of $\Omega$. To see this, take distinct $\eta_1,\eta_2\in\Omega$, and set $v=\eta_1-\eta_2\ne0$. The segment between them lies in $\Omega$. By the fundamental theorem of calculus, $$ v^{\mathsf T}\{m(\eta_1)-m(\eta_2)\} =\int_0^1 v^{\mathsf T}\nabla^2 A\bigl(\eta_2+t v\bigr)v\,dt. $$ The integrand is strictly positive for every $t\in[0,1]$, because the Hessian is positive definite and $v\ne0$. Thus the integral is positive, so $m(\eta_1)\ne m(\eta_2)$. This proves global injectivity under these conditions. It still does not identify the image of the map without further analysis.

These distinctions matter when selecting coordinates for calculation or interpretation. A mean parameter can be intuitive and locally usable while having a restricted range. Conversely, if the covariance matrix loses rank, the local inverse theorem cannot be applied in the full coordinate space. One should check both the domain and the covariance structure rather than assume that every expectation uniquely labels a model.

Common Mistake

A frequent error is to confuse the derivative of the mean map with the derivative of its inverse. The mean-map derivative is $dm/d\eta=A''(\eta)=\operatorname{Var}_\eta(T(X))$. Where this quantity is positive, the inverse derivative is its reciprocal, $d\eta/dm=1/A''(\eta)$. For the exponential model, $m=-1/\eta$ and $\eta=-1/m$; differentiating the latter gives $+1/m^2$, not $-1/m^2$. The sign must also agree with the direction of change: increasing $\eta$ toward zero increases $m$.

Key takeaway: In a regular exponential family, the mean parameter is the gradient of the log-partition function, and its Jacobian is the covariance matrix of the canonical statistic. A nonsingular covariance matrix guarantees a local inverse; the derivative of that inverse is the inverse covariance matrix.

Check Your Understanding

Use the gradient and covariance relationships to answer the following questions.

For a regular $d$-parameter exponential family, what is the Jacobian of the map from natural parameters to mean parameters?
What condition at $\eta_0$ allows the inverse function theorem to give a local mean parameterization?
For a Poisson distribution with mean $m=5$, find the natural parameter and the derivative $d\eta/dm$.
For the exponential model with rate $\lambda=4$, find $\eta$, $m$, the variance, and $d\eta/dm$. Check that the inverse derivative is positive.
Why does local invertibility not automatically show that a proposed mean-parameter space is the image of the full natural parameter space?