One-Parameter Mean Maps and Variance

Why the Mean Map Matters

In a one-parameter exponential family, the natural parameter controls the distribution through a tilt involving a canonical statistic. The mean of that statistic is often a more directly interpretable quantity, but translating between the two parameterizations requires knowing how the mean changes as the natural parameter changes. The key fact is precise: the derivative of the mean with respect to the natural parameter is the variance of the canonical statistic.

This identity gives more than a derivative formula. If the statistic is not almost surely constant, its variance is positive, so the mean map is strictly increasing. Consequently, any mean value in its attainable range corresponds to exactly one natural parameter. The range still matters: strict increase does not mean that every real number can occur as the mean.

One-Parameter Canonical Form

Definition: A one-parameter canonical exponential family, relative to a fixed measure $\nu$, has densities or mass functions $$ f_\eta(x)=h(x)\exp\{\eta T(x)-A(\eta)\}, $$ where $h(x)\geq 0$ is the carrier function, $T(x)$ is the canonical statistic, $\eta$ is the natural parameter, and $$ A(\eta)=\log\int h(x)e^{\eta T(x)}\,d\nu(x) $$ is the log-partition function. The natural parameter space is the set of $\eta$ for which this integral is finite. For the results below, work on an open interval $I$ contained in that space.

The open-interval condition ensures that each $\eta\in I$ has natural parameter values on both sides within the domain. This local room is what lets us justify differentiating the integral. We assume $A(\eta)$ is finite for every $\eta\in I$; no fixed-support family with parameter-dependent carrier or measure is being considered here.

The Derivative Identity

Theorem: Suppose $I$ is an open interval on which the normalizing integral defining $A$ is finite. For every $\eta\in I$, $$ \frac{d}{d\eta}E_\eta[T(X)] =\frac{d^2}{d\eta^2}A(\eta) =\operatorname{Var}_\eta(T(X)). $$ In particular, $E_\eta[T(X)]=A'(\eta)$.

Proof. Fix $\eta\in I$, and write $$ Z(\eta)=\int h(x)e^{\eta T(x)}\,d\nu(x), \qquad A(\eta)=\log Z(\eta). $$ Choose $\epsilon>0$ small enough that $\eta-2\epsilon$ and $\eta+2\epsilon$ belong to $I$. For $u$ sufficiently close to $\eta$, the integrands for the first two derivatives of $Z(u)$ are $h(x)T(x)e^{uT(x)}$ and $h(x)T(x)^2e^{uT(x)}$.

These integrands admit an integrable bound independent of such $u$. For $T(x)\geq 0$, polynomial factors in $T(x)$ can be absorbed into an exponential with a little extra rate: for $k=0,1,2$, there is a finite constant $C_k$ such that $$ |T(x)|^k e^{uT(x)} \leq C_k e^{(\eta+2\epsilon)T(x)}. $$ For $T(x)<0$, the corresponding bound is $$ |T(x)|^k e^{uT(x)} \leq C_k e^{(\eta-2\epsilon)T(x)}. $$ The integrals of these bounds after multiplication by $h(x)$ are finite by the choice of endpoints. The dominated convergence theorem therefore permits differentiating under the integral twice. It follows that $$ Z'(\eta)=\int h(x)T(x)e^{\eta T(x)}\,d\nu(x), \qquad Z''(\eta)=\int h(x)T(x)^2e^{\eta T(x)}\,d\nu(x). $$ Since $f_\eta(x)=h(x)e^{\eta T(x)}/Z(\eta)$, these equations give $$ \frac{Z'(\eta)}{Z(\eta)}=E_\eta[T(X)], \qquad \frac{Z''(\eta)}{Z(\eta)}=E_\eta[T(X)^2]. $$ Differentiating $A=\log Z$ yields $$ A'(\eta)=\frac{Z'(\eta)}{Z(\eta)}=E_\eta[T(X)] $$ and $$ A''(\eta) =\frac{Z''(\eta)}{Z(\eta)} -\left(\frac{Z'(\eta)}{Z(\eta)}\right)^2 =E_\eta[T(X)^2]-\{E_\eta[T(X)]\}^2 =\operatorname{Var}_\eta(T(X)). $$ As the mean is $A'(\eta)$, its derivative is $A''(\eta)$, proving the identity. $\square$

Monotonicity and the Mean Range

The variance is nonnegative, so the theorem first implies that the mean map cannot decrease. It is strictly increasing when the variance is positive. In the canonical form above, every distribution in the family has the same sets of probability zero as the measure with density proportional to $h$: the exponential tilt is strictly positive wherever $h$ is positive. Thus if $T$ is not constant almost everywhere on the carrier's support, it is not constant almost surely under any $P_\eta$, and its variance is positive at every $\eta\in I$.

Consequence: If $T$ is not almost surely constant, the mean map $$ m(\eta)=E_\eta[T(X)]=A'(\eta) $$ is continuous and strictly increasing on $I$. Its image $m(I)$ is an interval, and each value in $m(I)$ is attained at exactly one natural parameter. Values outside $m(I)$ are not attainable by any $\eta\in I$.

To describe the interval $m(I)$, let $I=(a,b)$, allowing infinite endpoints. Because $m$ is increasing, its endpoint limits $$ m_-=\lim_{\eta\downarrow a}m(\eta), \qquad m_+=\lim_{\eta\uparrow b}m(\eta) $$ exist as extended real numbers. For an open interval $I$, neither finite endpoint limit is attained at an interior $\eta$, so $m(I)=(m_-,m_+)$. This endpoint check is essential: a strictly increasing mean map may have a narrow range, even if $\eta$ itself ranges over all real numbers.

Worked Examples

Worked Example: Bernoulli Mean and the Logit Parameter

Let $X$ have a Bernoulli distribution with success probability $p\in(0,1)$. Its mass function can be written $$ P_\eta(X=x)=\exp\{x\eta-A(\eta)\},\qquad x\in\{0,1\}, $$ where $\eta=\log\{p/(1-p)\}$ and $A(\eta)=\log(1+e^\eta)$. Here $T(X)=X$.

Differentiate the log-partition function: $$ A'(\eta)=\frac{e^\eta}{1+e^\eta}=p. $$ A second derivative gives $$ A''(\eta)=\frac{e^\eta}{(1+e^\eta)^2}=p(1-p)=\operatorname{Var}_\eta(X). $$ For a target mean $m\in(0,1)$, set $m=e^\eta/(1+e^\eta)$. Then $m(1+e^\eta)=e^\eta$, so $m=e^\eta(1-m)$ and $$ \eta=\log\left(\frac{m}{1-m}\right). $$ The Bernoulli mean map is strictly increasing and its range is $(0,1)$, not all of $\mathbb R$.

Worked Example: Poisson Mean and Log-Rate

For $X\sim\operatorname{Poisson}(\lambda)$, with $\lambda>0$, $$ P(X=x)=\frac{1}{x!}\exp\{x\log\lambda-\lambda\},\qquad x=0,1,2,\ldots. $$ Thus $T(X)=X$, $\eta=\log\lambda$, $h(x)=1/x!$, and $A(\eta)=e^\eta$. Therefore $$ E_\eta[X]=A'(\eta)=e^\eta=\lambda, \qquad \operatorname{Var}_\eta(X)=A''(\eta)=e^\eta=\lambda. $$ Given a target mean $m>0$, the equation $m=e^\eta$ has the unique solution $$ \eta=\log m. $$ The attainable mean range is $(0,\infty)$; a zero mean would require a limiting rate, not a finite natural parameter.

Worked Example: Normal Mean with Known Variance

Suppose $X\sim N(\mu,\sigma^2)$, where $\sigma^2>0$ is fixed and known. Completing the square in the normal density gives the canonical form with $$ T(X)=X,\qquad \eta=\frac{\mu}{\sigma^2},\qquad A(\eta)=\frac{\sigma^2\eta^2}{2}. $$ Indeed, $\eta x-A(\eta)=\mu x/\sigma^2-\mu^2/(2\sigma^2)$, the part of the log-density involving $x$ and $\mu$. Hence $$ E_\eta[X]=A'(\eta)=\sigma^2\eta=\mu, \qquad \operatorname{Var}_\eta(X)=A''(\eta)=\sigma^2. $$ For a requested mean $m\in\mathbb R$, the unique natural parameter is $$ \eta=\frac{m}{\sigma^2}. $$ Unlike the Bernoulli and Poisson examples, the natural parameter and mean both range over all real numbers.

Using the Identity to Solve for a Natural Parameter

For a nonconstant canonical statistic, a target mean $m$ determines at most one $\eta$. To determine whether it does determine one, first check that $m$ lies between the endpoint limits of $A'(\eta)$. If it does, monotonicity guarantees a unique solution to $A'(\eta)=m$. In many familiar families, algebra gives the inverse directly, as in the examples above.

When no explicit inverse is available, the identity also supplies the slope needed for a numerical method. A Newton update for solving $A'(\eta)=m$ is $$ \eta_{\text{new}} =\eta-\frac{A'(\eta)-m}{A''(\eta)} =\eta-\frac{E_\eta[T(X)]-m}{\operatorname{Var}_\eta(T(X))}. $$ This update is defined when the variance is positive. Strict monotonicity ensures there is at most one solution, but does not by itself guarantee that an unrestricted Newton iteration converges from every starting value. A bracketed method, which keeps candidate values on either side of the target mean, can use monotonicity more robustly.

Common Pitfall

A common mistake is to say that the mean map is always strictly increasing merely because its derivative is a variance. A variance can equal zero: if $T(X)$ is almost surely constant, then $A''(\eta)=0$ and the mean is constant, not strictly increasing. The correction is to verify that the canonical statistic is nonconstant under the family. A second mistake is to invert the map without checking its range; for instance, the Bernoulli mean cannot equal $1.2$, even though the logit formula might tempt someone to treat every real target as admissible.

Key takeaway: On an open natural-parameter interval where the normalizer is finite, differentiating the log-partition function gives $A'(\eta)=E_\eta[T(X)]$ and $A''(\eta)=\operatorname{Var}_\eta(T(X))$. Positive variance makes the mean map strictly increasing, so every attainable mean identifies exactly one natural parameter.

Check Your Understanding

Use the derivative identity and the parameter ranges, not just algebraic inversion.

In a canonical family, what regularity condition on the natural parameter interval supports differentiating the normalizing integral twice?
Why does a positive variance of $T(X)$ imply that $E_\eta[T(X)]$ is strictly increasing?
For a Bernoulli family, what target means are attainable by finite natural parameters?
For the Poisson canonical form, solve for $\eta$ when the target mean is $4$, and state the variance at that parameter.
What additional condition is needed to conclude strict increase, rather than merely nondecrease, from the variance identity?