Natural Parameters And Mean Parameters

Two Parameter Scales for the Same Response Model

An exponential-family model can describe a response using a parameter that makes its probability mass or density take a convenient form. That parameter is the natural parameter, denoted by $\theta$. For interpretation, however, it is often more direct to describe the model by the expected response, denoted by $\mu$. These are two ways of indexing the same family of distributions, but they are not generally the same number.

The log-partition function links the two scales. In a one-parameter exponential-dispersion representation, its derivative gives the mean parameter: $\mu=b'(\theta)$, subject to the regularity conditions stated below. The map depends on the chosen representation and on which quantity is treated as the response. The examples make these distinctions concrete.

Definition: For a one-parameter exponential-dispersion family written as $$ f(y;\theta,\phi)=\exp\left\{\frac{y\theta-b(\theta)}{a(\phi)}+c(y,\phi)\right\}, $$ $\theta$ is the natural parameter, $b(\theta)$ is the log-partition function, and $a(\phi)>0$ is the dispersion scale. The mean parameter is $\mu=E[Y]$. When the natural-parameter domain is an open interval, the normalizing integral is finite there, and differentiation under that integral is valid locally, the standard identity is $$ \mu=b'(\theta). $$

The displayed form uses $Y$ as the canonical statistic, as is appropriate for the one-parameter response models considered here. In a different representation, the statistic or its scale may be changed; the numerical value called $\theta$ can then change as well. The mean $E[Y]$, by contrast, refers to the response itself and retains its usual interpretation.

Worked Examples of the Natural-to-Mean Map

Worked Example: Bernoulli Response

Let $Y$ be Bernoulli with success probability $p$, where $0<p<1$. Its probability mass function is $P(Y=y)=p^y(1-p)^{1-y}$ for $y\in\{0,1\}$. Rewriting its logarithm gives $$ \log P(Y=y)=y\log\left(\frac{p}{1-p}\right)+\log(1-p). $$

Set $\theta=\log\{p/(1-p)\}$. Since $p=e^\theta/(1+e^\theta)$, we have $\log(1-p)=-\log(1+e^\theta)$. Thus the canonical form is $$ P(Y=y)=\exp\{y\theta-b(\theta)\},\qquad b(\theta)=\log(1+e^\theta). $$

The mean of a Bernoulli variable is its success probability, so $$ \mu=p=\frac{e^\theta}{1+e^\theta}. $$ For $\theta=\log 3$, $e^\theta=3$, and therefore $\mu=3/(1+3)=0.75$. Conversely, for $p=0.2$, $\theta=\log(0.2/0.8)=\log(0.25)$, approximately $-1.386$. A positive mean below one can correspond to a negative natural parameter; the parameters are on different scales.

Worked Example: Poisson Response

Let $Y$ have a Poisson distribution with rate $\lambda>0$. Its mass function is $P(Y=y)=e^{-\lambda}\lambda^y/y!$ for nonnegative integers $y$. Taking logarithms and setting $\theta=\log\lambda$ yields $$ \log P(Y=y)=y\theta-e^\theta-\log(y!). $$

This is the exponential-family representation with $b(\theta)=e^\theta$ and $c(y)=-\log(y!)$. Because $E[Y]=\lambda$, the mean parameter is $$ \mu=\lambda=e^\theta. $$ If $\theta=\log(2.5)$, then $\lambda=e^{\log(2.5)}=2.5$, so the mean is 2.5 counts. The natural parameter can be any real number, while the mean is positive; for example, $\theta=0$ gives $\mu=1$, not a mean of zero.

Worked Example: Normal Response with Known Variance

Suppose $Y\sim N(m,\sigma^2)$, with known $\sigma^2>0$ and unknown location $m$. The log density, after collecting the terms that involve $m$, is $$ \log f(y)=\frac{ym}{\sigma^2}-\frac{m^2}{2\sigma^2}-\frac{y^2}{2\sigma^2}-\frac{1}{2}\log(2\pi\sigma^2). $$

Set $\theta=m/\sigma^2$, so $m=\sigma^2\theta$. The terms involving $\theta$ become $y\theta-\sigma^2\theta^2/2$. Hence $$ b(\theta)=\frac{\sigma^2\theta^2}{2},\qquad \mu=E[Y]=m=\sigma^2\theta. $$

For $\sigma^2=4$ and $\theta=0.5$, the corresponding normal mean is $m=4(0.5)=2$. The natural parameter is not the normal mean unless the variance happens to be one. This example also shows that the conversion can depend on a fixed model quantity, here $\sigma^2$.

Worked Example: Gamma Response with Fixed Shape

Let $Y$ have a gamma distribution with fixed shape $k>0$ and rate $r>0$, so its density is $r^k y^{k-1}e^{-ry}/\Gamma(k)$ for $y>0$. Take $\theta=-r$. Then $\theta<0$, and the density can be written $$ f(y)=\exp\left\{y\theta+k\log(-\theta)+(k-1)\log y-\log\Gamma(k)\right\}. $$

Comparing with the canonical form gives $b(\theta)=-k\log(-\theta)$, with the remaining terms assigned to the part not involving $\theta$. The gamma mean is $k/r$, and $r=-\theta$; therefore $$ \mu=\frac{k}{r}=-\frac{k}{\theta}. $$

For $k=3$ and $\theta=-2$, the rate is $r=2$ and the mean is $3/2=1.5$. The natural-parameter domain is restricted to negative values, because the rate must be positive. Unlike the Poisson map, this conversion is reciprocal rather than exponential.

What the Mapping Does—and Does Not—Say

In each example, the natural parameter is the coordinate appearing beside the response in the exponential-family expression, while the mean parameter is the expected value of the response. The log-partition function supplies the conversion: under the regularity conditions in the definition, differentiating $b$ gives the response mean. Its shape therefore determines how changes in $\theta$ correspond to changes in $\mu$.

The mean parameter is not always a probability. For a Bernoulli response, $\mu$ is a probability between zero and one. For a Poisson response, it is a positive expected count. For a normal response, it may be any real number. For the gamma example, it is positive. The range depends on the family and on the response being modeled.

The response definition matters even within a familiar family. If $Y$ is a binomial count from $m$ trials, its mean is $mp$, not $p$; if the response is the sample proportion $Y/m$, its mean is $p$. The natural parameter in the usual binomial count representation is $\log\{p/(1-p)\}$, but the log-partition function and mean depend on whether the modeled response is the count or the proportion. Always identify $Y$ before interpreting $\mu$.

Result: Bernoulli mean map and its inverse. For a Bernoulli response, $\mu=e^\theta/(1+e^\theta)$ maps every real $\theta$ to a value strictly between zero and one. Solving for $\theta$ gives $\theta=\log\{\mu/(1-\mu)\}$. Indeed, starting with the mean formula, multiply by $1+e^\theta$ to get $\mu+\mu e^\theta=e^\theta$. Thus $\mu=e^\theta(1-\mu)$, and $e^\theta=\mu/(1-\mu)$. Taking logarithms gives the stated inverse, which is defined for $0<\mu<1$.

Result: Poisson mean map and its inverse. For a Poisson response, $\mu=e^\theta$. Since the exponential of a real number is positive, every $\theta\in\mathbb{R}$ gives $\mu>0$. Conversely, for any $\mu>0$, taking $\theta=\log\mu$ gives $e^\theta=\mu$. Thus the map covers the positive means, with inverse $\theta=\log\mu$.

Common Mistake: Treating the Two Parameters as Synonyms

A frequent error is to read $\theta$ as the expected response simply because both quantities describe the same model. That is only true for particular parameterizations, not by definition. For example, in the normal model with known variance, $\theta=m/\sigma^2$, so using $\theta$ as the mean would give the wrong answer whenever $\sigma^2\ne1$. In the gamma example, $\theta=-r$ is negative while the mean $k/r$ is positive.

A related error is to identify the mean with a parameter from the original distribution without checking the response. A binomial success probability $p$ is the mean of a single Bernoulli trial, but the mean of a count over $m$ trials is $mp$. The correction in both cases is the same: write the chosen exponential-family representation, identify its response $Y$, and interpret $\mu$ as $E[Y]$.

Key takeaway: The natural parameter $\theta$ is the coordinate used in the exponential-family representation; the mean parameter $\mu$ is the expected response. For a regular one-parameter family in canonical form, the log-partition function links them by $\mu=b'(\theta)$. Always check the family, its parameter domain, and the definition of the response before interpreting either parameter.

Check Your Understanding

Use the response definition and the stated family to answer each question.

For a Bernoulli response with $\theta=0$, calculate the mean parameter.
A Poisson model has mean 4. What is its natural parameter?
For a normal response with known variance $\sigma^2=9$ and $\theta=1/3$, find the mean.
For a gamma model with fixed shape $k=2$ and $\theta=-4$, find the rate and the mean.
Explain why a binomial count from $m$ trials has mean $mp$, whereas a Bernoulli response has mean $p$.