Normal Mean and Precision as a Two-Parameter Family

Why Two Parameters Change the Representation

When the variance of a normal response is fixed, the density can be written as a one-parameter exponential family in its mean. If both the mean and variance are unknown, the density contains two distinct data-dependent terms: one involving $y$, and another involving $y^2$. The corresponding exponential-family representation therefore has two statistics and two natural parameters.

This form makes the roles of the mean and precision explicit, but the canonical parameters are not the mean and precision themselves. In particular, the coefficient of $y^2$ must be negative for the density to be integrable over the real line. We derive the representation and establish its exact parameter region below.

Definition: Write a normal random variable with unknown mean $\mu\in\mathbb{R}$ and unknown precision $\tau>0$ as $Y\sim N(\mu,1/\tau)$. Here precision is the reciprocal of the variance. Its density with respect to Lebesgue measure on $\mathbb{R}$ is $$ f(y;\mu,\tau)=\left(\frac{\tau}{2\pi}\right)^{1/2} \exp\left\{-\frac{\tau}{2}(y-\mu)^2\right\}. $$ A two-parameter exponential-family representation has the form $$ f(y;\eta)=h(y)\exp\left\{\eta_1T_1(y)+\eta_2T_2(y)-A(\eta)\right\}, $$ where $T_1,T_2$ are statistics, $\eta_1,\eta_2$ are natural parameters, $h$ is a parameter-free carrier term, and $A$ is the log-partition function.

Deriving the Canonical Form

Expand the square in the normal density’s exponent:

-\frac{\tau}{2}(y-\mu)^2 =-\frac{\tau}{2}y^2+\tau\mu y-\frac{\tau\mu^2}{2}.

The remaining factor $(\tau/(2\pi))^{1/2}$ does not depend on $y$, so its logarithm can be combined with the last term in this expansion. The coefficient of $y$ is $\tau\mu$, while the coefficient of $y^2$ is $-\tau/2$. Thus the statistics are $T_1(y)=y$ and $T_2(y)=y^2$, with natural parameters

\eta_1=\tau\mu,\qquad \eta_2=-\frac{\tau}{2}.

Writing the density in terms of these quantities gives

f(y;\eta_1,\eta_2) =\exp\left\{\eta_1y+\eta_2y^2-A(\eta_1,\eta_2)\right\},

where the carrier term is $h(y)=1$ with respect to Lebesgue measure. To obtain $A$, gather the terms that do not involve $y$:

A(\eta_1,\eta_2) =\frac{\tau\mu^2}{2}+\frac{1}{2}\log(2\pi)-\frac{1}{2}\log\tau =\frac{\eta_1^2}{-4\eta_2} +\frac{1}{2}\log(2\pi)-\frac{1}{2}\log(-2\eta_2).

The last expression uses $\tau=-2\eta_2$ and $\mu=\eta_1/\tau$. It is defined for $\eta_2<0$. We now show both why that condition is necessary and why every pair satisfying it gives a normal density.

Theorem: The natural-parameter region for this family, relative to Lebesgue measure on $\mathbb{R}$, is $$ \mathcal{H}=\{(\eta_1,\eta_2)\in\mathbb{R}^2:\eta_2<0\}. $$ For $(\eta_1,\eta_2)\in\mathcal{H}$, the log-partition function is $$ A(\eta_1,\eta_2) =-\frac{\eta_1^2}{4\eta_2} +\frac{1}{2}\log\left(\frac{\pi}{-\eta_2}\right). $$ The resulting density is normal with mean $\mu=-\eta_1/(2\eta_2)$ and precision $\tau=-2\eta_2$.

Proof. The normalizing integral for the unnormalized kernel is

I(\eta_1,\eta_2) =\int_{\mathbb{R}}\exp(\eta_1y+\eta_2y^2)\,dy.

First suppose $\eta_2<0$. Completing the square yields

\eta_2y^2+\eta_1y =\eta_2\left(y+\frac{\eta_1}{2\eta_2}\right)^2 -\frac{\eta_1^2}{4\eta_2}.

The second term on the right is constant in $y$. A translation of the integration variable, followed by the Gaussian integral, gives

I(\eta_1,\eta_2) =\exp\left(-\frac{\eta_1^2}{4\eta_2}\right) \int_{\mathbb{R}}\exp\left\{\eta_2\left(y+\frac{\eta_1}{2\eta_2}\right)^2\right\}\,dy =\exp\left(-\frac{\eta_1^2}{4\eta_2}\right) \sqrt{\frac{\pi}{-\eta_2}}.

This integral is finite, and taking its logarithm gives the stated $A$. If $\eta_2=0$, the integrand is $\exp(\eta_1y)$. For $\eta_1>0$ its integral diverges on the positive half-line; for $\eta_1<0$ it diverges on the negative half-line; and for $\eta_1=0$ it is identically one and has infinite integral. If $\eta_2>0$, then for sufficiently large $|y|$, $\eta_2y^2+\eta_1y\geq \eta_2y^2/2$, so the integrand is at least $\exp(\eta_2y^2/2)$ in the tails and its integral diverges. Therefore finiteness holds exactly when $\eta_2<0$.

For such a pair, set $\tau=-2\eta_2>0$ and $\mu=-\eta_1/(2\eta_2)$. Completing the square in the normalized density gives a kernel proportional to $\exp\{-\tau(y-\mu)^2/2\}$. The normalizing integral above supplies precisely the constant needed for $N(\mu,1/\tau)$. Hence every point of $\mathcal{H}$ corresponds to a normal distribution in the stated family, and the formulas recover $\eta_1=\tau\mu$ and $\eta_2=-\tau/2$. $\square$

Translating Between Parameterizations

The transformation is one-to-one: every usual parameter pair $(\mu,\tau)$ with $\mu\in\mathbb{R}$ and $\tau>0$ maps to exactly one canonical pair, and every canonical pair in $\mathcal{H}$ maps back to exactly one usual pair. The second natural parameter is always negative; it is not itself the precision. The precision is $-2\eta_2$.

Worked Example: From Mean and Variance to Natural Parameters

Let $Y\sim N(2,4)$, where the second argument is the variance. Find the precision, natural parameters, and log-partition function, then evaluate the density at $y=1$ in both parameterizations.

The precision is $\tau=1/4$. Therefore

\eta_1=\tau\mu=\frac14(2)=\frac12,\qquad \eta_2=-\frac{\tau}{2}=-\frac18.

The log-partition function, using the expression in $(\mu,\tau)$, is

A=\frac{\tau\mu^2}{2}+\frac12\log(2\pi)-\frac12\log\tau =\frac{(1/4)(4)}{2}+\frac12\log(2\pi)-\frac12\log(1/4) =\frac12+\frac12\log(2\pi)+\log 2.

Numerically, $A\approx2.1121$. The canonical log density at $y=1$ is

\eta_1y+\eta_2y^2-A =\frac12-\frac18-2.1121 \approx-1.7371.

Directly from the normal density, the log density is

\frac12\log\left(\frac{1/4}{2\pi}\right) -\frac{1/4}{2}(1-2)^2 =-\frac12\log(8\pi)-\frac18 \approx-1.7371.

The two calculations agree, and the natural parameter pair lies in the admissible region because $\eta_2=-1/8<0$.

Worked Example: Recovering Mean and Precision From Natural Parameters

Suppose $\eta_1=-3$ and $\eta_2=-3/2$. Identify the normal distribution and compute its log-partition function.

Since $\eta_2<0$, this pair is admissible. The precision and mean are

\tau=-2\eta_2=-2\left(-\frac32\right)=3,\qquad \mu=-\frac{\eta_1}{2\eta_2} =-\frac{-3}{2(-3/2)}=-1.

Thus the distribution is $N(-1,1/3)$. The log-partition function is

A=-\frac{(-3)^2}{4(-3/2)} +\frac12\log\left(\frac{\pi}{3/2}\right) =\frac32+\frac12\log\left(\frac{2\pi}{3}\right).

The positive value of $A$ is not a problem: it is the logarithm of the integral of the unnormalized kernel, not the logarithm of a probability.

Independent Observations and the Two Statistics

For independent observations $Y_1,\ldots,Y_n$ from the same $N(\mu,1/\tau)$ distribution, multiply their densities. The resulting joint density has the same natural parameters for every observation, and its data-dependent terms combine into $\sum_{i=1}^n Y_i$ and $\sum_{i=1}^n Y_i^2$:

f(y_1,\ldots,y_n;\eta_1,\eta_2) =\exp\left\{ \eta_1\sum_{i=1}^n y_i+ \eta_2\sum_{i=1}^n y_i^2- nA(\eta_1,\eta_2) \right\}.

This aggregation is the sample-level counterpart of the single-response statistics $y$ and $y^2$. It also shows why retaining only the sum of observations would not preserve both data-dependent terms when precision is unknown.

Worked Example: Writing a Three-Observation Likelihood

Take observations $y_1=0$, $y_2=1$, and $y_3=2$, with $\mu=1$ and $\tau=2$. Compute the log likelihood using the canonical representation and check it directly.

The sample statistics are

\sum_{i=1}^3 y_i=0+1+2=3,\qquad \sum_{i=1}^3 y_i^2=0^2+1^2+2^2=5.

The natural parameters are $\eta_1=\tau\mu=2$ and $\eta_2=-\tau/2=-1$. Also,

A=\frac{\tau\mu^2}{2}+\frac12\log(2\pi)-\frac12\log\tau =1+\frac12\log\pi.

The canonical log likelihood is therefore

\ell=2(3)-1(5)-3\left(1+\frac12\log\pi\right) =-2-\frac32\log\pi \approx-3.7171.

For a direct check, each observation has density $(1/\sqrt{\pi})\exp\{-(y_i-1)^2\}$. The squared deviations sum to $1+0+1=2$, so the joint log density is

-\frac32\log\pi-2=-2-\frac32\log\pi,

which matches the canonical calculation.

Common Pitfall and Why the Form Matters

A common mistake is to identify $\eta_1$ with the mean and $\eta_2$ with the precision. That is incorrect: $\eta_1=\tau\mu$, while $\eta_2=-\tau/2$. For example, in the first worked example the mean is $2$, but $\eta_1=1/2$; the variance is $4$, while $\eta_2=-1/8$. Always recover the usual parameters using $\tau=-2\eta_2$ and $\mu=\eta_1/\tau$.

A second error is to allow $\eta_2=0$ because the linear term $\eta_1y$ still looks like an exponential-family term. On the full real line, no choice of $\eta_1$ makes $\exp(\eta_1y)$ integrable. The negative quadratic coefficient is essential: it controls the tails and makes a probability density possible.

The two-statistic form matters whenever both location and spread are modeled as unknown. It keeps their contributions separate in the likelihood and makes the admissible parameter region visible. The key structure is $y$ paired with $\tau\mu$, and $y^2$ paired with $-\tau/2$, with the log-partition function supplying the normalization.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

For $Y\sim N(3,9)$, find $\tau$, $\eta_1$, and $\eta_2$.
Given $\eta_1=4$ and $\eta_2=-2$, recover the mean and variance.
Explain why $(\eta_1,\eta_2)=(0,0)$ cannot define a normalized density on the real line.
For independent observations, which two sums appear in the canonical likelihood when both mean and precision are unknown?
Why is the second natural parameter necessarily negative?