Normal Families With Unknown Mean and Variance

Why the Normal Density Uses Two Statistics

A normal model with both mean and variance unknown has two parameters, but its density does not place them symmetrically in the exponent. Expanding the squared deviation reveals the useful structure: the observation appears through $x$ and $x^2$, with one coefficient for each. Writing the density this way makes it possible to read off the natural parameters and use the log-partition function to recover moments.

Throughout, $X\sim N(\mu,\sigma^2)$, with $\sigma^2>0$, and the carrier measure is Lebesgue measure on $\mathbb{R}$. The focus is the family in which both $\mu$ and $\sigma^2$ may vary. The canonical statistics are $T_1(x)=x$ and $T_2(x)=x^2$.

Definition: A canonical representation of this normal family is a density of the form $f(x;\eta)=\exp\{\eta_1x+\eta_2x^2-A(\eta)\}$, relative to Lebesgue measure, where $\eta=(\eta_1,\eta_2)$ belongs to the set of parameters for which the density can be normalized. The function $A$ is the log-partition function.

Deriving the Canonical Parameters

Start with the normal density and expand its exponent. The terms depending on $x$ become a linear term and a quadratic term; the remaining terms do not depend on $x$ and contribute to the normalizing constant.

\begin{aligned} f(x;\mu,\sigma^2) &=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(x-\mu)^2}{2\sigma^2}\right\}\\ &=\exp\left\{ \frac{\mu}{\sigma^2}x -\frac{1}{2\sigma^2}x^2 -\frac{\mu^2}{2\sigma^2} -\frac{1}{2}\log(2\pi\sigma^2) \right\}. \end{aligned}

Thus the canonical parameters are $\eta_1=\mu/\sigma^2$ and $\eta_2=-1/(2\sigma^2)$. In particular, the coefficient of $x^2$ is negative whenever the variance is positive. Solving for the original parameters gives $\sigma^2=-1/(2\eta_2)$ and $\mu=-\eta_1/(2\eta_2)$, provided $\eta_2<0$.

Worked Example: Converting a Normal Model to Canonical Form

Let $X\sim N(3,4)$, where the second normal parameter is the variance. Find its canonical parameters and log-partition value.

The variance is $\sigma^2=4$, so $\eta_1=\mu/\sigma^2=3/4$, and $\eta_2=-1/(2\sigma^2)=-1/8$. The constant in the exponent is $\mu^2/(2\sigma^2)+\frac12\log(2\pi\sigma^2) =9/8+\frac12\log(8\pi)$. Therefore the density is $\exp\{(3/4)x-(1/8)x^2-A(3/4,-1/8)\}$, with $A(3/4,-1/8)=9/8+\frac12\log(8\pi)$.

As a direct check, substituting the parameters into the inverse formulas gives $-1/(2\eta_2)=-1/[2(-1/8)]=4$ and $-\eta_1/(2\eta_2)=-(3/4)/(-1/4)=3$, as required.

Natural Parameter Space and Log-Partition Function

For a proposed pair $(\eta_1,\eta_2)$, normalization requires the kernel $\exp(\eta_1x+\eta_2x^2)$ to have a finite integral over the real line. This condition determines the natural parameter space; it is not an extra convention imposed after choosing the parameters.

Theorem: Relative to Lebesgue measure on $\mathbb{R}$, the normal-family canonical kernel is integrable exactly when $\eta_2<0$, with $\eta_1$ unrestricted. The natural parameter space and log-partition function are $$ \mathcal{H}=\{(\eta_1,\eta_2)\in\mathbb{R}^2:\eta_2<0\}, \qquad A(\eta_1,\eta_2)=-\frac{\eta_1^2}{4\eta_2} +\frac12\log\left(\frac{-\pi}{\eta_2}\right). $$

To prove the claim, first suppose $\eta_2<0$. Completing the square yields

\eta_2x^2+\eta_1x =\eta_2\left(x+\frac{\eta_1}{2\eta_2}\right)^2 -\frac{\eta_1^2}{4\eta_2}.

Since $-\eta_2>0$, the first term is a negative quadratic in the shifted variable. The Gaussian integral is finite, and its value is

\begin{aligned} \int_{\mathbb{R}} e^{\eta_1x+\eta_2x^2}\,dx &=e^{-\eta_1^2/(4\eta_2)} \int_{\mathbb{R}} \exp\left\{\eta_2 \left(x+\frac{\eta_1}{2\eta_2}\right)^2\right\}\,dx\\ &=e^{-\eta_1^2/(4\eta_2)} \sqrt{\frac{\pi}{-\eta_2}}. \end{aligned}

Taking the logarithm gives the displayed formula for $A$. If $\eta_2>0$, the quadratic term grows toward $+\infty$ in both tails, so the integral diverges. If $\eta_2=0$ and $\eta_1=0$, the integrand is identically one and its integral is infinite. If $\eta_2=0$ and $\eta_1>0$, it grows without bound as $x\to+\infty$; if $\eta_1<0$, it grows without bound as $x\to-\infty$. The integral diverges in each case. This proves both the exact parameter space and the stated log-partition function. $\square$

Worked Example: Checking Integrability of Candidate Parameters

Consider $(\eta_1,\eta_2)=(2,-1/4)$. Because $\eta_2=-1/4<0$, the kernel is integrable. Its log-partition function is

A(2,-1/4) =-\frac{2^2}{4(-1/4)} +\frac12\log\left(\frac{-\pi}{-1/4}\right) =4+\frac12\log(4\pi).

By contrast, $(2,1/4)$ is not a valid natural parameter: its quadratic exponent is positive and the integral over $\mathbb{R}$ diverges. Nor is $(2,0)$ valid; its kernel is $e^{2x}$, which has an infinite integral on the positive half-line.

For the valid pair $(2,-1/4)$, the inverse formulas give $\sigma^2=-1/[2(-1/4)]=2$ and $\mu=-2/[2(-1/4)]=4$. Thus this canonical density represents $N(4,2)$.

Recovering the Moments

The log-partition derivatives can be calculated directly here. Their connection to moments also follows by differentiating the normalizing integral: on any closed bounded neighborhood of a parameter point in $\mathcal{H}$, the quadratic coefficient stays bounded above by a negative constant. The integrands obtained by differentiating are therefore bounded by an integrable Gaussian tail times a polynomial in $|x|$, which justifies differentiation under the integral.

Theorem: For $(\eta_1,\eta_2)\in\mathcal{H}$, the first derivatives of the log-partition function are the first and second raw moments: $$ \frac{\partial A}{\partial\eta_1}=E[X] =-\frac{\eta_1}{2\eta_2}, \qquad \frac{\partial A}{\partial\eta_2}=E[X^2] =\frac{\eta_1^2}{4\eta_2^2}-\frac{1}{2\eta_2}. $$ Consequently, $$ \operatorname{Var}(X)=E[X^2]-(E[X])^2 =-\frac{1}{2\eta_2}. $$

For a direct justification of the moment identities, write $Z(\eta)=\int_{\mathbb{R}}e^{\eta_1x+\eta_2x^2}\,dx=e^{A(\eta)}$. The integrability bound above permits differentiating $Z$ under the integral sign, giving $\partial Z/\partial\eta_1=\int x e^{\eta_1x+\eta_2x^2}\,dx$ and $\partial Z/\partial\eta_2=\int x^2e^{\eta_1x+\eta_2x^2}\,dx$. Dividing each identity by $Z$ yields $A_{\eta_1}=E[X]$ and $A_{\eta_2}=E[X^2]$, since $\partial A/\partial\eta_j=(\partial Z/\partial\eta_j)/Z$.

Now differentiate the explicit expression for $A$. For $\eta_1$, $\partial[-\eta_1^2/(4\eta_2)]/\partial\eta_1=-\eta_1/(2\eta_2)$, while the logarithmic term does not depend on $\eta_1$. For $\eta_2$, the derivative of the first term is $\eta_1^2/(4\eta_2^2)$, and the derivative of $\frac12\log(-\pi/\eta_2)$ is $-1/(2\eta_2)$. This gives the two raw moments in the theorem. Finally, \[ (E[X])^2=\frac{\eta_1^2}{4\eta_2^2}, \] so subtracting it from $E[X^2]$ leaves $-1/(2\eta_2)$, the stated variance. $\square$

Worked Example: Recovering Moments From Canonical Parameters

Use $(\eta_1,\eta_2)=(2,-1/4)$ to recover the first two raw moments and the variance.

The first derivative gives $E[X]=-\eta_1/(2\eta_2)=-2/[2(-1/4)]=4$. The second derivative gives

E[X^2] =\frac{2^2}{4(-1/4)^2}-\frac{1}{2(-1/4)} =\frac{4}{1/4}+2 =16+2=18.

Thus $\operatorname{Var}(X)=18-4^2=2$, which agrees with $-1/(2\eta_2)=-1/[2(-1/4)]=2$. These are the moments of $N(4,2)$, as also obtained by inverting the canonical parameters.

What the Canonical Representation Reveals

The natural parameters encode the mean and variance jointly rather than one at a time. The negative quadratic coefficient controls the spread: as $\eta_2$ approaches zero from below, the variance $-1/(2\eta_2)$ increases. The linear coefficient shifts the center relative to that spread, since $\mu=-\eta_1/(2\eta_2)$. Both parameters matter when interpreting either canonical coefficient in isolation.

The log-partition function does more than supply a normalizing constant. Its first derivatives give the raw moments corresponding to the canonical statistics $X$ and $X^2$. For example, the second derivative with respect to $\eta_1$ is $-1/(2\eta_2)=\operatorname{Var}(X)$. Thus the moment calculations are consistent with the general log-partition derivative results in the earlier material on exponential families, while the explicit formulas here follow from the normal density itself.

Common Pitfall

Common mistake: Treating every real pair $(\eta_1,\eta_2)$ as a valid normal canonical parameter, or assuming that $\eta_2$ may be positive because $\eta_1$ can have either sign. The correction is to check integrability of the full kernel over $\mathbb{R}$: its quadratic coefficient must satisfy $\eta_2<0$. A linear term cannot counteract a positive quadratic tail, and when the quadratic coefficient is zero, no linear exponential is integrable over both tails.

When converting back to the familiar parameters, keep the sign visible: $\sigma^2=-1/(2\eta_2)>0$. Dropping the minus sign would imply a negative variance and would signal that the proposed canonical parameter was outside the natural parameter space.

Check Your Understanding

Use the canonical form and derivations above to answer these questions.

For $X\sim N(-2,9)$, calculate $\eta_1$ and $\eta_2$, and state the value of the canonical statistic coefficient multiplying $x^2$.
Why does the kernel fail to be integrable on $\mathbb{R}$ when $\eta_2=0$, both when $\eta_1=0$ and when $\eta_1\ne0$?
For $(\eta_1,\eta_2)=(1,-1/2)$, calculate $E[X]$, $E[X^2]$, and $\operatorname{Var}(X)$.
Explain why $\eta_2$ must be negative even when $\eta_1$ is also negative.
Starting from the first derivatives of $A$, show how subtracting $(E[X])^2$ from $E[X^2]$ produces the variance formula.