Writing The Normal Distribution In Exponential Form

Why Write the Normal Distribution in Exponential Form?

A normal response with known variance is one of the basic distributions used in generalized linear modeling. Writing its density in exponential-family form makes the role of the mean precise: the mean is not itself the canonical natural parameter unless the variance equals one. The variance determines the conversion between the two.

This representation also identifies the cumulant function, from which the mean and variance can be recovered. The focus here is the normal model with a fixed, known variance. It does not cover the different modeling questions that arise when the variance is unknown or modeled as a parameter.

Definition: Suppose $Y\sim N(\mu,\sigma^2)$, where $\sigma^2>0$ is known and $\mu$ is the parameter. In canonical exponential-family form, the density is $$ f(y;\eta)=\exp\{y\eta-A(\eta)+c(y)\}, $$ where $\eta$ is the natural parameter, $A(\eta)$ is the cumulant function, and $c(y)$ contains terms that do not depend on $\eta$. For the known-variance normal model, $$ \eta=\frac{\mu}{\sigma^2},\qquad A(\eta)=\frac{\sigma^2\eta^2}{2},\qquad c(y)=-\frac{y^2}{2\sigma^2}-\frac12\log(2\pi\sigma^2). $$

Deriving the Canonical Form

Begin with the normal density and expand its squared term. This is the key algebraic step: it separates the part involving the observation $y$ and parameter $\mu$ from the part involving only $y$.

\begin{aligned} f(y;\mu) &=\frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{(y-\mu)^2}{2\sigma^2}\right\}\\ &=\exp\left\{ \frac{y\mu}{\sigma^2} -\frac{\mu^2}{2\sigma^2} -\frac{y^2}{2\sigma^2} -\frac12\log(2\pi\sigma^2) \right\}. \end{aligned}

Set $\eta=\mu/\sigma^2$, so $\mu=\sigma^2\eta$. Then $\mu^2/(2\sigma^2)=\sigma^2\eta^2/2$, and $y\mu/\sigma^2=y\eta$. These substitutions give the stated functions $A$ and $c$.

f(y;\eta)= \exp\left\{ y\eta-\frac{\sigma^2\eta^2}{2} -\frac{y^2}{2\sigma^2} -\frac12\log(2\pi\sigma^2) \right\}.

The normalizing role of $A$ can be checked directly. Combining the terms that depend on $y$ and $\eta$ gives

y\eta-\frac{\sigma^2\eta^2}{2}-\frac{y^2}{2\sigma^2} =-\frac{(y-\sigma^2\eta)^2}{2\sigma^2}.

Consequently, the density is a normal density centered at $\sigma^2\eta$, with variance $\sigma^2$. Its integral over the real line is one. This confirms that the quadratic term $A(\eta)=\sigma^2\eta^2/2$, together with the term $c(y)$, supplies the required normalization.

Mean and Variance from the Cumulant Function

For a density in the canonical form above, the cumulant function generates the mean and variance through its first two derivatives. Here is a direct justification for the normal model. For any real $t$, multiply the density by $e^{ty}$. The result is the same exponential-family expression with $\eta$ replaced by $\eta+t$, apart from a factor that does not depend on $y$. Integrating therefore gives the moment-generating function

E[e^{tY}] =\exp\{A(\eta+t)-A(\eta)\}.

The first derivative at $t=0$ gives the mean. The first derivative of the log moment-generating function gives the mean, and its second derivative gives the variance. Since $A(\eta)=\sigma^2\eta^2/2$, these derivatives are

A'(\eta)=\sigma^2\eta=\mu, \qquad A''(\eta)=\sigma^2.

Result: In canonical form for $Y\sim N(\mu,\sigma^2)$ with known $\sigma^2$, $E[Y]=A'(\eta)=\mu$ and $\operatorname{Var}(Y)=A''(\eta)=\sigma^2$. Equivalently, the variance function is constant: it does not depend on the mean.

A related exponential-family convention writes a density as $\exp\{[y\theta-b(\theta)]/a(\phi)+c(y,\phi)\}$. In that convention, one can take $\theta=\mu$, $b(\theta)=\theta^2/2$, $a(\phi)=\sigma^2$, and $c(y,\phi)=-y^2/(2\sigma^2)-\tfrac12\log(2\pi\sigma^2)$. This is consistent with the canonical form: $\eta=\theta/a(\phi)=\mu/\sigma^2$. When identifying the canonical natural parameter, retain the factor $1/\sigma^2$; the mean $\mu$ is not generally that parameter.

Worked Example: Writing a Normal Density and Evaluating It

A hypothetical measurement $Y$ follows $N(3,4)$, where the second parameter is the variance. Express its density in canonical exponential-family form, then calculate the density at $y=5$. Give the density to four decimal places.

Method. The variance is $\sigma^2=4$ and the mean is $\mu=3$. Thus the natural parameter and cumulant function are

\eta=\frac{3}{4}, \qquad A(\eta)=\frac{4\eta^2}{2}=2\eta^2.

The data-only term is $c(y)=-y^2/8-\tfrac12\log(8\pi)$. The density in canonical form is therefore

f(y)=\exp\left\{\frac34y-\frac98-\frac{y^2}{8} -\frac12\log(8\pi)\right\}.

At $y=5$, the exponent is

\frac34(5)-\frac98-\frac{25}{8}-\frac12\log(8\pi) =\frac{30-9-25}{8}-\frac12\log(8\pi) =-\frac12-\frac12\log(8\pi).

Hence $f(5)=e^{-1/2}/\sqrt{8\pi}\approx 0.1210$, rounded to four decimal places. As a check, the original normal density gives the same value because $5-3=2$ and $(5-3)^2/(2\cdot4)=1/2$.

Common mistake avoided: using $\eta=3$, the mean, instead of $\eta=3/4$. The density’s $y$-coefficient is $\mu/\sigma^2$, not $\mu$.

Likelihoods in Natural-Parameter Form

For independent observations $Y_1,\ldots,Y_n$ from the same known-variance normal model, multiply their densities, or equivalently add their log densities. The parameter-dependent terms combine through the sample sum. This makes clear why the sum of the observations is the part of the data that enters the likelihood through the canonical parameter.

\ell(\eta) =\eta\sum_{i=1}^{n}y_i-nA(\eta)+\sum_{i=1}^{n}c(y_i).

The final sum does not depend on $\eta$. Differentiating the log-likelihood gives $\sum_i y_i-n\sigma^2\eta$. Setting this derivative to zero yields $\widehat{\eta}=\bar{y}/\sigma^2$, or $\widehat{\mu}=\bar{y}$. This is a likelihood calculation for the specified known-variance model; it does not make the variance an estimated parameter.

Worked Example: Maximum Likelihood with Known Variance

Three independent hypothetical measurements follow $Y_i\sim N(\mu,4)$, with the same unknown mean and known variance $4$. The observed values are $1,2,4$. Find the maximum likelihood estimates of $\eta$ and $\mu$, and calculate the maximized log-likelihood to four decimal places.

Method. First compute the sample size and sums:

n=3,\qquad \sum_i y_i=1+2+4=7,\qquad \sum_i y_i^2=1+4+16=21.

For variance $4$, $\ell(\eta)=7\eta-3(2\eta^2)+\sum_i c(y_i)$. Differentiating and setting the result equal to zero gives

\frac{d\ell}{d\eta}=7-12\eta=0, \qquad \widehat{\eta}=\frac{7}{12}, \qquad \widehat{\mu}=4\widehat{\eta}=\frac73.

The parameter-dependent part of the log-likelihood at this estimate is

7\left(\frac{7}{12}\right)-3\left(\frac{4(7/12)^2}{2}\right) =\frac{49}{12}-\frac{49}{24} =\frac{49}{24}.

The data-only terms sum to $-21/8-\tfrac32\log(8\pi)$. Therefore

\ell(\widehat{\eta}) =\frac{49}{24}-\frac{21}{8}-\frac32\log(8\pi) =-\frac{7}{12}-\frac32\log(8\pi) \approx -5.4196.

As a check, the fitted mean is $7/3$, and the squared deviations sum to $(1-7/3)^2+(2-7/3)^2+(4-7/3)^2=14/3$. The original normal log-likelihood is $-\tfrac32\log(8\pi)-(14/3)/8$, which is the same expression, $-\tfrac32\log(8\pi)-7/12$.

Common mistake avoided: treating $7/12$ as the estimated mean. It is the estimated natural parameter; multiplying by the known variance gives $\widehat{\mu}=7/3$.

Likelihood Ratios and the Parameterization

When comparing two proposed means under the same known variance, the data-only term $c(y)$ cancels in a likelihood ratio. The natural-parameter form then provides a short calculation. The result can also be checked using the squared deviations in the original normal density.

Worked Example: Comparing Two Proposed Means

A hypothetical observation is $y=4$. Compare its likelihood under $N(3,9)$ with its likelihood under $N(1,9)$, where the variance is known to be $9$ in both cases. Find $L(3)/L(1)$, rounded to four decimal places.

Method. The natural parameters are $\eta_3=3/9=1/3$ and $\eta_1=1/9$. Their cumulant values are

A(\eta_3)=\frac92\left(\frac13\right)^2=\frac12, \qquad A(\eta_1)=\frac92\left(\frac19\right)^2=\frac1{18}.

Subtracting the log-likelihoods cancels $c(4)$:

\log\frac{L(3)}{L(1)} =4\left(\frac13-\frac19\right)-\left(\frac12-\frac1{18}\right) =\frac89-\frac49 =\frac49.

Thus $L(3)/L(1)=e^{4/9}\approx 1.5596$, rounded to four decimal places. As an independent check, the squared deviations from the two means are $1$ and $9$. The log likelihood ratio from the normal density is $-(1-9)/(2\cdot9)=4/9$, agreeing with the exponential-family calculation.

Common mistake avoided: subtracting the cumulant values incorrectly. Here $1/2-1/18=4/9$, so the log ratio is $8/9-4/9=4/9$, not $1/3$.

Common Traps and a Practical Check

The normal exponential-family form is compact, but several distinct quantities can be confused. Keep the variance fixed throughout each calculation and check that the coefficients in the exponent match the expanded normal density.

Confusing mean and natural parameter. The canonical parameter is $\eta=\mu/\sigma^2$. Recover the mean with $\mu=\sigma^2\eta$.
Using the wrong cumulant function. In canonical form with coefficient $y\eta$, use $A(\eta)=\sigma^2\eta^2/2$. A different-looking $b(\theta)=\theta^2/2$ belongs to the convention that divides the parameter-dependent expression by the dispersion $\sigma^2$.
Dropping a factor of two. The cumulant is $\sigma^2\eta^2/2$, not $\sigma^2\eta^2$. Its derivatives must give $A'(\eta)=\sigma^2\eta$ and $A''(\eta)=\sigma^2$.
Putting parameter terms into $c(y)$. In canonical form, $c(y)$ cannot depend on $\eta$. It contains the squared observation term and the normalizing constant.
Changing the model mid-calculation. These formulas assume a known, fixed variance. If the variance is not fixed, the parameterization and likelihood require additional care.

Key takeaway: For a normal response with known variance, write the canonical form with $\eta=\mu/\sigma^2$, $A(\eta)=\sigma^2\eta^2/2$, and $c(y)=-y^2/(2\sigma^2)-\tfrac12\log(2\pi\sigma^2)$. Then $A'(\eta)=\mu$ and $A''(\eta)=\sigma^2$. These identities are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, use the canonical form for a normal response with known variance. Answers are included for checking.

A hypothetical response has distribution $N(\mu,16)$, with $\mu=-4$. Find its natural parameter, cumulant value, mean, and variance. (Answer: $\eta=-1/4$, $A(\eta)=1/2$, mean $-4$, variance $16$.)
An observation $y=2$ is compared under $N(1,9)$ and $N(4,9)$. Find $L(1)/L(4)$. (Answer: $\log[L(1)/L(4)]=1/6$, so $L(1)/L(4)=e^{1/6}\approx1.1814$.)
Independent observations from $N(\mu,1)$ are $-1,0,2$. Find $\widehat{\eta}$ and $\widehat{\mu}$. (Answer: $\widehat{\eta}=1/3$ and $\widehat{\mu}=1/3$.)
For a known variance of $2$ and natural parameter $\eta=1/2$, calculate the corresponding mean and cumulant value, then state the variance. (Answer: $\mu=1$, $A(\eta)=1/4$, and variance $2$.)
Three independent observations from $N(\mu,4)$ are $0,2,4$. Find the maximum likelihood estimates of $\eta$ and $\mu$. (Answer: $\widehat{\eta}=1/2$ and $\widehat{\mu}=2$.)

What You'll Learn