Worked Derivations for Normal, Gamma, and Inverse-Gaussian Densities

From Densities to Exponential-Family Forms

A useful way to check a continuous response model is to start from its density, take logarithms, and sort the terms by what they depend on. The terms involving the response and the model parameter form the parameter kernel. Everything else belongs to the carrier, which may depend on the response and the dispersion but not on the natural parameter. This tutorial carries out that calculation for the normal, gamma, and inverse-Gaussian distributions.

We use the dispersion-scaled form

f(y;\theta,\phi) = h(y,\phi)\exp\left\{\frac{y\theta-b(\theta)}{a(\phi)}\right\},

where \(y\) is the observed response, \(\theta\) is the natural parameter, \(\phi\) is a dispersion parameter, \(a(\phi)>0\), \(b\) is the log-partition function, and \(h\) is the carrier. The carrier may depend on \(\phi\), but it must not depend on \(\theta\). The density is with respect to the usual Lebesgue measure on its support.

Definition: A density has the displayed dispersion-scaled representation if it can be written as a carrier \(h(y,\phi)\) times the exponential of a parameter kernel \(\{y\theta-b(\theta)\}/a(\phi)\). The response support must be specified, and the chosen parameter domain must make the density well-defined.

The calculation is an algebraic sorting exercise, but the sorting criterion matters: after choosing \(\theta\), \(b\), and \(a(\phi)\), any term that still varies with \(\theta\) cannot be left in \(h\). We will identify each component directly from the log density rather than infer it from a memorized table.

Worked Derivation: Normal Density

Worked Example: Normal Response with Dispersion

Let \(Y\sim N(\mu,\phi)\), where \(\phi>0\) is the variance and \(\mu\in\mathbb{R}\). The response support is \(y\in\mathbb{R}\), and the density is

f(y;\mu,\phi) = \frac{1}{\sqrt{2\pi\phi}} \exp\left\{-\frac{(y-\mu)^2}{2\phi}\right\}.

Take the logarithm and expand the square:

\begin{aligned} \log f(y;\mu,\phi) &=-\frac12\log(2\pi\phi)-\frac{(y-\mu)^2}{2\phi}\\ &=-\frac12\log(2\pi\phi)-\frac{y^2}{2\phi} +\frac{y\mu}{\phi}-\frac{\mu^2}{2\phi}\\ &=-\frac12\log(2\pi\phi)-\frac{y^2}{2\phi} +\frac{y\mu-\mu^2/2}{\phi}. \end{aligned}

The final numerator has the required form \(y\theta-b(\theta)\) when \(\theta=\mu\) and \(b(\theta)=\theta^2/2\). Thus \(a(\phi)=\phi\). Exponentiating the remaining terms gives the carrier:

h(y,\phi) = \frac{1}{\sqrt{2\pi\phi}} \exp\left\{-\frac{y^2}{2\phi}\right\}.

Therefore the dispersion-scaled representation is

f(y;\theta,\phi) = h(y,\phi) \exp\left\{\frac{y\theta-\theta^2/2}{\phi}\right\}, \qquad \theta\in\mathbb{R},\ \phi>0.

The carrier contains a term involving \(y^2\), but no term involving \(\theta\). It need not itself integrate to one; the complete product is the density.

Worked Derivation: Gamma Density

Worked Example: Gamma Response with Mean and Dispersion

Let \(Y\) have a gamma distribution with shape \(1/\phi\) and scale \(\phi\mu\), where \(\mu>0\) and \(\phi>0\). This parameterization gives the density

f(y;\mu,\phi) = \frac{y^{1/\phi-1}\exp\{-y/(\phi\mu)\}} {\Gamma(1/\phi)(\phi\mu)^{1/\phi}}, \qquad y>0.

Taking logarithms, including the terms in the normalizing factor, yields

\begin{aligned} \log f(y;\mu,\phi) ={}&\left(\frac1\phi-1\right)\log y-\frac{y}{\phi\mu} -\log\Gamma(1/\phi)-\frac1\phi\log\phi-\frac1\phi\log\mu\\ ={}&\left(\frac1\phi-1\right)\log y-\log\Gamma(1/\phi) -\frac1\phi\log\phi\\ &+\frac{-y/\mu-\log\mu}{\phi}. \end{aligned}

Set \(\theta=-1/\mu\), so \(\theta<0\) and \(\mu=-1/\theta\). Then \(-y/\mu=y\theta\), while \(\log\mu=\log(-1/\theta)=-\log(-\theta)\). Consequently,

-y/\mu-\log\mu = y\theta+\log(-\theta) = y\theta-b(\theta), \qquad b(\theta)=-\log(-\theta).

With \(a(\phi)=\phi\), all remaining terms are independent of \(\theta\). The carrier and resulting representation are

\begin{aligned} h(y,\phi) &= \frac{y^{1/\phi-1}\exp\{-\log\Gamma(1/\phi)-(1/\phi)\log\phi\}} {1}\\ &= \frac{y^{1/\phi-1}}{\Gamma(1/\phi)\phi^{1/\phi}},\\ f(y;\theta,\phi) &=h(y,\phi) \exp\left\{\frac{y\theta+\log(-\theta)}{\phi}\right\}, \qquad \theta<0,\ \phi>0,\ y>0. \end{aligned}

As a direct check, substituting \(\theta=-1/\mu\) into the exponential factor gives \(\exp\{-y/(\phi\mu)-(1/\phi)\log\mu\}\). Multiplying by \(h(y,\phi)\) recovers the original gamma density exactly.

Worked Derivation: Inverse-Gaussian Density

Worked Example: Inverse-Gaussian Response with Dispersion

For mean \(\mu>0\) and dispersion \(\phi>0\), the inverse-Gaussian density on \(y>0\) is

f(y;\mu,\phi) = \frac{1}{\sqrt{2\pi\phi y^3}} \exp\left\{-\frac{(y-\mu)^2}{2\phi\mu^2y}\right\}.

Expand the quadratic numerator in the exponent. Since \((y-\mu)^2=y^2-2y\mu+\mu^2\), division by \(2\phi\mu^2y\) gives

-\frac{(y-\mu)^2}{2\phi\mu^2y} = -\frac{y}{2\phi\mu^2}+\frac{1}{\phi\mu}-\frac{1}{2\phi y}.

The complete log density is therefore

\log f(y;\mu,\phi) = -\frac12\log(2\pi\phi)-\frac32\log y -\frac{1}{2\phi y} +\frac{-y/(2\mu^2)+1/\mu}{\phi}.

Choose \(\theta=-1/(2\mu^2)\), which ranges over \(\theta<0\). Because \(\mu>0\), \(1/\mu=\sqrt{-2\theta}\). Hence the parameter-dependent numerator becomes \(y\theta+\sqrt{-2\theta}\), or \(y\theta-b(\theta)\) with \(b(\theta)=-\sqrt{-2\theta}\). Taking \(a(\phi)=\phi\), the carrier is

h(y,\phi) = \frac{1}{\sqrt{2\pi\phi}\,y^{3/2}} \exp\left\{-\frac{1}{2\phi y}\right\}.

Thus

f(y;\theta,\phi) = h(y,\phi) \exp\left\{\frac{y\theta+\sqrt{-2\theta}}{\phi}\right\}, \qquad \theta<0,\ \phi>0,\ y>0.

For a check, substitute \(\theta=-1/(2\mu^2)\). Then \(y\theta=-y/(2\mu^2)\) and \(\sqrt{-2\theta}=1/\mu\), reproducing the two parameter-dependent terms in the expanded log density.

Result: The Three Dispersion-Scaled Forms

The calculations establish the following representations. In each row, the support is fixed as the natural parameter varies, and the carrier is free of that natural parameter.

Result: For the stated parameterizations, the normal, gamma, and inverse-Gaussian densities have dispersion-scaled exponential-family forms with the components shown below. The identities follow by expanding the log densities and exponentiating the terms left after extracting \(\{y\theta-b(\theta)\}/\phi\).

Family and support	Natural parameter \(\theta\)	\(b(\theta)\)	Carrier \(h(y,\phi)\)
Normal, \(y\in\mathbb{R}\)	\(\mu\)	\(\theta^2/2\)	\((2\pi\phi)^{-1/2}\exp\{-y^2/(2\phi)\}\)
Gamma, \(y>0\)	\(-1/\mu<0\)	\(-\log(-\theta)\)	\(y^{1/\phi-1}/\{\Gamma(1/\phi)\phi^{1/\phi}\}\)
Inverse Gaussian, \(y>0\)	\(-1/(2\mu^2)<0\)	\(-\sqrt{-2\theta}\)	\((2\pi\phi)^{-1/2}y^{-3/2}\exp\{-1/(2\phi y)\}\)

To verify the result in each case, insert the table entries into \(h(y,\phi)\exp\{(y\theta-b(\theta))/\phi\}\). For the normal, the exponent adds \(-y^2/(2\phi)\) to \((y\mu-\mu^2/2)/\phi\), giving \(-(y-\mu)^2/(2\phi)\). For the gamma, it adds \(-y/(\phi\mu)-(\log\mu)/\phi\) to the carrier’s \((1/\phi-1)\log y-\log\Gamma(1/\phi)-(\log\phi)/\phi\), which is the original log density. For the inverse Gaussian, it adds \(-y/(2\phi\mu^2)+1/(\phi\mu)\) to the carrier terms, giving the expanded original log density. These checks also confirm that no term depending on \(\theta\) was assigned to a carrier.

Why the Sorting Matters

The factorization is useful because it makes the parameter dependence visible. A reliable audit is to write the full log density, isolate the proposed kernel, and then inspect every leftover term. If a leftover term changes when \(\theta\) changes, the proposed carrier is not a carrier for that representation. Conversely, dependence on \(y\) or \(\phi\) in the carrier is allowed.

The support also needs attention. Here the normal support is \(\mathbb{R}\) and the gamma and inverse-Gaussian supports are \((0,\infty)\); none changes as the natural parameter varies. A term that makes a density zero outside its support is not an excuse to ignore parameter-dependent support in a different family. The support and the algebraic factorization both need to be checked.

Common Mistakes

Putting every term without \(y\) into the carrier. For the gamma density, \(-(\log\mu)/\phi\) contains no \(y\), but it depends on the natural parameter. It must remain in the parameter kernel as part of \(-b(\theta)/\phi\).
Dropping normalizing terms too early. Terms such as \(-\log\Gamma(1/\phi)\) and \(-(1/\phi)\log\phi\) are essential parts of the gamma carrier. Omitting them means the displayed expression no longer equals the density.
Using an invalid natural-parameter domain. The gamma and inverse-Gaussian choices require \(\theta<0\). Taking a logarithm of \(-\theta\) or a square root of \(-2\theta\) requires that restriction.
Confusing the carrier with a density by itself. The carrier need not integrate to one. Normalization belongs to the full expression, including its parameter kernel.

Key takeaway: Expand first, choose the natural parameter and dispersion scaling, then verify that the remaining carrier has no natural-parameter dependence and that the reconstructed expression is exactly the original density.

Check Your Understanding

For each question, keep the full log density in view and check the parameter dependence of every term.

For the normal representation, identify the carrier and explain why its dependence on \(y\) is allowed.
In the gamma derivation, show that \(\theta=-1/\mu\) implies \(-\log\mu=\log(-\theta)\).
Why must the gamma natural-parameter domain be restricted to \(\theta<0\)?
Expand the inverse-Gaussian quadratic and identify which term involving \(y\) belongs in the carrier.
Suppose a proposed carrier contains \(\log\mu\) in the gamma representation. What condition does this violate, and how should that term be reassigned?