Deriving A Canonical Link From A Distribution

Why Derive a Canonical Link?

A generalized linear model connects predictors to a response distribution through a link function. For a canonical link, the link is not chosen arbitrarily: it is obtained from the response distribution’s exponential-family form. The derivation has two main steps. First, find the mean as a function of the natural parameter. Then invert that relationship to express the natural parameter as a function of the mean.

This distinction matters because the natural parameter and the mean are generally different quantities. A scale or dispersion parameter may also appear in the exponential-family expression, but it does not automatically become part of the canonical link. The derivation makes clear which parameter is being linked to the linear predictor.

Definition: For a scalar response $Y$, write a regular one-parameter exponential family in the form $$ f(y;\theta,\phi)=\exp\left\{\frac{y\theta-b(\theta)}{a(\phi)}+c(y,\phi)\right\}. $$ Here $\theta$ is the natural parameter, $b$ is the cumulant function, and $a(\phi)>0$ is a scale factor. If $b'$ is one-to-one on the allowed natural-parameter space, the canonical link is $g(\mu)=\theta=(b')^{-1}(\mu)$, where $\mu=E[Y]$. The relationship must be restricted to the parameter space on which the distribution is defined.

This form assumes that the response enters through $y\theta$, that the normalizing function is $b(\theta)$, and that differentiation with respect to $\theta$ can be interchanged with integration or summation. For a regular, non-degenerate family, $b''(\theta)>0$; consequently, $b'$ is strictly increasing on an interval of natural parameters and can be inverted there. If the response enters through a statistic $T(Y)$ rather than $Y$, the corresponding derivative gives $E[T(Y)]$; here we use the scalar-response form above.

From the Normalizer to the Mean

The cumulant function is determined by the requirement that the density sums or integrates to one. The following derivation shows why its first derivative gives the mean and its second derivative determines the variance.

Differentiate the log density.
For fixed $\phi$, the derivative with respect to $\theta$ is $(y-b'(\theta))/a(\phi)$.

Take its expectation.
Under the regularity conditions, the expected score is zero. Thus $(E[Y]-b'(\theta))/a(\phi)=0$, so $\mu=b'(\theta)$.

Differentiate once more.
The expected second derivative of the log density is $-b''(\theta)/a(\phi)$. Equating its negative to the Fisher information, which is $\operatorname{Var}(Y)/a(\phi)^2$, gives $\operatorname{Var}(Y)=a(\phi)b''(\theta)$.

Invert the mean relationship.
Since $\mu=b'(\theta)$, solve for $\theta$ as a function of $\mu$. That function is the canonical link.

The same mean result follows directly from normalization. If $Z(\theta)$ is the integral or sum of $\exp\{y\theta/a(\phi)\}\exp\{c(y,\phi)\}$, normalization implies $Z(\theta)=\exp\{b(\theta)/a(\phi)\}$. Differentiating $\log Z(\theta)$ gives $E[Y]/a(\phi)$; differentiating the right-hand side gives $b'(\theta)/a(\phi)$. Equating them yields $E[Y]=b'(\theta)$. This also explains why the scale factor cancels from the mean relationship.

\mu=b'(\theta), \qquad \operatorname{Var}(Y)=a(\phi)b''(\theta), \qquad g(\mu)=\theta=(b')^{-1}(\mu).

In a generalized linear model, the linear predictor is set equal to the linked mean, $g(\mu)=\eta$. Under a canonical link, therefore, $\eta=\theta$. This is the defining connection: the predictor represents the natural parameter, not necessarily the response mean itself. If the mean relationship cannot be inverted over the relevant range, a canonical link cannot be defined there by this formula.

Worked Derivations

Worked Example: A Three-Point Response

A response $Y$ takes values $0,1,2$ and has probabilities proportional to $1,e^\theta,e^{2\theta}$. Derive the mean and canonical link, then check the result when $\theta=0$. Name the method and identify a likely mistake.

Method. Normalize the probabilities by writing the cumulant function as $b(\theta)=\log(1+e^\theta+e^{2\theta})$. The exponential-family form is $\exp\{y\theta-b(\theta)\}$ times the base weight, which is 1 for each of the three response values.

Differentiate the cumulant function:

b'(\theta)=\frac{e^\theta+2e^{2\theta}}{1+e^\theta+e^{2\theta}}.

Thus $\mu=(e^\theta+2e^{2\theta})/(1+e^\theta+e^{2\theta})$. To invert, set $z=e^\theta$, so $z>0$. Multiplying by the denominator and collecting terms gives

\mu(1+z+z^2)=z+2z^2, \qquad (2-\mu)z^2+(1-\mu)z-\mu=0.

The response is bounded between 0 and 2, and for finite $\theta$ its mean satisfies $0<\mu<2$. The positive root of this quadratic is

z=\frac{\mu-1+\sqrt{1+6\mu-3\mu^2}}{2(2-\mu)}.

Since $\theta=\log z$, the canonical link is the logarithm of this positive root. For the check at $\theta=0$, $z=e^0=1$, and the three probabilities are each $1/3$. Their mean is $(0+1+2)/3=1$. Substituting $\mu=1$ into the root gives $z=(0+\sqrt{4})/2=1$, hence $\theta=\log 1=0$, as required.

Common mistake avoided: Using $b(\theta)$ itself as the mean would give $\log 3$ at $\theta=0$, not 1. The mean is the derivative $b'(\theta)$, not the cumulant function $b(\theta)$.

Worked Example: A Shifted Poisson Family

Suppose $Y$ has support $1,2,\ldots$ with probabilities $P(Y=y)=e^{-e^\theta}e^{(y-1)\theta}/(y-1)!$. Derive its canonical link and find the mean and variance when the linear predictor is $\eta=\log 2$. State the method and the error it prevents.

Method. Rewrite the probability as

P(Y=y)=\exp\{y\theta-(\theta+e^\theta)-\log((y-1)!)\}.

Therefore $a(\phi)=1$, $b(\theta)=\theta+e^\theta$, and $c(y,\phi)=-\log((y-1)!)$. Differentiate:

b'(\theta)=1+e^\theta,\qquad b''(\theta)=e^\theta.

It follows that $\mu=1+e^\theta$. Solving gives $e^\theta=\mu-1$, so the canonical link is $g(\mu)=\log(\mu-1)$, defined for $\mu>1$. With $\eta=\theta=\log 2$, $e^\theta=2$, and the fitted mean is $\mu=1+2=3$. The variance is $a(\phi)b''(\theta)=1(2)=2$.

Check. This family is one plus a Poisson random variable with parameter $e^\theta=2$. Its mean is $1+2=3$ and its variance is 2, agreeing with the exponential-family calculations.

Common mistake avoided: Writing $\log\mu$ as the canonical link would be incorrect. The mean relation is $\mu=1+e^\theta$, so the quantity to invert is $\mu-1$, not $\mu$.

Worked Example: A Normal Response With Known Scale

Let $Y\sim N(\theta,9)$, where the second argument denotes the variance. Derive the canonical link and, for a model with fitted mean $12.4$, report the natural parameter and variance. Explain the scale-related trap.

Method. Expanding the normal density gives

f(y;\theta)=\exp\left\{\frac{y\theta-\theta^2/2}{9} -\frac{y^2}{18}-\frac{1}{2}\log(18\pi)\right\}.

Thus $a(\phi)=9$, $b(\theta)=\theta^2/2$, and $b'(\theta)=\theta$, while $b''(\theta)=1$. Therefore $\mu=\theta$ and the canonical link is $g(\mu)=\mu$. For fitted mean $12.4$, the natural parameter is $\theta=12.4$. The variance is $9(1)=9$, consistent with the stated distribution.

Check. Substituting $\theta=12.4$ into the normal model gives mean $12.4$ and variance 9, so both the mean relationship and the scale are consistent.

Common mistake avoided: Dividing the mean by 9 to obtain the natural parameter confuses the scale in the exponent with the natural parameter. Here the exponent contains $(y\theta-b(\theta))/9$, but the mean is still $b'(\theta)=\theta$.

How to Check a Derived Link

A reliable derivation should pass several checks before it is used in a model. First, differentiate the stated cumulant function carefully; a missing term or sign changes both the mean and the link. Next, solve $\mu=b'(\theta)$ for $\theta$, rather than solving the reverse relationship by inspection. Finally, check that the inverse is defined over the distribution’s possible mean values and that substituting the recovered $\theta$ reproduces the starting mean.

Step	Question to check	Reason
Identify the form	Is the exponent written as $(y\theta-b(\theta))/a(\phi)+c(y,\phi)$?	The derivative rule depends on identifying the natural parameter and cumulant correctly.
Differentiate	Is the mean $b'(\theta)$?	The mean is given by the first derivative, not by $b$.
Invert	Can $\mu=b'(\theta)$ be solved for $\theta$ on the allowed range?	The canonical link must return a valid natural parameter.
Check scale	Does $a(\phi)$ affect the mean relationship in this parameterization?	It affects the variance, but cancels from the mean derivation.

A common exam trap is to confuse the canonical link with a familiar transformation of the response. The canonical link follows from the exponential-family parameterization: take the derivative of $b$, then invert. It is not found by guessing a logarithm or by inserting the scale factor into the link. Also check parameter constraints: if the natural parameter is restricted, the linked mean must lie in the corresponding mean range.

Key takeaway: In the scalar regular exponential-family form, derive $\mu=b'(\theta)$ from normalization or the expected score. The canonical link is the inverse relationship, $g(\mu)=(b')^{-1}(\mu)$, on the valid parameter range; the scale factor enters the variance, not this mean-to-natural-parameter inversion.

Check Your Understanding

For each problem, identify the cumulant function, derive the mean-natural parameter relationship, and give the requested canonical link or value.

A family has $b(\theta)=\log(3+e^\theta)$, with unit scale. Find its canonical link for $0<\mu<1$. (Answer: $g(\mu)=\log(3\mu/(1-\mu))$.)
A family has $b(\theta)=4\theta^2$ and unit scale. Derive its mean and canonical link. (Answer: $\mu=8\theta$, so $g(\mu)=\mu/8$.)
A response on $\{0,1,2\}$ has cumulant $b(\theta)=\log(1+2e^\theta+e^{2\theta})$. Find its mean at $\theta=0$, and determine the natural parameter that gives mean 1. (Answer: The mean at $\theta=0$ is 1, and $\theta=0$ gives mean 1.)
A shifted Poisson family has $b(\theta)=\theta+e^\theta$ and unit scale. Find the natural parameter when the mean is 5. (Answer: $\theta=\log 4$.)
A family is written as $f(y;\theta,\phi)=\exp\{(y\theta-\theta^2/2)/\phi+c(y,\phi)\}$, with $\phi>0$. Identify its mean, variance, and canonical link. (Answer: $\mu=\theta$, $\operatorname{Var}(Y)=\phi$, and $g(\mu)=\mu$.)

Step	Question to check	Reason
Identify the form	Is the exponent written as \((y\theta-b(\theta))/a(\phi)+c(y,\phi)\)?	The derivative rule depends on identifying the natural parameter and cumulant correctly.
Differentiate	Is the mean \(b'(\theta)\)?	The mean is given by the first derivative, not by \(b\).
Invert	Can \(\mu=b'(\theta)\) be solved for \(\theta\) on the allowed range?	The canonical link must return a valid natural parameter.
Check scale	Does \(a(\phi)\) affect the mean relationship in this parameterization?	It affects the variance, but cancels from the mean derivation.

What You'll Learn