Canonical And Noncanonical Links In Estimation

From the Mean Model to the Likelihood

A generalized linear model (GLM) connects observations to predictors in two stages: the mean is linked to the linear predictor, and the response distribution is specified by an exponential family. These stages fit together especially simply when the link is canonical. With a noncanonical link, the mean is still modeled through the predictor, but the natural parameter of the response distribution becomes a further transformation of that predictor. This changes the likelihood’s dependence on the regression coefficients and therefore its derivatives.

The distinction is about the model’s mathematics, not a claim that one link is always preferable. A canonical link often gives a particularly simple score and a transparent concavity result. A noncanonical link may be useful for a different mean curve, but its likelihood derivatives must account for how the link maps the predictor to the natural parameter.

Definition: Suppose a response distribution belongs to a one-parameter exponential family with density or mass function $$ f(y;\theta,\phi)=\exp\left\{\frac{y\theta-b(\theta)}{a(\phi)}+c(y,\phi)\right\}, $$ where $\theta$ is the natural parameter, $\phi$ is a dispersion parameter, and $a(\phi)>0$. Its mean is $\mu=b'(\theta)$. The canonical link is $g(\mu)=\theta$, or equivalently $g=(b')^{-1}$ on the relevant mean range. In a GLM, $\eta_i=x_i^\mathsf{T}\beta=g(\mu_i)$. Thus, under a canonical link, $\eta_i=\theta_i$.

Here $x_i$ is the fixed predictor vector for observation $i$, $\beta$ is the vector of regression coefficients, and $\eta_i$ is the linear predictor. We consider independent observations, a common fixed dispersion parameter, and a differentiable log-likelihood. In the exponential-family form above, differentiating the mean gives $\operatorname{Var}(Y_i)=a(\phi)b''(\theta_i)$; for the results below we take $b''(\theta_i)>0$ in the parameter region under consideration.

A noncanonical link still defines $\eta_i=g(\mu_i)$. But the natural parameter is then $$ \theta_i=(b')^{-1}(\mu_i)=(b')^{-1}\bigl(g^{-1}(\eta_i)\bigr). $$ Write this composite mapping as $\theta_i=h(\eta_i)$. Under a canonical link, $h(\eta)=\eta$; otherwise it is generally a nonlinear transformation. That difference is what enters the likelihood’s derivatives with respect to $\beta$.

Worked Likelihood Comparisons

Worked Example: Logistic Regression Has a Canonical Score

Let $Y_i$ be Bernoulli with success probability $\mu_i$. Its log-likelihood contribution, excluding no terms that depend on $\mu_i$, is $$ \ell_i=y_i\log(\mu_i)+(1-y_i)\log(1-\mu_i). $$ For the logit link, $\eta_i=x_i^\mathsf{T}\beta=\log\{\mu_i/(1-\mu_i)\}$, so $\mu_i=e^{\eta_i}/(1+e^{\eta_i})$. The Bernoulli natural parameter is $\theta_i=\log\{\mu_i/(1-\mu_i)\}$, hence $\theta_i=\eta_i$: this is the canonical link.

For data $(x_i,y_i)=(0,1),(1,0),(2,1)$, take $x_i=(1,x_i)^\mathsf{T}$ and evaluate the score at $\beta=(0,0)^\mathsf{T}$. Every predictor is $0$, so every fitted probability is $1/2$. The canonical score is $\sum_i x_i(y_i-\mu_i)$. Its intercept component is $ (1-1/2)+(0-1/2)+(1-1/2)=1/2$. Its slope component is $0(1-1/2)+1(0-1/2)+2(1-1/2)=1/2$. Thus $U(\beta)=(1/2,1/2)^\mathsf{T}$ at this parameter value. The score is a sum of predictor vectors multiplied by response residuals, without an extra link-derivative factor.

Worked Example: Probit Regression Adds a Derivative Factor

For a Bernoulli response, use the probit link $g(\mu)=\Phi^{-1}(\mu)$, where $\Phi$ is the standard normal distribution function. Then $\mu=\Phi(\eta)$, but the natural parameter remains $\theta=\log\{\mu/(1-\mu)\}$. At $\eta=0$, $\mu=\Phi(0)=1/2$. Writing $\varphi$ for the standard normal density, the chain rule gives $$ \left.\frac{d\theta}{d\eta}\right|_{\eta=0} =\left.\frac{\mu'}{\mu(1-\mu)}\right|_{\eta=0} =\frac{1/\sqrt{2\pi}}{(1/2)(1/2)} =\frac{4}{\sqrt{2\pi}} =2\sqrt{\frac{2}{\pi}}. $$

For one observation with $x=1$, $y=1$, and predictor $\eta=\beta=0$, the Bernoulli score with respect to $\beta$ is $$ U(\beta)=(y-\mu)\frac{d\theta}{d\eta}\frac{d\eta}{d\beta} =1\cdot\frac12\cdot 2\sqrt{\frac{2}{\pi}} =\sqrt{\frac{2}{\pi}} \approx 0.798. $$ This agrees with direct differentiation of $\log\Phi(\beta)$ at zero: its derivative is $\varphi(0)/\Phi(0)=\sqrt{2/\pi}$. The probit link gives a different derivative factor from the canonical logit link even though both models describe the mean of a Bernoulli response.

Worked Example: Poisson Identity Link Changes the Score

For a Poisson response with mean $\mu$, the log-likelihood for one count $y$ is $y\log(\mu)-\mu-\log(y!)$. The natural parameter is $\theta=\log(\mu)$, so the canonical link is the log link. At $\mu=2$ and $y=3$, using a single intercept predictor, the canonical predictor is $\eta=\beta=\log 2$. Differentiating the log-likelihood with respect to $\beta$, with $\mu=e^\beta$, gives $$ \frac{d\ell}{d\beta} =\left(\frac{y}{\mu}-1\right)\frac{d\mu}{d\beta} =\left(\frac{3}{2}-1\right)2 =1. $$

Now use the identity link, so $\mu=\eta=\beta$, and evaluate at $\beta=2$, which gives the same mean and observed count. Its score is $$ \frac{d\ell}{d\beta}=\frac{y}{\mu}-1=\frac{3}{2}-1=\frac12. $$ The identity link is noncanonical for the Poisson family. Since its predictor maps directly to the mean rather than to $\log(\mu)$, the likelihood changes with $\beta$ at a different rate. For this identity-link model, the predictor must also stay in a region giving an admissible positive Poisson mean.

The Score: What Canonicality Simplifies

For independent observations, the log-likelihood, up to terms that do not depend on $\beta$, is $$ \ell(\beta)=\sum_{i=1}^n\frac{y_i\theta_i-b(\theta_i)}{a(\phi)}. $$ The following derivation makes the canonical and noncanonical cases explicit.

Result: Assume the observations are independent and follow the exponential-family model above, $x_i$ and $\phi$ are fixed, and the functions involved are differentiable. If $\eta_i=x_i^\mathsf{T}\beta$, then $$ U(\beta)=\frac{\partial\ell}{\partial\beta} =\sum_{i=1}^n x_i\frac{y_i-\mu_i}{a(\phi)}\frac{d\theta_i}{d\eta_i}. $$ For a canonical link, $d\theta_i/d\eta_i=1$, so $$ U(\beta)=\sum_{i=1}^n x_i\frac{y_i-\mu_i}{a(\phi)}. $$

Derivation. Since $d\eta_i/d\beta=x_i$, differentiating the $i$th log-likelihood contribution by the chain rule gives $$ \frac{\partial\ell_i}{\partial\beta} =\frac{y_i-b'(\theta_i)}{a(\phi)} \frac{d\theta_i}{d\eta_i}x_i. $$ The mean identity $b'(\theta_i)=\mu_i$ yields the stated sum. For a canonical link $\theta_i=\eta_i$, the derivative is one, giving the simplified score. This result concerns a fixed dispersion parameter; it does not assert that the score equation has a finite solution or that a particular numerical algorithm will find one.

The noncanonical factor can also be expressed in terms of the mean and the link. On a region where the relevant derivatives exist and $g'(\mu_i)\ne 0$, $d\mu_i/d\eta_i=1/g'(\mu_i)$. Meanwhile $d\mu_i/d\theta_i=b''(\theta_i)$. Therefore $$ \frac{d\theta_i}{d\eta_i} =\frac{1}{b''(\theta_i)g'(\mu_i)}. $$ For the canonical link $g'(\mu_i)=1/b''(\theta_i)$, which again makes this factor one. For a noncanonical link, the factor generally varies with the fitted mean, changing how each residual contributes to the score.

Likelihood Curvature Under a Canonical Link

The same distinction appears in the second derivative. For a canonical link, the log-likelihood is concave in $\beta$ under standard regularity conditions. This does not automatically extend to every noncanonical GLM: its Hessian contains an additional term involving the observed residuals.

Theorem: Suppose the observations are independent, the design vectors $x_i$ are fixed, $a(\phi)>0$ is fixed, and the log-likelihood has the exponential-family form above. For a canonical link, suppose $b$ is twice differentiable and $b''(\eta_i)>0$ throughout the parameter region. Then the Hessian is $$ \frac{\partial^2\ell}{\partial\beta\partial\beta^\mathsf{T}} =-\frac{1}{a(\phi)}\sum_{i=1}^n b''(\eta_i)x_ix_i^\mathsf{T}. $$ It is negative semidefinite, so the log-likelihood is concave on any convex parameter region satisfying these assumptions. If the design matrix has full column rank, it is negative definite there and the log-likelihood is strictly concave.

Proof. With a canonical link, $\theta_i=\eta_i=x_i^\mathsf{T}\beta$, and the score is $$ U(\beta)=\frac{1}{a(\phi)}\sum_{i=1}^n x_i\{y_i-b'(\eta_i)\}. $$ Differentiating each term, using $d b'(\eta_i)/d\beta=b''(\eta_i)x_i$, gives the displayed Hessian. For any vector $v$ of the same dimension as $\beta$, $$ v^\mathsf{T}\frac{\partial^2\ell}{\partial\beta\partial\beta^\mathsf{T}}v =-\frac{1}{a(\phi)}\sum_{i=1}^n b''(\eta_i)(x_i^\mathsf{T}v)^2\le 0. $$ This proves negative semidefiniteness and hence concavity on a convex region. If the design matrix has full column rank, every nonzero $v$ has $x_i^\mathsf{T}v\ne0$ for at least one $i$. Since every $b''(\eta_i)>0$, the sum is then strictly positive before the minus sign, so the quadratic form is negative for every nonzero $v$. This proves negative definiteness and strict concavity. $\square$

For a noncanonical link, write $\theta_i=h(\eta_i)$. Differentiating the general score once more gives $$ \frac{\partial^2\ell}{\partial\beta\partial\beta^\mathsf{T}} =\frac{1}{a(\phi)}\sum_{i=1}^n \left[-b''(\theta_i)\{h'(\eta_i)\}^2 +(y_i-\mu_i)h''(\eta_i)\right]x_ix_i^\mathsf{T}. $$ The first term is nonpositive when $b''>0$; the second depends on the observed residual and the curvature of the composite map $h$. Thus the canonical concavity argument does not apply unchanged. This formula identifies an extra source of curvature; it does not imply that every noncanonical log-likelihood is nonconcave.

Why the Distinction Matters

Canonical links are often convenient because the coefficient score has a direct residual form and the log-likelihood curvature follows from the exponential-family function $b$. These properties can simplify estimation and analysis. But canonicality does not guarantee that a finite maximum-likelihood estimate exists, that coefficients are uniquely identified without adequate design information, or that a fitted model is appropriate for a scientific question.

A noncanonical link can be chosen to express a desired relationship between predictors and the mean. Its likelihood is still evaluated from the chosen response distribution, but differentiation must include the transformation from predictor to natural parameter. In practice, omitting that chain-rule factor gives an incorrect score and can lead to incorrect conclusions about curvature or estimation.

Common Mistake

A frequent error is to assume that every GLM score is simply $\sum_i x_i(y_i-\mu_i)/a(\phi)$. That expression follows when the canonical parameter equals the linear predictor. For a noncanonical link, the correct expression includes $d\theta_i/d\eta_i$. In the probit example at $\eta=0$, that factor is $2\sqrt{2/\pi}$, while the Bernoulli residual for $y=1$ is $1/2$; their product is $\sqrt{2/\pi}$, not $2\sqrt{2/\pi}$. The factor must be calculated from the stated link and response family, not inferred from the residual alone.

Key takeaway: Canonicality means that the linear predictor is the natural parameter. This makes the score and Hessian especially simple. With a noncanonical link, the predictor and natural parameter are connected by a composition, and its derivative—and, for curvature, its second derivative—must be carried through the likelihood calculations.

Check Your Understanding

Use the exponential-family likelihood and chain rule to answer these questions.

In a canonical-link GLM, what is the relationship between the linear predictor and the natural parameter?
For a noncanonical link, which factor multiplies the residual in the regression-coefficient score?
For a Bernoulli probit model at $\eta=0$, calculate $d\theta/d\eta$, where $\theta$ is the Bernoulli natural parameter.
State the conditions in the theorem that ensure strict concavity for the canonical-link log-likelihood.
Why does the canonical-link concavity proof not establish concavity for every noncanonical link?