Canonical-Link Score Equations

Why Canonical Links Simplify the Score

A GLM regression score measures how the log-likelihood changes when a regression coefficient changes. In a general link model, that derivative carries a factor for how quickly the mean changes with the linear predictor. Under a canonical link, the linear predictor is the natural parameter of the response distribution, and this extra factor disappears. The resulting score has a direct residual form: each observation contributes its covariate vector multiplied by its observed-minus-fitted response.

This form is useful for more than simplifying differentiation. At a finite interior solution, setting the score to zero requires the fitted responses to reproduce the observed responses in every design direction, with the appropriate observation weights. The statement concerns weighted covariate totals; it does not generally say that each individual fitted response equals its observed response.

Model and Notation

Suppose $Y_1,\ldots,Y_n$ are independent responses. Observation $i$ has fixed covariate vector $x_i=(x_{i1},\ldots,x_{ip})^\mathsf{T}$, and the linear predictor is $\eta_i=x_i^\mathsf{T}\beta$, where $\beta$ is a $p$-vector of regression coefficients. Assume each response has an exponential-dispersion density or mass function of the form below, with support not depending on $\beta$:

f(y_i;\theta_i,\phi) = \exp\left\{ \frac{y_i\theta_i-b(\theta_i)}{a_i(\phi)} +c(y_i,\phi) \right\}.

Here $\theta_i$ is the natural parameter, $b$ is differentiable, and $a_i(\phi)>0$ is the dispersion scale for observation $i$. The common case with prior weight $w_i>0$ has $a_i(\phi)=\phi/w_i$. We treat $\phi$ as fixed when differentiating with respect to $\beta$. The mean is $\mu_i=E[Y_i]=b'(\theta_i)$.

Definition: The canonical link is the link for which the linear predictor equals the natural parameter: $\eta_i=\theta_i$. Thus, under this link, $\theta_i=x_i^\mathsf{T}\beta$, and the fitted mean is $\mu_i=b'(x_i^\mathsf{T}\beta)$.

The exponential-dispersion form gives the mean $b'(\theta_i)$; this follows by differentiating the normalizing integral (or sum) with respect to $\theta_i$, under the usual regularity conditions that permit that differentiation. The score derivation below needs the log-likelihood to be differentiable in $\beta$, and the support to be fixed so that no parameter-dependent boundary terms arise.

Deriving the Canonical Regression Score

Independence makes the log-likelihood the sum of the individual log-likelihood contributions. For fixed observations, $\phi$, and covariates, terms $c(y_i,\phi)$ do not depend on $\beta$. Under a canonical link, the part that does depend on $\beta$ is $\{y_i x_i^\mathsf{T}\beta-b(x_i^\mathsf{T}\beta)\}/a_i(\phi)$.

\ell(\beta) = \sum_{i=1}^n \left\{ \frac{y_i x_i^\mathsf{T}\beta-b(x_i^\mathsf{T}\beta)} {a_i(\phi)} +c(y_i,\phi) \right\}.

Theorem: Under the model and differentiability conditions above, the regression score under a canonical link is $$ U(\beta) = \frac{\partial\ell(\beta)}{\partial\beta} = \sum_{i=1}^n \frac{x_i(y_i-\mu_i)}{a_i(\phi)}. $$ If $a_i(\phi)=\phi/w_i$, this becomes $$ U(\beta) = \frac{1}{\phi}\sum_{i=1}^n w_i x_i(y_i-\mu_i). $$

Proof. For each observation, the chain rule gives $\partial(x_i^\mathsf{T}\beta)/\partial\beta=x_i$. Since $b$ is differentiable, the derivative of $b(x_i^\mathsf{T}\beta)$ is $b'(x_i^\mathsf{T}\beta)x_i=\mu_i x_i$. The term $c(y_i,\phi)$ has derivative zero with respect to $\beta$. Therefore

\frac{\partial\ell(\beta)}{\partial\beta} = \sum_{i=1}^n \frac{y_i x_i-\mu_i x_i}{a_i(\phi)} = \sum_{i=1}^n \frac{x_i(y_i-\mu_i)}{a_i(\phi)}.

When $a_i(\phi)=\phi/w_i$, its reciprocal is $w_i/\phi$, giving the stated weighted form. $\square$

The dispersion factor is part of the score, even though it may not affect the location of its zeros. If all observations have the same dispersion $\phi$, multiplying the score by $\phi$ does not change its zero set because $\phi>0$. With observation-specific scales, the factors $1/a_i(\phi)$ need not be common and cannot generally be discarded.

Score Equations as Balance Conditions

Write $X$ for the $n$-by-$p$ design matrix whose $i$th row is $x_i^\mathsf{T}$, and let $D$ be diagonal with entries $D_{ii}=1/a_i(\phi)$. Let $y$ and $\mu$ denote the vectors of observed and fitted responses. The score is $X^\mathsf{T}D(y-\mu)$. Consequently, a finite interior solution of the likelihood equations must satisfy the following balance condition.

Theorem: The canonical-link score equations $U(\beta)=0$ are equivalent to $$ X^\mathsf{T}Dy=X^\mathsf{T}D\mu. $$ In coordinates, for every covariate $j=1,\ldots,p$, $$ \sum_{i=1}^n \frac{x_{ij}y_i}{a_i(\phi)} = \sum_{i=1}^n \frac{x_{ij}\mu_i}{a_i(\phi)}. $$ For $a_i(\phi)=\phi/w_i$, the common factor $1/\phi$ cancels, leaving equality of the $w_i$-weighted covariate totals.

Proof. Substituting the score formula gives $U(\beta)=X^\mathsf{T}D(y-\mu)$. This vector is zero exactly when $X^\mathsf{T}Dy-X^\mathsf{T}D\mu=0$, which is the displayed matrix equation. Its $j$th component is the difference between the two sums in the coordinate expression. Conversely, equality of those sums for every column of $X$ makes every component of the score zero. $\square$

This also shows precisely which totals are balanced. For any vector $v$ in the column space of $X$, write $v=Xc$ for some coefficient vector $c$. At a score root,

v^\mathsf{T}D(y-\mu) = c^\mathsf{T}X^\mathsf{T}D(y-\mu) = 0.

Thus the weighted residual vector is orthogonal to every design column and every linear combination of those columns. In a model with an intercept, the vector of ones is a design column, so the scale-weighted observed and fitted response totals agree. Without an intercept, that total-balance conclusion does not follow unless the constant vector is in the column space of the design matrix.

A zero score is a necessary first-order condition for an unconstrained, finite interior maximum. It does not by itself prove that such a maximum exists, that it is unique, or that a particular root is a maximum rather than another stationary point. The balance equations describe the score root; likelihood existence and curvature require additional analysis.

Worked Examples

Worked Example: Poisson Regression With a Group Indicator

Suppose six independent counts follow a Poisson regression with log link and linear predictor $\eta_i=\beta_0+\beta_1 z_i$, where $z_i=0$ for group A and $z_i=1$ for group B. The observed counts are $1,2,3$ in group A and $2,4,6$ in group B. Find the fitted means at a score root.

For a Poisson response, $b(\theta)=e^\theta$, so under the canonical log link $\mu_i=e^{\eta_i}$. Here $a_i(\phi)=1$. The intercept equation sets the total observed count equal to the total fitted count: $\sum_i y_i=\sum_i\mu_i$. Its two sides are $1+2+3+2+4+6=18$ and the sum of the six fitted means.

The slope equation is $\sum_i z_i(y_i-\mu_i)=0$. Since $z_i=1$ only in group B, this requires the fitted total for group B to equal its observed total: $\sum_{i\in B}\mu_i=2+4+6=12$. Subtracting from the intercept equation shows that the group A fitted total is $18-12=6$.

Each group has three observations, and the model gives each observation in a group the same fitted mean. Therefore $\mu_A=6/3=2$, and $\mu_B=12/3=4$. These values are achieved by $\beta_0=\log 2$ and $\beta_1=\log 4-\log 2=\log 2$. Substitution gives the intercept balance $3(2)+3(4)=18$ and group B balance $3(4)=12$, as required.

Worked Example: Bernoulli Model With an Intercept Only

Five independent Bernoulli observations are $1,0,1,1,0$, and the model has only an intercept with the canonical logit link. Find the score-root fitted probability and intercept.

The canonical parameter is $\theta=\log\{p/(1-p)\}=\beta_0$, and the fitted probability is $p=e^{\beta_0}/(1+e^{\beta_0})$. The intercept score equation is $\sum_i(y_i-p)=0$, so $5p=\sum_i y_i$. There are three observed successes, giving $p=3/5=0.6$.

The corresponding intercept is $\beta_0=\log\{p/(1-p)\}=\log\{(3/5)/(2/5)\}=\log(3/2)$, approximately $0.4055$. The fitted total is $5(0.6)=3$, exactly the observed success total. The individual responses are not all equal to $0.6$; it is their total that the intercept equation balances.

Worked Example: Gaussian Identity Link With Prior Weights

Consider three independent Gaussian responses $y=(2,4,5)$, a model with an intercept only, known variance $\phi=\sigma^2$, and prior weights $w=(1,2,1)$. Find the score-root fitted mean and verify the dispersion factor in the score.

For a Gaussian model with identity link, the natural parameter is the mean, so the link is canonical. The observation scales are $a_i(\phi)=\sigma^2/w_i$. The intercept score is therefore $\sum_i w_i(y_i-\mu)/\sigma^2=0$. Since $\sigma^2>0$, multiplying through by $\sigma^2$ gives $(1+2+1)\mu=1(2)+2(4)+1(5)=15$.

Thus the fitted mean is $\mu=15/4=3.75$. The weighted fitted total is $1(3.75)+2(3.75)+1(3.75)=15$, which equals the weighted observed total $2+8+5=15$. Directly, the score is $\{1(2-3.75)+2(4-3.75)+1(5-3.75)\}/\sigma^2=(-1.75+0.5+1.25)/\sigma^2=0$.

Common Pitfall and Interpretation

A common mistake is to write the canonical score as $\sum_i x_i(y_i-\mu_i)$ without checking the dispersion or prior-weight convention. That expression is correct only when the observation scales are all one, or when the omitted factor is a common positive constant and only the zero equation is being stated. With $a_i(\phi)=\phi/w_i$, the score itself contains $w_i/\phi$; with unequal observation-specific scales, the relative weights affect the equation as well.

A second mistake is to interpret the equations as requiring $y_i=\mu_i$ for each observation. They instead require the weighted residuals to sum to zero after projection onto each design column. For example, an intercept balances a weighted overall total, while a group-indicator column balances the total within the indicated group. A covariate column with positive and negative values balances a signed covariate-weighted total, not necessarily any ordinary subgroup total.

Key takeaway: Under a canonical link, each regression-score component is a dispersion-adjusted covariate-weighted sum of residuals. At a finite score root, weighted observed and fitted totals agree in every design direction; the equation does not imply observation-by-observation equality.

Check Your Understanding

Use the canonical score and balance interpretation to answer the following.

For $a_i(\phi)=\phi/w_i$, write the canonical regression score and identify the factor that can be removed when solving its zero equation.
Why does an intercept imply equality of the scale-weighted observed and fitted response totals?
In an intercept-plus-binary-indicator model, what total does the indicator score balance?
Does a zero canonical score imply $y_i=\mu_i$ for every observation? Explain.
Why is a score root not, by itself, proof that a finite maximum-likelihood estimate exists?

What You'll Learn