Differentiating Through The GLM Mean And Link

Why the Link Derivative Matters

In a generalized linear model, a regression coefficient changes the linear predictor, the linear predictor determines the mean through the inverse link, and the mean determines the likelihood. A derivative with respect to a coefficient therefore passes through each of these steps. Keeping those steps distinct clarifies where the design matrix enters and why the derivative of the link—or, equivalently, of its inverse—matters.

The goal here is to differentiate a likelihood contribution with respect to regression coefficients. The result applies to different response distributions and links, provided the required derivatives exist. It does not require rewriting the likelihood in a particular exponential-family form.

Set-Up and Chain Rule

Suppose there are $n$ independent observations and $p$ regression coefficients collected in $\beta=(\beta_1,\ldots,\beta_p)^\mathsf{T}$. Let $x_i^\mathsf{T}$ be row $i$ of the fixed design matrix $X$, and write the linear predictor and mean for observation $i$ as

\eta_i=x_i^\mathsf{T}\beta,\qquad \mu_i=g^{-1}(\eta_i).

Here $g$ is the link function, so $g(\mu_i)=\eta_i$. Assume the likelihood depends on $\beta$ only through these means, with any other parameters held fixed. Write the log-likelihood contribution from observation $i$ as $\ell_i(y_i;\mu_i)$, and define its derivative with respect to the mean by

q_i(\mu_i)=\frac{\partial \ell_i(y_i;\mu_i)}{\partial \mu_i}.

The link is assumed differentiable at the relevant mean, with $g'(\mu_i)\ne 0$. Under these conditions the inverse function is differentiable there, and its derivative is $1/g'(\mu_i)$. The observations are finite in number, so differentiating the sum of their log-likelihood contributions requires no interchange of a limit and an integral.

Chain-rule result: For each observation $i$ and coefficient $\beta_j$, provided the conditions above hold, $$ \frac{\partial \ell_i}{\partial \beta_j} = \frac{\partial \ell_i}{\partial \mu_i} \frac{1}{g'(\mu_i)} x_{ij}. $$ Equivalently, if $h=g^{-1}$ is differentiable, the middle factor is $h'(\eta_i)$. The design entry $x_{ij}$ describes how changing $\beta_j$ changes $\eta_i$; the inverse-link derivative describes how that change moves $\mu_i$.

To verify the result, hold all coefficients except $\beta_j$ fixed. Since $\eta_i=\sum_{k=1}^{p}x_{ik}\beta_k$, its derivative with respect to $\beta_j$ is $x_{ij}$. Also, differentiating $g(\mu_i)=\eta_i$ with respect to $\beta_j$ gives

g'(\mu_i)\frac{\partial\mu_i}{\partial\beta_j} = x_{ij}, \qquad \frac{\partial\mu_i}{\partial\beta_j} = \frac{x_{ij}}{g'(\mu_i)}.

Applying the ordinary chain rule to $\ell_i(y_i;\mu_i)$ now gives the stated derivative. If the full log-likelihood is the sum $\ell(\beta)=\sum_{i=1}^{n}\ell_i(y_i;\mu_i)$, its derivative with respect to $\beta_j$ is the sum of these observation-level derivatives:

\frac{\partial\ell}{\partial\beta_j} = \sum_{i=1}^{n} q_i(\mu_i)\frac{x_{ij}}{g'(\mu_i)}.

The formula separates three factors: sensitivity of the likelihood to the mean, sensitivity of the mean to the linear predictor, and sensitivity of the linear predictor to the coefficient. A zero design entry makes that observation contribute no derivative for that coefficient. A nonzero entry does not by itself determine the direction of change; the likelihood and link factors matter too.

Design-Matrix Form

The same calculation can be written for all observations and coefficients at once. Let $q$ be the $n$-vector with entries $q_i(\mu_i)$, and let $D$ be the diagonal matrix whose $i$th diagonal entry is $1/g'(\mu_i)$. Applying the chain rule to each component of $\mu=(\mu_1,\ldots,\mu_n)^\mathsf{T}$ gives the following identity.

Design-matrix derivative: Under the stated differentiability conditions, $$ \frac{\partial\mu}{\partial\beta^\mathsf{T}}=DX, \qquad \nabla_\beta\ell=X^\mathsf{T}Dq. $$ The first matrix has one row per observation and one column per coefficient. The second expression adds each observation’s contribution to the derivative for each coefficient.

For the first identity, row $i$, column $j$ of the derivative matrix is $\partial\mu_i/\partial\beta_j=x_{ij}/g'(\mu_i)$, which is row $i$, column $j$ of $DX$. For the second, differentiating the finite sum of log-likelihood contributions gives $X^\mathsf{T}Dq$: its $j$th entry is $\sum_i x_{ij}q_i(\mu_i)/g'(\mu_i)$, exactly the coefficient derivative above. This establishes the matrix identity without assuming any particular response distribution. $\square$

In this expression $D$ is evaluated at the fitted means, so its entries generally depend on $\beta$. The identity gives the derivative at a specified coefficient vector; it does not claim that $D$ stays fixed when the coefficients change.

Worked Examples

Worked Example: Bernoulli Response With a Logit Link

Suppose one independent response is Bernoulli with observed value $y=1$, and its model has $x^\mathsf{T}=(1,2)$, $\beta=(0,\log 2)^\mathsf{T}$, and logit link $g(\mu)=\log\{\mu/(1-\mu)\}$. Find the derivative of this observation’s log-likelihood with respect to each coefficient.

The linear predictor is $\eta=0+2\log 2=\log 4$, so $\mu=e^\eta/(1+e^\eta)=4/5$. The Bernoulli log-likelihood contribution is $\ell=y\log\mu+(1-y)\log(1-\mu)$, and its mean derivative is

\frac{\partial\ell}{\partial\mu} = \frac{y}{\mu}-\frac{1-y}{1-\mu} = \frac{1}{\mu} = \frac{5}{4}.

The link derivative is $g'(\mu)=1/\{\mu(1-\mu)\}=25/4$, so $1/g'(\mu)=4/25$. Thus the derivative with respect to the linear predictor is $(5/4)(4/25)=1/5$. Multiplying by the design entries gives

\frac{\partial\ell}{\partial\beta_1}=\frac{1}{5}(1)=\frac{1}{5}, \qquad \frac{\partial\ell}{\partial\beta_2}=\frac{1}{5}(2)=\frac{2}{5}.

The intercept’s design entry is one, while the second coefficient’s entry is two; that is why the second derivative is twice the first.

Worked Example: Bernoulli Response With a Probit Link

Now take a Bernoulli response with $y=1$, design row $x^\mathsf{T}=(1,-1)$, and coefficients $\beta=(0,0)^\mathsf{T}$. Use the probit link $g(\mu)=\Phi^{-1}(\mu)$, where $\Phi$ is the standard normal distribution function. Compute both coefficient derivatives.

Here $\eta=0$ and $\mu=\Phi(0)=1/2$. The Bernoulli likelihood has $\partial\ell/\partial\mu=y/\mu=2$. Since the derivative of $\Phi$ is the standard normal density $\phi$, the inverse link satisfies $d\mu/d\eta=\phi(0)=1/\sqrt{2\pi}$. The derivative with respect to the linear predictor is therefore $2/\sqrt{2\pi}=\sqrt{2/\pi}$. Multiplication by the two design entries yields

\frac{\partial\ell}{\partial\beta_1} = \sqrt{\frac{2}{\pi}}, \qquad \frac{\partial\ell}{\partial\beta_2} = -\sqrt{\frac{2}{\pi}}.

The negative sign is not caused by the link: it comes from the second design entry, which makes the linear predictor decrease as $\beta_2$ increases. Compared with the logit example, the link changes the factor translating a predictor change into a mean change.

Worked Example: Normal Response With an Identity Link

Let $Y$ have a normal distribution with mean $\mu$ and known variance $\sigma^2=4$. For one observation $y=3$, use the identity link, $x^\mathsf{T}=(1,2)$, and $\beta=(1,1/2)^\mathsf{T}$. Find the coefficient derivatives.

The identity link gives $\eta=\mu=1+2(1/2)=2$. Ignoring no terms that depend on $\mu$, the log-likelihood is $-\tfrac12\log(2\pi\sigma^2)-(y-\mu)^2/(2\sigma^2)$. Its mean derivative is $(y-\mu)/\sigma^2=(3-2)/4=1/4$. For the identity link $g'(\mu)=1$, so the derivative with respect to the predictor is also $1/4$. Consequently,

\frac{\partial\ell}{\partial\beta_1} = \frac{1}{4}(1)=\frac{1}{4}, \qquad \frac{\partial\ell}{\partial\beta_2} = \frac{1}{4}(2)=\frac{1}{2}.

This example isolates the role of the design row: the identity link contributes no additional scaling, but the coefficients still affect the likelihood through their respective design entries.

Interpreting and Checking the Derivative

The derivative is local: it describes the rate of change at the current coefficient values, with the other coefficients held fixed for a partial derivative. Its sign combines the effect of the observation on the likelihood as the mean changes with the direction in which that coefficient moves the predictor. With multiple observations, those contributions are summed, and different rows of the design matrix can contribute in opposite directions.

A useful check is to follow the chain in order rather than trying to remember a single formula:

Start with the likelihood’s mean derivative.
Compute $\partial\ell_i/\partial\mu_i$ from the specified observation model, holding other parameters fixed.

Translate from mean to predictor.
Multiply by $d\mu_i/d\eta_i$, which is $1/g'(\mu_i)$ when the inverse-link derivative is expressed through the link.

Translate from predictor to coefficient.
Multiply by $x_{ij}$, then sum over observations if differentiating the full log-likelihood.

Common Mistakes

Using the link derivative in the wrong direction. Since $g(\mu)=\eta$, the required mean change per predictor change is $d\mu/d\eta=1/g'(\mu)$, not $g'(\mu)$. Alternatively, differentiate $g^{-1}$ directly.
Evaluating the derivative at the wrong argument. In $1/g'(\mu_i)$, the link derivative is evaluated at the mean. If using $(g^{-1})'(\eta_i)$, the inverse-link derivative is evaluated at the predictor.
Forgetting the design entry. The derivative with respect to $\beta_j$ includes $x_{ij}$. Only an intercept column necessarily has entries all equal to one.
Confusing a mean derivative with a coefficient derivative. The quantity $\partial\ell_i/\partial\mu_i$ is only the first stage. It is not yet the derivative with respect to a regression coefficient.
Assuming the link factor is constant across observations. In general, $1/g'(\mu_i)$ depends on the observation’s fitted mean and varies from row to row.

A finite-change check can help catch sign and scaling errors: perturb one coefficient by a small amount, recalculate the predictor and mean, and verify that the resulting change in log-likelihood is approximately the derivative times the perturbation. This is a diagnostic, not a substitute for the chain-rule calculation.

Check Your Understanding

For each question, identify the relevant chain-rule factors before simplifying.

For a log link, what is $d\mu_i/d\eta_i$ in terms of $\mu_i$?
If $x_{ij}=0$, what is the contribution of observation $i$ to the derivative with respect to $\beta_j$, and why?
For the probit example, which factor determines the negative sign in the derivative with respect to the second coefficient?
State the matrix of derivatives of the means with respect to the regression coefficients using $D$ and $X$.
Why is it incorrect to multiply $\partial\ell_i/\partial\mu_i$ by $g'(\mu_i)$ when differentiating through the mean?