Why Noncanonical Links Need Their Own IRLS Quantities
A generalized linear model has two distinct ingredients governing how the mean responds to predictors: the variance function describes variability at a given mean, while the inverse link describes how the mean changes with the linear predictor. For a canonical link these ingredients simplify together in the IRLS weights. For a noncanonical link, keeping them separate is essential: the inverse-link derivative appears in both the score and the working response, while the variance function appears in the weights.
We derive these quantities for independent observations, then check them in three models. Throughout, use the unit-dispersion convention so the formulas show the link and variance contributions without a common dispersion factor. The mean model is \(\eta_i=x_i^\mathsf{T}\beta\), with \(\mu_i=h(\eta_i)\), where \(h=g^{-1}\) is the inverse link.
Assume each \(Y_i\) has a regular one-parameter exponential-family likelihood with unit dispersion, mean \(\mu_i\), and variance \(\operatorname{Var}(Y_i)=V(\mu_i)>0\). Assume the support does not depend on the model parameters, and the relevant likelihood derivatives exist. The observations are independent. The link is differentiable, \(h'(\eta_i)\) exists and is nonzero at the current iterate, and the design matrix \(X\), whose \(i\)th row is \(x_i^\mathsf{T}\), has full column rank when a unique coefficient update is claimed.
Deriving the Score for a General Link
In the unit-dispersion exponential-family representation, the part of the log-likelihood depending on the mean can be written as \(y\theta-b(\theta)\), where \(\theta\) is the natural parameter, \(b'(\theta)=\mu\), and \(b''(\theta)=V(\mu)\). The chain rule first gives the derivative with respect to the mean, and then with respect to \(\beta\).
Proof. Since \(d\mu/d\theta=b''(\theta)=V(\mu)\), the inverse derivative is \(d\theta/d\mu=1/V(\mu)\). Differentiating the log-likelihood contribution \(y_i\theta_i-b(\theta_i)\) with respect to \(\theta_i\) gives \(y_i-b'(\theta_i)=y_i-\mu_i\). Therefore
Next, \(\mu_i=h(\eta_i)\) and \(\eta_i=x_i^\mathsf{T}\beta\), so \(\partial\mu_i/\partial\beta=x_i h'(\eta_i)\). Multiplying the derivatives by the chain rule yields
Summing over independent observations gives the stated score. \(\square\)
The formula already separates the two ingredients. The residual \(y_i-\mu_i\) is multiplied by \(h'(\eta_i)\), which comes from the link, and divided by \(V(\mu_i)\), which comes from the response distribution. Replacing one of these terms with the other is not a harmless notational change.
From the Score to the Working Response and Weights
At a current coefficient vector \(\beta\), define a working response \(z_i\) so that its difference from the current linear predictor is the observed residual divided by the local mean slope \(h'(\eta_i)\). Define the working weight to combine the squared slope with the inverse variance.
The working response is a local, iteration-dependent quantity. It is not generally an observed response or a new measurement. The weights are positive whenever \(V(\mu_i)>0\) and \(h'(\eta_i)\ne 0\).
Proof. Since \(\eta_i=x_i^\mathsf{T}\beta\), the working-response definition gives \(z_i-\eta_i=(y_i-\mu_i)/h'(\eta_i)\). Hence
Multiplying by \(x_i\) and summing reproduces the score just derived, so \(U(\beta)=X^\mathsf{T}W(z-X\beta)\).
For the information, the score contribution from observation \(i\) is \(x_i h'(\eta_i)(Y_i-\mu_i)/V(\mu_i)\). Its expectation is zero, and its covariance matrix is
Here the information identity applies under the stated regularity conditions; equivalently, this covariance is the expected Fisher information contribution. Independence makes the total information the sum of these contributions, giving \(I(\beta)=X^\mathsf{T}WX\).
Fisher scoring adds \(I(\beta)^{-1}U(\beta)\) to the current coefficient vector. If \(X\) has full column rank and all \(w_i>0\), then \(X^\mathsf{T}WX\) is positive definite: for any nonzero vector \(a\), \(a^\mathsf{T}X^\mathsf{T}WXa=\sum_i w_i(x_i^\mathsf{T}a)^2>0\). The update is therefore uniquely defined, and
The final expression is the weighted least-squares coefficient vector for response \(z\), design \(X\), and weights \(W\). This proves the stated IRLS representation. \(\square\)
Worked Examples
Worked Example: Poisson Responses With an Identity Link
Let \(Y\) be Poisson with mean \(\mu\), so \(V(\mu)=\mu\). Use the identity link \(g(\mu)=\mu\), whose inverse is \(h(\eta)=\eta\) and derivative is \(h'(\eta)=1\). At a valid current mean \(\mu=\eta=2\), suppose the observed count is \(y=3\).
The working weight is
The working response is
The score contribution apart from the design factor is \(h'(y-\mu)/V=1(1)/2=1/2\). The weighted working residual gives the same value: \(w(z-\eta)=(1/2)(3-2)=1/2\). Here the working response happens to equal the observed count because \(h'=1\); the weight still depends on the Poisson variance \(\mu\).
Worked Example: Gamma Responses With a Reciprocal Link
For a Gamma response with unit-dispersion variance function \(V(\mu)=\mu^2\), take \(g(\mu)=1/\mu\). Then \(h(\eta)=1/\eta\), \(h'(\eta)=-1/\eta^2=-\mu^2\). At \(\mu=2\), \(\eta=1/2\), and observed value \(y=3\), the derivative is \(h'=-4\), while the variance function is \(V(2)=4\).
The weight is
The working response is
Check the score factor against the weighted residual:
The negative inverse-link derivative is not an error: increasing \(\eta=1/\mu\) decreases the mean. The weight remains positive because it contains the squared derivative.
Worked Example: Bernoulli Responses With a Probit Link
Let \(Y\) be Bernoulli with success probability \(\mu\), so \(V(\mu)=\mu(1-\mu)\). Under the probit link, \(h(\eta)=\Phi(\eta)\), where \(\Phi\) is the standard normal cumulative distribution function; thus \(h'(\eta)=\varphi(\eta)\), its density. At \(\eta=0\), \(\mu=\Phi(0)=0.5\) and \(\varphi(0)=1/\sqrt{2\pi}\approx0.39894\). Suppose \(y=1\).
The variance is \(V(0.5)=0.5(0.5)=0.25\), so the working weight is
The working response is
The score factor is \(0.39894(0.5)/0.25\approx0.79788\). The weighted working residual is \(0.63662(1.25331-0)\approx0.79788\), matching it up to rounding. The derivative \(0.39894\) and variance \(0.25\) have distinct roles even though both affect the resulting weight.
Common Pitfall and Practical Interpretation
A frequent error is to use \(1/V(\mu_i)\) as the entire IRLS weight, or to put \(V(\mu_i)\) in the working-response denominator. That reasoning treats the link as if it had unit slope. The correct local change is \(d\mu_i=h'(\eta_i)d\eta_i\): translating a mean residual into a linear-predictor residual divides by \(h'(\eta_i)\). The score then contains one factor of \(h'\), and expressing that score as a weighted working residual requires the squared factor in \(w_i\):
The variance function does not replace the inverse-link derivative. It quantifies response variability at the fitted mean; the derivative quantifies how a change in the linear predictor moves that mean. A quick check is to multiply \(w_i\) by \(z_i-\eta_i\): the result must equal \(h'(\eta_i)(y_i-\mu_i)/V(\mu_i)\), the score contribution before multiplying by \(x_i\).
These formulas matter because they allow Fisher scoring to be implemented as successive weighted least-squares fits while respecting both the response distribution and the chosen link. They also expose when an iteration is problematic: if \(h'(\eta_i)\) is zero or undefined, the stated working response is not defined; if the mean leaves the domain of the response model, the likelihood itself is not valid there. The working quantities should therefore be evaluated at admissible current means and predictors.
Check Your Understanding
Use the noncanonical-link formulas to answer the following.
- For a model with \(h'(\eta_i)=2\), \(V(\mu_i)=5\), and residual \(y_i-\mu_i=1\), calculate \(z_i-\eta_i\), \(w_i\), and \(w_i(z_i-\eta_i)\).
- Why does a negative inverse-link derivative not imply a negative working weight?
- For a Bernoulli response, identify the variance function and explain why it is not the same quantity as the derivative under a probit link.
- State the condition on \(X\) and the weights that ensures a unique weighted least-squares coefficient update.
- Explain why a working response should not generally be interpreted as an observed outcome.