Why the Link Changes the IRLS Weights
In a generalized linear model, iteratively reweighted least squares (IRLS) turns an update for the linear predictor into a weighted least-squares problem. For a Gamma response, the variance grows as the square of the mean. But the weights do not depend on the variance alone: they also depend on how the chosen link changes when the mean changes. Consequently, log, identity, and reciprocal-mean links give different weights, even when the fitted mean is the same.
We derive the working response and weights, then compare the three links. The calculations concern an IRLS update at a current set of fitted means; they do not imply that every update is automatically valid. In particular, identity and reciprocal links require attention to their domains.
Gamma IRLS Quantities
Let \(Y_i\) be independent Gamma responses with mean \(\mu_i>0\) and variance \(\operatorname{Var}(Y_i)=\phi\mu_i^2/a_i\), where \(\phi>0\) is the common dispersion and \(a_i>0\) is a prior weight. Let \(\eta_i=x_i^\mathsf{T}\beta=g(\mu_i)\), with \(g\) the link and \(x_i\) the covariate vector. Write \(\mu_i' = d\mu_i/d\eta_i\), evaluated at the current fitted value.
The working response \(z_i\) is a temporary target on the linear-predictor scale, not a new observation from a Gamma distribution. The weight \(W_i\) describes the information in that local target. In general GLM notation the denominator is the variance function; here the Gamma variance function is \(V(\mu)=\mu^2\), with the dispersion and prior weight accounted for separately.
Deriving the Working Response and Weight
The derivation uses the Gamma score and Fisher information for the linear predictor. For one observation with prior weight \(a_i\), the score for \(\eta_i\) is $$ U_i(\eta_i)=\frac{a_i}{\phi}\frac{(y_i-\mu_i)\mu_i'}{\mu_i^2}. $$ Indeed, in the exponential-dispersion representation, differentiating the log-likelihood with respect to \(\mu_i\) gives \(a_i(y_i-\mu_i)/(\phi\mu_i^2)\); multiplying by \(d\mu_i/d\eta_i\) gives the displayed score. The expected information is $$ \mathcal I_i(\eta_i)=\operatorname{Var}\{U_i(\eta_i)\} =\frac{a_i}{\phi}\frac{(\mu_i')^2}{\mu_i^2}. $$ The variance equality follows directly from \(\operatorname{Var}(Y_i)=\phi\mu_i^2/a_i\). Independence makes the information matrix for \(\beta\) the sum of the individual contributions \(\mathcal I_i x_i x_i^\mathsf{T}\).
To verify the result, collect the current predictors into \(\eta=X\beta\), and let \(W\) be the diagonal matrix with entries \(W_i\). The score for \(\beta\) is \(X^\mathsf{T}W(z-\eta)\): its \(i\)th contribution is \(x_iW_i(z_i-\eta_i)=x_iU_i(\eta_i)\). The Fisher information matrix is \(X^\mathsf{T}WX\). Therefore the Fisher-scoring step is $$ \beta_{\mathrm{new}} =\beta+(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W(z-X\beta) =(X^\mathsf{T}WX)^{-1}X^\mathsf{T}Wz. $$ The last expression is exactly the weighted least-squares coefficient estimate. Nonsingularity ensures this update is uniquely defined. This derivation is local: \(z\) and \(W\) are recalculated from the current fit at each iteration.
Weights for the Three Links
The Gamma mean and variance function stay the same across links; what changes is \(\mu'=d\mu/d\eta\). We use the reciprocal-mean convention \(g(\mu)=1/\mu\), not the canonical negative-inverse convention \(g(\mu)=-1/\mu\).
| Link | \(\eta\) and \(\mu'\) | Working response \(z_i\) | Weight without \(1/\phi\), for \(a_i=1\) |
|---|---|---|---|
| Log: \(g(\mu)=\log\mu\) | \(\eta=\log\mu,\ \mu'=\mu\) | \(\eta+(y-\mu)/\mu\) | \(1\) |
| Identity: \(g(\mu)=\mu\) | \(\eta=\mu,\ \mu'=1\) | \(\eta+(y-\mu)=y\) | \(1/\mu^2\) |
| Reciprocal mean: \(g(\mu)=1/\mu\) | \(\eta=1/\mu,\ \mu'=-\mu^2\) | \(\eta-(y-\mu)/\mu^2\) | \(\mu^2\) |
For example, the reciprocal derivative follows by differentiating \(\mu=1/\eta\): \(d\mu/d\eta=-1/\eta^2=-\mu^2\). Thus a positive residual \(y-\mu\) moves the reciprocal-link working response downward. The sign is important: because increasing \(\eta\) decreases \(\mu\), a downward movement in \(\eta\) corresponds locally to an increase in the mean.
Worked Example: Computing the Log-Link Quantities
Suppose one observation has \(y=6\), current fitted mean \(\mu=4\), prior weight \(a=1\), and dispersion \(\phi=0.25\). Under the log link, \(\eta=\log 4\) and \(\mu'=\mu=4\). The working response is $$ z=\log 4+\frac{6-4}{4}=\log 4+0.5\approx1.8863. $$
The weight without the dispersion factor is \((\mu')^2/\mu^2=4^2/4^2=1\). Including dispersion gives \(W=1/0.25=4\). A positive residual increases the working predictor, as expected for a link whose mean increases with its predictor. The working response is on the log-mean scale; it is not the observed response \(6\).
Worked Example: Computing the Identity-Link Quantities
Keep \(y=6\), \(\mu=4\), \(a=1\), and \(\phi=0.25\), but use the identity link. Now \(\eta=4\) and \(\mu'=1\), so $$ z=4+\frac{6-4}{1}=6. $$
The weight without dispersion is \(1^2/4^2=1/16=0.0625\). The likelihood-scaled weight is \(W=0.0625/0.25=0.25\). The working response equals the observed response in this case because the identity predictor is the mean itself. The calculation is valid at this current fit because \(\mu=4>0\); an identity-link model must still ensure that fitted means remain positive.
Worked Example: Computing the Reciprocal-Mean Quantities
Again take \(y=6\), \(\mu=4\), \(a=1\), and \(\phi=0.25\). Under the reciprocal-mean link, \(\eta=1/4=0.25\) and \(\mu'=-4^2=-16\). Hence $$ z=0.25+\frac{6-4}{-16}=0.25-0.125=0.125. $$
The weight without dispersion is \((-16)^2/4^2=256/16=16\), so the likelihood-scaled weight is \(W=16/0.25=64\). The response exceeds the fitted mean, but the working predictor decreases: this is correct because the reciprocal link is decreasing in the mean. The working predictor \(0.125\) remains in the reciprocal link’s positive domain.
How the Weights Depend on the Fitted Mean
For equal prior weights, comparing at the same fitted mean isolates the effect of the link. The log-link weight without dispersion is constant; the identity-link weight decreases as the mean grows; and the reciprocal-mean weight increases as the mean grows. The likelihood-scaled weights have the same pattern, multiplied by \(1/\phi\).
Worked Example: Comparing Weights at Three Means
Take unit prior weights and compare the weights without \(1/\phi\) at \(\mu=2,4,8\). For the log link the weight is always 1. For the identity link it is \(1/\mu^2\); for the reciprocal-mean link it is \(\mu^2\).
| Fitted mean \(\mu\) | Log link | Identity link | Reciprocal-mean link |
|---|---|---|---|
| 2 | 1 | \(1/4=0.25\) | \(4\) |
| 4 | 1 | \(1/16=0.0625\) | \(16\) |
| 8 | 1 | \(1/64=0.015625\) | \(64\) |
As \(\mu\) doubles from 2 to 4, the identity weight is divided by 4 and the reciprocal weight is multiplied by 4. Doubling again from 4 to 8 repeats those changes. If \(\phi=0.25\), every entry in the table is multiplied by \(1/\phi=4\); for instance, at \(\mu=4\) the three likelihood-scaled weights are \(4\), \(0.25\), and \(64\).
One Fisher-Scoring Step in an Intercept-Only Model
The weights determine how the working responses are combined in the update. Consider two independent observations with responses \(y_1=2\) and \(y_2=8\), an intercept-only log-link model, unit prior weights, and current common fitted mean \(\mu_1=\mu_2=4\). The current predictor is \(\eta=\log 4\). Since both log-link weights are equal, the updated intercept is the ordinary average of the two working responses: $$ z_1=\log 4+\frac{2-4}{4}=\log 4-0.5,\qquad z_2=\log 4+\frac{8-4}{4}=\log 4+1, $$ $$ \eta_{\mathrm{new}}=\frac{z_1+z_2}{2} =\log 4+\frac{0.5}{2} =\log 4+0.25. $$
The updated mean is \(\mu_{\mathrm{new}}=\exp(\eta_{\mathrm{new}})=4\exp(0.25)\approx5.136\). The common dispersion factor would multiply both weights and so would not change this intercept update. This is one Fisher-scoring step from the specified starting value, not a claim that the step itself reaches the maximum-likelihood estimate.
Common Mistakes and Practical Checks
- Using only the variance function. The incorrect argument is that \(V(\mu)=\mu^2\), so every link must have weights proportional to \(1/\mu^2\). The correction is that the IRLS weight also contains \((d\mu/d\eta)^2\). This derivative cancels the mean dependence for the log link and reverses it for the reciprocal-mean link.
- Dropping the reciprocal-link sign in the working response. For \(g(\mu)=1/\mu\), \(\mu'=-\mu^2\). Replacing it by \(+\mu^2\) gives the wrong direction of adjustment. The squared derivative in the weight hides the sign, but the working response does not.
- Mixing up link conventions. The reciprocal-mean link here is \(1/\mu\). The canonical negative-inverse link is \(-1/\mu\). Their derivatives have opposite signs, so their working responses have opposite signs in the residual adjustment, even though the squared derivative gives the same formula for the weight as a function of \(\mu\).
- Confusing scaled and unscaled weights. Omitting \(1/\phi\) is harmless for the weighted least-squares coefficient update when dispersion is common, but the resulting numbers are not the likelihood-scaled information weights. State which convention is being used.
- Ignoring admissible means. Gamma means must be positive. The log link ensures positive means for every finite predictor. Identity-link fits must be checked for positive means, while the reciprocal-mean link requires positive predictors so that its implied means are positive.
The distinction between the links matters computationally: it determines how much the local update emphasizes observations at different fitted means. It also makes clear why an IRLS weight is not a general measure of an observation’s importance independent of the model. It is the information for a particular local predictor, under a particular link and fitted mean.
Check Your Understanding
Use the reciprocal-mean convention \(g(\mu)=1/\mu\) and distinguish likelihood-scaled weights from weights that omit \(1/\phi\).
- For a Gamma response with unit prior weight, derive the log-link working response and weight from \(d\mu/d\eta=\mu\).
- At \(\mu=5\), what are the identity-link and reciprocal-mean weights without the dispersion factor?
- Under the reciprocal-mean link, explain why a positive residual lowers the working predictor.
- If every observation has the same dispersion \(\phi\), why does including \(1/\phi\) not change the weighted least-squares coefficient update?
- Which of the three links guarantees a positive fitted mean for every finite linear predictor, and what domain check is needed for the reciprocal-mean link?