Gamma Log-Link Working Weights

Why the Gamma Log-Link Update Has a Special Simplicity

For a Gamma generalized linear model with a log link, fitted means can differ substantially across observations. Nevertheless, when all observations share a common dispersion parameter, their IRLS working weights are equal. The reason is a precise cancellation: the Gamma variance grows as the square of the mean, while the squared derivative of the mean with respect to the linear predictor grows at exactly the same rate.

This does not make the entire IRLS step independent of the fitted means. The working response still depends on each current fitted mean, and the design matrix still determines the weighted least-squares fit. What becomes constant is the weight assigned to each observation at a given iteration.

Gamma Model and Log-Link Setup

Assume $Y_1,\ldots,Y_n$ are independent Gamma observations with positive means $\mu_i=E[Y_i]$ and a common dispersion $\phi>0$. Use the mean–variance parameterization $\operatorname{Var}(Y_i)=\phi\mu_i^2$. Equivalently, $Y_i$ has shape $1/\phi$ and scale $\phi\mu_i$. The coefficient update treats $\phi$ as fixed.

Definition: In a Gamma GLM with log link, the linear predictor is $\eta_i=x_i^\mathsf{T}\beta$, and the mean is $\mu_i=\exp(\eta_i)$. The Gamma variance function is $V(\mu_i)=\mu_i^2$, so $\operatorname{Var}(Y_i)=\phi V(\mu_i)=\phi\mu_i^2$.

Here $x_i^\mathsf{T}$ is row $i$ of the design matrix $X$. At an IRLS iteration, let the current coefficient vector be $\beta$, giving $\eta_i=x_i^\mathsf{T}\beta$ and $\mu_i=\exp(\eta_i)$. The general IRLS formulas, using the current mean and link, are

z_i=\eta_i+(Y_i-\mu_i)\frac{d\eta_i}{d\mu_i}, \qquad w_i=\frac{(d\mu_i/d\eta_i)^2}{\operatorname{Var}(Y_i)}.

These formulas turn the Fisher-scoring coefficient step into a weighted least-squares fit of the working responses $z_i$ on the design matrix. The calculations below specialize them to the Gamma log link.

Deriving the Working Response and Weight

For the log link, $\eta_i=\log(\mu_i)$. Differentiating in either direction gives $d\eta_i/d\mu_i=1/\mu_i$ and $d\mu_i/d\eta_i=\mu_i$. Substitution in the working-response formula yields

z_i=\eta_i+\frac{Y_i-\mu_i}{\mu_i} =\eta_i+\frac{Y_i}{\mu_i}-1.

For the weight, use both the derivative and the Gamma variance:

w_i =\frac{\mu_i^2}{\phi\mu_i^2} =\frac{1}{\phi}.

Theorem: For independent Gamma responses with common dispersion $\phi$, a log link, and no additional observation-specific prior weights, the IRLS working response and weight at the current iteration are $$ z_i=\eta_i+\frac{Y_i-\mu_i}{\mu_i}, \qquad w_i=\frac{1}{\phi}\quad (i=1,\ldots,n). $$ Thus, for fixed common dispersion, the working weights are equal across observations, even when their current fitted means are unequal.

To establish that these formulas give the Fisher-scoring update, consider the weighted least-squares criterion at the current iteration. With $W$ the diagonal matrix whose entries are $w_i$, the coefficient increment $\delta$ solves

X^\mathsf{T}WX\delta=X^\mathsf{T}W(z-\eta), \qquad \beta_{\text{new}}=\beta+\delta.

The Gamma log likelihood also provides a direct check. For shape $1/\phi$ and scale $\phi\mu_i$, the part of the log density depending on $\mu_i$ is $-\log(\mu_i)/\phi-Y_i/(\phi\mu_i)$. Since $\log(\mu_i)=\eta_i=x_i^\mathsf{T}\beta$, differentiation gives the coefficient score

U(\beta)=\frac{1}{\phi}\sum_{i=1}^n x_i \left(\frac{Y_i}{\mu_i}-1\right).

The expected information is $I(\beta)=X^\mathsf{T}X/\phi$. In fact, differentiating the score gives the negative Hessian $\sum_i x_ix_i^\mathsf{T}Y_i/(\phi\mu_i)$, whose expectation is $X^\mathsf{T}X/\phi$, because $E[Y_i]=\mu_i$. Therefore the Fisher-scoring increment solves

\frac{1}{\phi}X^\mathsf{T}X\delta = \frac{1}{\phi}X^\mathsf{T}(Y/\mu-\mathbf{1}),

where $Y/\mu-\mathbf{1}$ denotes the vector with entries $Y_i/\mu_i-1$. In the IRLS equations, $W=I_n/\phi$, where $I_n$ is the $n$-dimensional identity matrix, and $z-\eta$ has precisely those entries. The equations are therefore identical.

Result: Under the stated assumptions, the common factor $1/\phi$ cancels from the coefficient increment. Equivalently, each iteration fits $z_i$ by ordinary least squares on $X$, although the working responses themselves depend on the current fitted means. This statement concerns the coefficient step with common dispersion; it does not assert that the response variances are equal.

Worked Calculations

Worked Example: Updating a Gamma Intercept

Suppose three independent Gamma observations are $Y=(1,2,4)$, the model has an intercept only, the common dispersion is $\phi=0.5$, and the current intercept is $\beta=\log 2$. Calculate the working responses and the updated intercept.

The current fitted mean for every observation is $\mu_i=\exp(\log 2)=2$. The working weight is $1/\phi=2$ for each observation. The responses are

z_1=\log 2+\frac{1-2}{2}=\log 2-0.5,\qquad z_2=\log 2+\frac{2-2}{2}=\log 2,\qquad z_3=\log 2+\frac{4-2}{2}=\log 2+1.

With an intercept only and equal weights, the weighted least-squares estimate is the mean of the three working responses:

\beta_{\text{new}} =\frac{z_1+z_2+z_3}{3} =\log 2+\frac{-0.5+0+1}{3} =\log 2+\frac{1}{6} \approx 0.8598.

The score and information give the same step. The score is $U=(1/\phi)\sum_i(Y_i/\mu_i-1)=2(-0.5+0+1)=1$, and the expected information for an intercept is $I=3/\phi=6$. Thus the increment is $U/I=1/6$, agreeing with the working-response calculation.

Worked Example: A Two-Coefficient Update

Take $x_i=(1,x_i)^\mathsf{T}$ with predictor values $x=(0,1,2)$, responses $Y=(1,2,4)$, common dispersion $\phi=0.5$, and current coefficients $\beta=(\log 2,0)^\mathsf{T}$. Find the Fisher-scoring increment using the IRLS equations.

All current means are $2$, so the working responses are $\log 2-0.5,\log 2,\log 2+1$, as calculated above. The design cross-product and information matrix are

X^\mathsf{T}X= \begin{pmatrix} 3&3\\ 3&5 \end{pmatrix}, \qquad I(\beta)=\frac{1}{0.5}X^\mathsf{T}X = \begin{pmatrix} 6&6\\ 6&10 \end{pmatrix}.

The residual-like vector $Y/\mu-\mathbf{1}$ is $(-0.5,0,1)^\mathsf{T}$. Consequently, the score is

U(\beta) =\frac{1}{0.5}X^\mathsf{T} \begin{pmatrix}-0.5\\0\\1\end{pmatrix} = \begin{pmatrix}1\\4\end{pmatrix}.

Solving $I(\beta)\delta=U(\beta)$ gives $6\delta_0+6\delta_1=1$ and $6\delta_0+10\delta_1=4$. Subtracting the first equation from the second gives $4\delta_1=3$, so $\delta_1=3/4$. Substitution gives $6\delta_0=1-6(3/4)=-3.5$, hence $\delta_0=-7/12$. The updated vector is therefore

\beta_{\text{new}}= \begin{pmatrix} \log 2-7/12\\ 3/4 \end{pmatrix}.

As a check, multiplying the information matrix by the increment gives $(6(-7/12)+6(3/4),\,6(-7/12)+10(3/4))^\mathsf{T}=(1,4)^\mathsf{T}$, exactly the score. Equal working weights do not imply equal coefficient changes: the design matrix and the individual working responses determine the fitted update.

Worked Example: Different Means, Equal Weights

Consider two current fitted means $\mu_1=1$ and $\mu_2=4$, with $\phi=0.2$ and observed responses $Y_1=2$, $Y_2=2$. Compute the working quantities and compare the response variances.

The current predictors are $\eta_1=\log 1=0$ and $\eta_2=\log 4$. The working responses are

z_1=0+\frac{2-1}{1}=1,\qquad z_2=\log 4+\frac{2-4}{4}=\log 4-0.5.

Both working weights equal $1/0.2=5$. But the response variances are not equal: $\operatorname{Var}(Y_1)=0.2(1)^2=0.2$, whereas $\operatorname{Var}(Y_2)=0.2(4)^2=3.2$. The corresponding derivatives $d\mu_i/d\eta_i$ are $1$ and $4$; their squares offset the differing variances in the weight formula.

Interpretation and Common Pitfall

The constant-weight property depends on both parts of the calculation: the log link gives $d\mu_i/d\eta_i=\mu_i$, and the Gamma variance is proportional to $\mu_i^2$. If the dispersion is common, the remaining factor is the same for every observation. This is a property of the Gamma log-link working regression, not a claim that the data have equal variance or that their working responses are equal.

Wrong reasoning: “The weights are constant, so the Gamma observations have equal variances.” This confuses $w_i$, the IRLS weight, with $\operatorname{Var}(Y_i)$.
Correction: The variances are $\phi\mu_i^2$ and generally differ when the means differ. The weights are constant because the numerator $(d\mu_i/d\eta_i)^2=\mu_i^2$ cancels the $\mu_i^2$ in the variance.
Another common error: Replacing $z_i$ with $Y_i$ because the working regression is a least-squares calculation. The working response is $z_i=\eta_i+(Y_i-\mu_i)/\mu_i$, recalculated from the current fitted mean at each iteration; it is not the observed response.
Check the assumptions: If observation-specific prior weights are included, or dispersion differs by observation, the resulting weights need not be equal. The constant $1/\phi$ result uses a common dispersion and no additional observation-specific weights.

Key takeaway: For the Gamma log-link GLM with common dispersion, $z_i=\eta_i+(Y_i-\mu_i)/\mu_i$ and $w_i=1/\phi$. The weights are constant across observations, while the working responses continue to reflect the current means and observed data.

Check Your Understanding

Use the current fitted means and the common dispersion specified in each question.

Starting from $w_i=(d\mu_i/d\eta_i)^2/\operatorname{Var}(Y_i)$, show which two factors cancel for the Gamma log-link model.
If $\eta_i=0.7$, $\mu_i=\exp(0.7)$, and $Y_i=1.5$, write the working response in terms of these values. What is the weight when $\phi=0.25$?
For two observations with means $1$ and $3$, and common dispersion $0.4$, calculate their response variances and their IRLS weights. Are the variances equal?
Explain why equal working weights do not imply equal working responses.
State one assumption under which the conclusion that all working weights equal $1/\phi$ would no longer apply as written.

What You'll Learn