Constructing the IRLS Working Weights

Why IRLS Needs an Observation-Level Weight

In a generalized linear model, observations need not have the same response variance. Moreover, a given change in the mean response can correspond to a small or large change in the linear predictor, depending on the link. The IRLS weight combines these two features: it reflects both the response’s variance at its current mean and the sensitivity of the mean to the linear predictor.

The weight is not determined by the observed residual. At a given iteration it is calculated from the current fitted mean, the variance function, and the link. This distinction matters: the working response includes the observed response, while the weight describes how much information the model assigns to that observation at the current mean.

Definition: For observation $i$, let $\mu_i=E[Y_i]$, let $\eta_i=g(\mu_i)$ be the linear predictor, and write $q_i=d\mu_i/d\eta_i$. If $\operatorname{Var}(Y_i)=\phi V(\mu_i)$, the unscaled IRLS working weight is $$ w_i=\frac{q_i^2}{V(\mu_i)} =\frac{(d\mu_i/d\eta_i)^2}{V(\mu_i)}. $$ Here $V(\mu_i)>0$ is the variance function and $\phi>0$ is the dispersion parameter. With this convention the expected information includes the factor $1/\phi$ separately.

The derivative $q_i$ is evaluated at the current fitted mean, and the weight is nonnegative. The formula assumes a differentiable link with a locally differentiable inverse, so that $d\mu_i/d\eta_i$ exists, and a mean in the interior of the model’s parameter space where $V(\mu_i)>0$. When $q_i=0$, the formula gives zero weight; the regular, informative case has $q_i\ne 0$.

Deriving the Weight from the Likelihood

A derivation makes clear why the derivative is squared and why the variance function appears in the denominator. Suppose the observations are independent and have exponential-dispersion-family densities of the form

f(y_i;\theta_i,\phi) =\exp\left\{\frac{y_i\theta_i-b(\theta_i)}{\phi} +c(y_i,\phi)\right\},

where $b$ is twice differentiable, $\mu_i=b'(\theta_i)$, and $b''(\theta_i)=V(\mu_i)>0$. These conditions give $\operatorname{Var}(Y_i)=\phi V(\mu_i)$. Assume also that the support does not depend on the model parameters and that the differentiations used below are valid. The predictor is $\eta_i=x_i^\mathsf{T}\beta$, with $\mu_i$ determined by the inverse link.

Theorem: Under these assumptions, the contribution of observation $i$ to the expected information for its linear predictor is $w_i/\phi$. Its contribution to the expected information for $\beta$ is therefore $x_i x_i^\mathsf{T}w_i/\phi$.

Proof. Since $d\mu_i/d\theta_i=b''(\theta_i)=V(\mu_i)$, the inverse-function rule gives $d\theta_i/d\mu_i=1/V(\mu_i)$. Differentiating the log density first with respect to $\theta_i$, then with respect to $\mu_i$, yields

\frac{\partial \ell_i}{\partial\theta_i} =\frac{y_i-\mu_i}{\phi}, \qquad \frac{\partial\ell_i}{\partial\mu_i} =\frac{y_i-\mu_i}{\phi V(\mu_i)}.

By the chain rule, $\partial\mu_i/\partial\eta_i=q_i$, so the score for $\eta_i$ is

U_i(\eta_i) =\frac{\partial\ell_i}{\partial\eta_i} =\frac{q_i(y_i-\mu_i)}{\phi V(\mu_i)}.

The expected information is the variance of this score, since its expectation is zero. Here $q_i$, $\mu_i$, and $V(\mu_i)$ are fixed at the parameter value under consideration. Thus

\begin{aligned} \operatorname{Var}\{U_i(\eta_i)\} &=\frac{q_i^2}{\phi^2 V(\mu_i)^2} \operatorname{Var}(Y_i)\\ &=\frac{q_i^2}{\phi^2 V(\mu_i)^2} \phi V(\mu_i) =\frac{q_i^2}{\phi V(\mu_i)} =\frac{w_i}{\phi}. \end{aligned}

Finally, $\partial\eta_i/\partial\beta=x_i$, so the score for $\beta$ is $x_i U_i(\eta_i)$. Its covariance matrix is $x_i x_i^\mathsf{T}\operatorname{Var}\{U_i(\eta_i)\}=x_i x_i^\mathsf{T}w_i/\phi$, as claimed. $\square$

This calculation also separates the two ingredients of the weight. The variance function $V(\mu_i)$ penalizes a response that is variable at its current mean. The squared derivative $q_i^2$ measures how strongly the mean changes when the predictor changes. A large response variance tends to reduce the weight; a large absolute link sensitivity tends to increase it. Neither ingredient alone gives the full weight.

Three Worked Weight Calculations

Worked Example: A Normal Response with a Square-Root Link

Suppose $Y_i\sim N(\mu_i,\phi)$, so $V(\mu_i)=1$, and use the link $\eta_i=\sqrt{\mu_i}$, with $\mu_i>0$. At a current fitted mean of $\mu_i=4$, calculate the unscaled weight and the variance of the working response when $\phi=2$.

The inverse link is $\mu_i=\eta_i^2$. Therefore

q_i=\frac{d\mu_i}{d\eta_i}=2\eta_i=2\sqrt{\mu_i}=4, \qquad w_i=\frac{4^2}{1}=16.

The dispersion is not included in this unscaled weight. The observation’s expected information for $\eta_i$ is $w_i/\phi=16/2=8$. The working-response variance, derived below, is $\phi/w_i=2/16=1/8$. The weight is large here because the mean changes rapidly with the predictor at $\mu_i=4$, while the variance function is constant.

Worked Example: A Gamma Response with an Inverse Link

Suppose $Y_i$ has a gamma GLM distribution with mean $\mu_i>0$ and variance $\phi\mu_i^2$, so $V(\mu_i)=\mu_i^2$. Use the inverse link $\eta_i=1/\mu_i$. At $\mu_i=2$, take $\phi=0.25$, and suppose the observed response is $y_i=1.5$.

The inverse link is $\mu_i=1/\eta_i$, so

q_i=\frac{d\mu_i}{d\eta_i}=-\frac{1}{\eta_i^2}=-\mu_i^2=-4.

Consequently,

w_i=\frac{(-4)^2}{2^2}=\frac{16}{4}=4.

The response variance at this mean is $\phi V(\mu_i)=0.25(4)=1$, and the expected information for $\eta_i$ is $w_i/\phi=4/0.25=16$. The working response at this current predictor is

z_i=\eta_i+\frac{y_i-\mu_i}{q_i} =\frac12+\frac{1.5-2}{-4} =0.5+0.125=0.625.

Its variance under the model at this mean is $1/16$. The observed value $y_i$ changes $z_i$, but it does not change the weight $w_i=4$.

Worked Example: An Inverse Gaussian Response with an Identity Link

Suppose $Y_i$ follows an inverse Gaussian GLM distribution with mean $\mu_i>0$ and variance $\phi\mu_i^3$. Thus $V(\mu_i)=\mu_i^3$. Use the identity link $\eta_i=\mu_i$, and evaluate the weight at $\mu_i=2$ with $\phi=0.5$.

Because the identity link has $q_i=d\mu_i/d\eta_i=1$, the weight is

w_i=\frac{1^2}{2^3}=\frac18.

The expected information for $\eta_i$ is $w_i/\phi=(1/8)/0.5=1/4$. The response variance is $\phi V(\mu_i)=0.5(8)=4$; correspondingly, the working-response variance is $\phi/w_i=0.5/(1/8)=4$. Here the identity link contributes no change in scale, and the small weight reflects the large variance function at $\mu_i=2$.

Why the Weight Matches Working-Response Variability

The working response at a current fitted value is $z_i=\eta_i+(Y_i-\mu_i)/q_i$, for $q_i\ne0$. It is a local linear adjustment: the response residual is divided by the derivative that converts a change in predictor into a change in mean. The relationship between its variance and the IRLS weight is exact at the mean used to construct it.

Result: If $E[Y_i]=\mu_i$, $\operatorname{Var}(Y_i)=\phi V(\mu_i)$, and $z_i=\eta_i+(Y_i-\mu_i)/q_i$ with fixed $\eta_i,\mu_i,q_i$ and $q_i\ne0$, then $$ E[z_i]=\eta_i, \qquad \operatorname{Var}(z_i)=\frac{\phi}{w_i}. $$

Derivation. Since $\eta_i,\mu_i,q_i$ are fixed in this calculation and $E[Y_i]=\mu_i$,

E[z_i]=\eta_i+\frac{E[Y_i]-\mu_i}{q_i}=\eta_i.

Subtracting this expectation from $z_i$ leaves $(Y_i-\mu_i)/q_i$. Therefore

\operatorname{Var}(z_i) =\frac{\operatorname{Var}(Y_i)}{q_i^2} =\frac{\phi V(\mu_i)}{q_i^2} =\frac{\phi}{w_i}.

This is an exact variance calculation, not an approximation: the working response is an affine function of $Y_i$ when its current quantities are held fixed. It explains why observations with greater working-response variance receive less weight, and why the dispersion factor can be kept separate from the unscaled weight.

Common Mistake: Using Only the Inverse Response Variance

A tempting shortcut is to set $w_i=1/V(\mu_i)$, on the grounds that less variable observations should count more. That omits the link sensitivity. For the gamma inverse-link example, this shortcut gives $1/V(2)=1/4$, whereas the correct weight is $q_i^2/V(2)=16/4=4$. The factor $q_i^2$ cannot be dropped: it accounts for how response-scale changes translate to predictor-scale changes.

A second error is to put the observed residual into the weight. In the formula $w_i=q_i^2/V(\mu_i)$, the current mean determines both $q_i$ and $V(\mu_i)$; the observed response appears in the working response and the score, not in this observation-level weight. A residual can be large without changing the weight at a fixed current mean.

Key takeaway: Compute $q_i=d\mu_i/d\eta_i$ at the current fitted mean, square it, and divide by $V(\mu_i)$. Under the convention used here, keep $\phi$ separate: the expected information uses $w_i/\phi$, and the working response has variance $\phi/w_i$ when evaluated at its model mean.

Check Your Understanding

Use the weight definition and the assumptions stated above to answer these questions.

If $V(\mu_i)=9$ and $d\mu_i/d\eta_i=3$, what is $w_i$, and what is the expected information for $\eta_i$ when $\phi=2$?
Why does the derivative $d\mu_i/d\eta_i$ enter the weight as a square?
For a model with $q_i=2$, $V(\mu_i)=4$, and $\phi=3$, calculate the variance of the working response at its model mean.
Which quantities in the unscaled weight can depend on the current fitted mean, and does the observed residual enter the weight directly?
Explain why $1/V(\mu_i)$ alone is generally not the IRLS weight.