Working Responses and Weights in IRLS

Why IRLS Uses Working Responses and Weights

In a generalized linear model (GLM), the expected response is related to a linear predictor through a link function. Except for special cases, estimating the coefficients is not an ordinary least-squares problem. Fisher scoring provides an iterative solution: at each step, the model is locally represented by a weighted least-squares problem. The working response supplies the temporary response for that problem, and the working weight specifies how much each observation contributes to it.

These quantities depend on the current fitted means. They are recalculated as the coefficient estimates change, so they are not generally fixed properties of the observations. Understanding their construction makes the IRLS algorithm easier to interpret and helps identify errors such as using the wrong derivative of the link.

Definition: For observation \(i\), let \(y_i\) be the observed response, \(\mu_i=E[Y_i]\) its fitted mean, and \(\eta_i=g(\mu_i)\) its linear predictor, where \(g\) is a differentiable link function. Suppose \(\operatorname{Var}(Y_i)=\phi V(\mu_i)/a_i\), where \(V\) is the variance function, \(\phi\) is the dispersion parameter, and \(a_i\) is a known observation or prior weight. The IRLS working response and working weight are

z_i=\eta_i+(y_i-\mu_i)g'(\mu_i), \qquad W_i=\frac{a_i}{\phi V(\mu_i)\{g'(\mu_i)\}^2}.

Here \(g'(\mu_i)\) means the derivative of the link with respect to its mean argument. The working response is a temporary quantity for an iteration, not a new observed response.

The convention for \(a_i\) matters. If each observation has unit prior weight, take \(a_i=1\). For a binomial proportion based on \(n_i\) trials, the variance of the proportion is \(p_i(1-p_i)/n_i\), so \(a_i=n_i\). These observation weights are distinct from the working weights \(W_i\), which also depend on the current mean, link, and dispersion.

Deriving the Working Quantities

Assume independent responses with the stated mean-variance relationship and a current coefficient estimate that gives \(\eta_i=g(\mu_i)\). For a mean \(\mu\) close to \(\mu_i\), a first-order Taylor approximation gives \(g(\mu)\approx g(\mu_i)+g'(\mu_i)(\mu-\mu_i)\). Substitute the observed response \(y_i\) for the nearby mean in this local approximation. The resulting approximate linear-predictor-scale response is \(g(\mu_i)+g'(\mu_i)(y_i-\mu_i)\), which is \(z_i\).

The variance of this local response is approximately the variance of \(Y_i\) multiplied by the squared derivative:

\operatorname{Var}(z_i) \approx \{g'(\mu_i)\}^2\operatorname{Var}(Y_i) =\{g'(\mu_i)\}^2\frac{\phi V(\mu_i)}{a_i}.

The inverse of this variance is the working weight. Thus observations whose transformed responses are more precise receive greater weight. This also explains the formula’s assumptions: the approximation is local, the link must be differentiable at the fitted mean, and the stated variance model must be appropriate. A poor current fit can make the local approximation less accurate, and a zero or undefined derivative requires special numerical handling rather than blind application of the formula.

Result: If the linear predictor is \(\eta_i=x_i^\mathsf{T}\beta\), the coefficient update at an IRLS step is obtained by fitting the working responses \(z_i\) to the model’s predictor structure using weights \(W_i\). In matrix form, with design matrix \(X\), working-weight matrix \(W\), and working-response vector \(z\), the weighted least-squares normal equations are

X^\mathsf{T}WX\,\beta_{\text{new}}=X^\mathsf{T}Wz.

This is a local step based on the current fitted means; it does not mean the original GLM has become an ordinary least-squares model.

The connection to Fisher scoring can be seen from the score contribution. For a GLM under the stated variance model, the score for \(\beta\) is \(X^\mathsf{T}D(y-\mu)\), where the diagonal entries of \(D\) are \(a_i\{d\mu_i/d\eta_i\}/\{\phi V(\mu_i)\}\). Since \(d\mu_i/d\eta_i=1/g'(\mu_i)\), while \(z_i-\eta_i=g'(\mu_i)(y_i-\mu_i)\), it follows observation by observation that \(W_i(z_i-\eta_i)=a_i(y_i-\mu_i)/\{\phi V(\mu_i)g'(\mu_i)\}\). This is exactly the score contribution on the predictor scale. The weighted least-squares step therefore implements the Fisher-scoring update, rather than being an unrelated fitting procedure.

Worked Examples

Worked Example: Normal Responses With an Identity Link

A hypothetical pricing analysis uses independent normal responses with an identity link and known dispersion \(\phi=4\). Two observations have prior weights \(a_1=2\) and \(a_2=1\). Their observed values are \(y_1=10\) and \(y_2=14\), while the current fitted means are \(\mu_1=9\) and \(\mu_2=15\). Find both working responses and weights, then state which observation has greater influence in the local weighted least-squares fit.

Method. For a normal response, \(V(\mu)=1\). The identity link is \(g(\mu)=\mu\), so \(g'(\mu)=1\). Therefore \(z_i=\mu_i+(y_i-\mu_i)=y_i\), and \(W_i=a_i/\phi\).

For observation 1, the residual is \(y_1-\mu_1=10-9=1\). Thus \(z_1=9+1(1)=10\), and \(W_1=2/4=0.5\).

For observation 2, the residual is \(y_2-\mu_2=14-15=-1\). Thus \(z_2=15+1(-1)=14\), and \(W_2=1/4=0.25\).

The first observation has twice the working weight of the second, \(0.5/0.25=2\), because its prior weight is twice as large. The common mistake this example avoids is treating the working weight as necessarily one: even with an identity link, the dispersion and prior weight determine it. The result also checks correctly against the identity-link property: each working response equals its observed response.

Worked Example: Binomial Proportion With a Logit Link

A hypothetical portfolio segment has 12 policies with a specified event out of 20 trials. Use a binomial model for the observed proportion, a logit link, unit dispersion \(\phi=1\), and current predictor \(\eta=0\). Calculate the working response and working weight. Keep four decimal places in the final values.

Method. The observed proportion is \(y=12/20=0.6\). The logit link is \(g(p)=\log\{p/(1-p)\}\), whose inverse gives the current fitted probability \(p=\mu=1/(1+e^{-\eta})\). At \(\eta=0\), \(p=0.5\). The variance function for a binomial proportion is \(V(p)=p(1-p)\), and the number of trials gives prior weight \(a=20\).

The derivative of the logit link with respect to \(p\) is \(g'(p)=1/\{p(1-p)\}\). At \(p=0.5\), this is \(1/(0.5\cdot0.5)=4\). Hence \(z=0+4(0.6-0.5)=0.4000\).

The working weight is \(W=20/\{1\cdot[0.5(1-0.5)]\cdot4^2\}=20/(0.25\cdot16)=20/4=5.0000\). Equivalently, for a binomial proportion with a logit link, \(W=np(1-p)/\phi\), giving \(20(0.5)(0.5)=5\).

Check. The observed proportion exceeds the current fitted probability, so the working response \(0.4000\) exceeds the current predictor \(0\), as expected because the logit link is increasing. A common error is to use \(0.8\) for the working response. That incorrectly doubles the increment; the correct calculation is \(4(0.6-0.5)=0.4\), not \(8(0.6-0.5)\). The weight is \(5.0000\), not a value obtained by omitting the 20 trials.

Worked Example: Two Poisson Observations With a Log Link

For a hypothetical frequency model, two independent Poisson counts are \(y_1=3\) and \(y_2=8\). Use a log link, unit prior weights, and \(\phi=1\). At the current coefficient estimate, the fitted means are \(\mu_1=2\) and \(\mu_2=4\). Calculate each working response and weight, and identify the quantities passed to the weighted least-squares step. Round working responses to four decimal places.

Method. For a Poisson response, \(V(\mu)=\mu\). The log link has \(g(\mu)=\log\mu\) and \(g'(\mu)=1/\mu\). Substitution into the general formulas gives \(z_i=\log(\mu_i)+(y_i-\mu_i)/\mu_i\) and \(W_i=\mu_i\) under unit prior weight and unit dispersion.

For the first observation, \(\eta_1=\log 2=0.6931\), and the residual is \(3-2=1\). Therefore \(z_1=\log 2+1/2=0.693147+0.5=1.1931\), and \(W_1=2\).

For the second observation, \(\eta_2=\log 4=1.3863\), and the residual is \(8-4=4\). Therefore \(z_2=\log 4+4/4=1.386294+1=2.3863\), and \(W_2=4\).

The local weighted least-squares data are thus responses \((1.1931,2.3863)\) with weights \((2,4)\). For example, the weighted response contributions are \(2(1.193147)=2.386294\) and \(4(2.386294)=9.545176\), before rounding. These are inputs to the coefficient step; calculating a new coefficient estimate would additionally require the design matrix and predictor structure.

Check. Since both observed counts exceed their fitted means, each working response is above its current predictor. A common mistake is to use \(g'(\eta)=1/\eta\); the derivative required here is \(g'(\mu)=1/\mu\), because the link is differentiated with respect to the mean. The resulting weights also check against \(W_i=\mu_i\) for this Poisson log-link setup.

How Link and Variance Determine the Weights

The three examples show that the same observed residual can lead to different working quantities under different models. The response is adjusted by the link derivative, while the weight reflects both the response variance and that derivative. Two familiar cases illustrate how these pieces combine:

\begin{aligned} \text{Poisson with log link:}\quad & z_i=\log(\mu_i)+\frac{y_i-\mu_i}{\mu_i}, & W_i&=\frac{a_i\mu_i}{\phi},\\ \text{Binomial proportion with logit link:}\quad & z_i=\eta_i+\frac{y_i-p_i}{p_i(1-p_i)}, & W_i&=\frac{a_i p_i(1-p_i)}{\phi}. \end{aligned}

These simplified forms assume the variance functions and prior weights stated above. They should not be transferred to a different link or response definition without re-deriving them. In particular, a binomial count and a binomial proportion have different variance expressions unless the number of trials is incorporated as a prior weight.

A working weight is a local precision, not a permanent credibility weight or a rating relativity. It can change from iteration to iteration because the fitted mean changes. In a model with a nonlinear link, observations may receive different working weights even when their prior weights are equal. The weighted least-squares calculation is therefore a computational step for estimating the GLM coefficients, not a new actuarial weighting scheme.

Common Exam Traps

Differentiating with respect to the wrong quantity. Use \(g'(\mu_i)\), the derivative of \(g\) with respect to the mean, not the derivative with respect to the predictor.
Dropping the squared derivative in the weight. The weight contains \(\{g'(\mu_i)\}^2\), because variance scales by the square of a derivative.
Forgetting the trial count for a binomial proportion. With \(n_i\) trials, \(a_i=n_i\); omitting it understates the working weight by a factor of \(n_i\).
Confusing the pseudo-response with observed data. \(z_i\) can lie outside the original response range. It is used for the current weighted least-squares step, not interpreted as a plausible observed probability or count.
Assuming weights remain fixed. The working weights are recalculated from the current means; fixed prior weights alone do not make the IRLS weights fixed.

Key takeaway: Linearize the link to obtain \(z_i=\eta_i+(y_i-\mu_i)g'(\mu_i)\), and invert the approximate variance of that transformed response to obtain \(W_i=a_i/\{\phi V(\mu_i)[g'(\mu_i)]^2\}\). Together, they make each Fisher-scoring iteration a weighted least-squares step.

Check Your Understanding

For each problem, calculate the requested working quantities using the stated current fitted mean and model assumptions.

A Poisson observation has \(y=5\), current predictor \(\eta=\log 4\), unit prior weight, and \(\phi=1\). Under a log link, find \(z\) and \(W\). (Answer: \(z=1.6363,\ W=4.0000\).)
A binomial proportion has 18 successes in 30 trials. With logit link, current predictor \(\eta=\log 2\), and \(\phi=1\), find \(z\) and \(W\). (Answer: \(z=0.3931,\ W=6.6667\).)
A normal response has \(y=7\), current mean \(\mu=9\), identity link, prior weight \(a=3\), and known dispersion \(\phi=2\). Find \(z\) and \(W\). (Answer: \(z=7.0000,\ W=1.5000\).)
A binomial proportion has 10 successes in 16 trials. With logit link, current predictor \(\eta=\log 3\), and \(\phi=1\), find \(z\) and \(W\). (Answer: \(z=0.4319,\ W=3.0000\).)
A Poisson count is \(y=2\), with current predictor \(\eta=\log 5\), unit prior weight, and \(\phi=1\). Under a log link, find \(z\) and \(W\), and state whether the working response is above or below the current predictor. (Answer: \(z=1.0094,\ W=5.0000\); it is below the predictor \(\log 5\approx1.6094\).)

What You'll Learn