The IRLS Working Response

Why a GLM Uses a Working Response

In a generalized linear model, the observed response $y_i$ is on the mean scale, while the model’s linear predictor $\eta_i$ is on a scale determined by the link. Except for an identity link, changing the linear predictor does not change the fitted mean in a linear way. Fisher scoring handles this nonlinearity by replacing the mean function, locally, with its tangent at the current iterate.

The resulting working response is a value on the linear-predictor scale. It is chosen so that the tangent approximation to the mean reaches the observed response. The construction is local: it does not say that the original mean function reaches the response at that working-response value.

Definition: At iteration $t$, let the current linear predictor for observation $i$ be $\eta_i^{(t)}$, and let $\mu_i^{(t)}=\mu_i(\eta_i^{(t)})$, where $\mu_i(\eta)$ denotes the model mean as a function of the linear predictor. If $\mu_i$ is differentiable at $\eta_i^{(t)}$ and $\left.\frac{d\mu_i}{d\eta}\right|_{\eta_i^{(t)}}\ne 0$, the IRLS working response is $$ z_i^{(t)} = \eta_i^{(t)} + \frac{y_i-\mu_i^{(t)}}{\left.\dfrac{d\mu_i}{d\eta}\right|_{\eta_i^{(t)}}}. $$

Deriving the Formula from a Tangent Line

Fix one observation and temporarily omit its subscript. Write the mean function as $\mu(\eta)$, and suppose the current predictor is $\eta^{(t)}$. For a candidate predictor $\eta$ near $\eta^{(t)}$, its first-order Taylor approximation is

\mu(\eta) \approx \mu^{(t)}+ \left.\frac{d\mu}{d\eta}\right|_{\eta^{(t)}}(\eta-\eta^{(t)}).

The working response is the predictor value at which this tangent line equals the observed response $y$. Set the right-hand side equal to $y$, then solve for $\eta$:

y = \mu^{(t)}+ \left.\frac{d\mu}{d\eta}\right|_{\eta^{(t)}}(\eta-\eta^{(t)}) \quad\Longrightarrow\quad \eta = \eta^{(t)} + \frac{y-\mu^{(t)}}{\left.\dfrac{d\mu}{d\eta}\right|_{\eta^{(t)}}}.

This solution is $z^{(t)}$. The nonzero-derivative condition matters: it ensures the tangent line is not horizontal, so there is a unique predictor value at which the line equals $y$. For a differentiable inverse link $g^{-1}$, the derivative can also be written as $\frac{d\mu}{d\eta}=\frac{d}{d\eta}g^{-1}(\eta)$, evaluated at the current predictor.

Result: The working response is the unique value $z^{(t)}$ satisfying $$ y=\mu^{(t)}+\left.\frac{d\mu}{d\eta}\right|_{\eta^{(t)}}(z^{(t)}-\eta^{(t)}). $$ This is an exact identity for the tangent line, not generally for the original mean function.

What the Working Response Represents

The distinction between the tangent line and the original mean function is essential. The working response is calculated from the observed response and the current fitted mean. It is not, in general, a new observation from the model, a transformed version of $y$ that is valid at every iteration, or a value satisfying $\mu(z^{(t)})=y$.

For a GLM with $\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}+o_i$, where $o_i$ is a known offset, the predictor changes linearly with the coefficient vector. Around the current value $\boldsymbol{\beta}^{(t)}$,

\eta_i = \eta_i^{(t)}+\mathbf{x}_i^\mathsf{T}(\boldsymbol{\beta}-\boldsymbol{\beta}^{(t)}).

Using this change in the tangent approximation gives

\mu_i(\eta_i) \approx \mu_i^{(t)} + \left.\frac{d\mu_i}{d\eta}\right|_{\eta_i^{(t)}} \mathbf{x}_i^\mathsf{T}(\boldsymbol{\beta}-\boldsymbol{\beta}^{(t)}).

By the definition of $z_i^{(t)}$, the same approximation can be written as

\mu_i(\eta_i) \approx \mu_i^{(t)} + \left.\frac{d\mu_i}{d\eta}\right|_{\eta_i^{(t)}} \left[ \mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}+o_i-\eta_i^{(t)} \right], \qquad z_i^{(t)}=o_i+\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta} \ \text{at the tangent target.}

The last relation says that the working response provides a linear-predictor target for the current local approximation. Across observations, those targets need not correspond exactly to one coefficient vector; the iterative procedure uses them to update the model. The construction of the accompanying weights and the resulting weighted least-squares system are separate steps.

Worked Examples

Worked Example: Gaussian Identity Link

Suppose $Y$ has a Gaussian GLM mean $\mu=\eta$, and for one observation the current predictor is $\eta^{(t)}=3.7$ while the observed response is $y=4.2$. Find the working response.

For the identity link, $\mu(\eta)=\eta$, so $\mu^{(t)}=3.7$ and $\frac{d\mu}{d\eta}=1$. Therefore,

z^{(t)} = 3.7+\frac{4.2-3.7}{1} = 3.7+0.5 = 4.2.

Here the tangent line is the mean function itself, so the working response equals the observation exactly. This exactness is a consequence of the identity link, not a general property of working responses.

Worked Example: A Bernoulli Observation with a Logit Link

For a Bernoulli response with a logit link, $\mu(\eta)=p(\eta)=\frac{1}{1+\exp(-\eta)}$. Suppose the current predictor is $\eta^{(t)}=0$ and the observed outcome is $y=1$. Compute the working response and check the tangent-line calculation.

At $\eta^{(t)}=0$, the fitted probability is $p^{(t)}=\frac12$. Differentiating the logistic mean gives $\frac{dp}{d\eta}=p(1-p)$, so the derivative at the current predictor is $\frac12(1-\frac12)=\frac14$. Thus

z^{(t)} = 0+\frac{1-\frac12}{\frac14} = \frac{\frac12}{\frac14} = 2.

The tangent line at zero is $p\approx\frac12+\frac14(\eta-0)$. At $\eta=z^{(t)}=2$, that line gives $\frac12+\frac14(2)=1$, as required by the definition. The original logistic mean at $2$, however, is $\frac{1}{1+\exp(-2)}\approx0.881$, not $1$. This calculation shows why $z^{(t)}$ is a target for the tangent approximation rather than an exact inverse-link transform of the observed outcome.

Worked Example: A Poisson Observation with a Log Link

For a Poisson GLM with a log link, $\mu(\eta)=\exp(\eta)$. Suppose $y=3$ and the current predictor is $\eta^{(t)}=\log 2$. Find $z^{(t)}$, and verify the tangent-line equality.

The fitted mean is $\mu^{(t)}=\exp(\log 2)=2$. The derivative is $\frac{d\mu}{d\eta}=\exp(\eta)$, which is $2$ at the current predictor. Hence

z^{(t)} = \log 2+\frac{3-2}{2} = \log 2+\frac12 \approx 1.1931.

The tangent line evaluated at this value is $2+2(z^{(t)}-\log 2)=2+2(\frac12)=3$, exactly the observed response. But the original mean at that predictor is $\exp(z^{(t)})=2\exp(1/2)\approx3.297$, which differs from $3$. Again, the tangent construction is exact for the local linear approximation, not for the nonlinear mean curve.

Worked Example: Grouped Binomial Proportion

Suppose a group contains $m=10$ trials, with $y=7$ successes. Use the observed proportion $y/m$ as the response in a logit mean model. If the current predictor is $0$, calculate the working response.

The response on the mean scale is $\bar y=7/10=0.7$, and the current fitted proportion is $p^{(t)}=0.5$. The derivative of the fitted proportion with respect to the predictor is $p^{(t)}(1-p^{(t)})=0.25$. Therefore,

z^{(t)} = 0+\frac{0.7-0.5}{0.25} = \frac{0.2}{0.25} = 0.8.

The tangent line gives $0.5+0.25(0.8)=0.7$, matching the observed proportion. Using the count $7$ instead of the proportion $0.7$ in this calculation would change the response scale and would not represent the stated mean model.

Local Accuracy and Its Limits

The working-response formula is based on a first-order approximation, so its accuracy depends on how well the tangent represents the mean function near the current predictor. This can be made precise. Suppose $\mu$ is twice continuously differentiable on an interval containing $\eta^{(t)}$ and $z^{(t)}$, and $|\mu''(\eta)|\le M$ throughout that interval. Taylor’s theorem gives

\mu(z^{(t)}) = \mu^{(t)} + \mu'(\eta^{(t)})(z^{(t)}-\eta^{(t)}) + R, \qquad |R|\le\frac{M}{2}|z^{(t)}-\eta^{(t)}|^2.

The definition of $z^{(t)}$ makes the first two terms on the right equal to $y$. Consequently,

|\mu(z^{(t)})-y| \le \frac{M}{2}|z^{(t)}-\eta^{(t)}|^2.

This bound explains the word “local”: when the working response lies close to the current predictor and the curvature is controlled, the original mean at the working response is close to $y$. If the response is far from the current fitted mean, the working response can be far from the current predictor, and the tangent approximation may be poor there. The inequality does not guarantee a small error in that case.

Common Mistakes

Treating $z^{(t)}$ as an exact transformed outcome. The formula solves for where the tangent line reaches $y$, not where the original mean function reaches $y$. The logistic example gives $z^{(t)}=2$, although the logistic mean at $2$ is about $0.881$, not $1$.
Putting the derivative in the numerator. The tangent change in the mean is $\mu'(\eta^{(t)})(\eta-\eta^{(t)})$. Solving for the predictor change requires division by $\mu'(\eta^{(t)})$.
Using the wrong response scale. For a grouped binomial model written for the observed proportion, use the proportion in $y-\mu^{(t)}$, not the number of successes.
Ignoring a zero derivative. If $\mu'(\eta^{(t)})=0$, the formula is undefined; the tangent line is horizontal and cannot be inverted by this construction.
Assuming the working response stays fixed. It depends on the current predictor and fitted mean, so it is recalculated as the iteration changes.

Key takeaway: The IRLS working response is the predictor-scale value at which the current tangent approximation to the mean equals the observed response. Its role is local and iterative; it is not generally an exact transformation of the data.

Check Your Understanding

Use the tangent-line definition to answer the following questions.

Starting from the first-order Taylor approximation to $\mu(\eta)$, derive the formula for $z^{(t)}$.
For a mean function with identity link, what is the working response, and why is it exact in this case?
For a log-link model with current predictor $\eta^{(t)}=\log 4$, fitted mean $4$, and observed response $y=5$, calculate the working response.
Why does a working response from a logit model not have to lie between zero and one?
What condition on the derivative of the mean function is needed for the stated working-response formula to be defined, and what does that condition ensure?

What You'll Learn