Why Working Responses Are Not Observed Data

What a Working Response Represents

In IRLS, a weighted least-squares calculation is used to obtain a coefficient update for a generalized linear model. Its response variable is called the working response. Despite the name, it is generally not a new measurement of the outcome. It is a temporary quantity constructed from the observed response and the model’s current fitted mean to express a local update as a weighted regression.

This distinction matters because the working response can have values the outcome could never take. A Bernoulli observation is either 0 or 1, for example, but its working response can be negative or greater than 1. That is not a data error: the working response is an intermediate calculation, not a proposed replacement for the recorded outcome.

Definition: For observation $i$, let $y_i$ be the observed response, $\mu_i$ the current fitted mean, $\eta_i=g(\mu_i)$ the current linear predictor, and $g$ a differentiable link function. The IRLS working response is $$ z_i=\eta_i+(y_i-\mu_i)g'(\mu_i). $$ It is recalculated from the current fitted mean at each iteration. The corresponding working weight is $$ w_i=\frac{1}{\phi V(\mu_i)\{g'(\mu_i)\}^2}, $$ where $\operatorname{Var}(Y_i)=\phi V(\mu_i)$.

These formulas assume a GLM with independent observations, a differentiable link, and positive variance and working weights at the current fitted means. Here $g'(\mu_i)$ is the derivative of the link with respect to the mean. The working response is on the scale of the linear predictor, because it is built by taking a first-order change from $\mu_i$ to $y_i$ through the link.

Why the Weighted Regression Uses This Quantity

The formula for $z_i$ has a local interpretation. Near the current fitted mean, a change in the mean produces an approximate change in the linear predictor given by the derivative of the link. Replacing the mean by the observed response in that first-order approximation gives

g(y_i)\ \approx\ g(\mu_i)+(y_i-\mu_i)g'(\mu_i)=z_i.

This is a local linearization, not a claim that $g(y_i)$ is always defined or that $z_i$ is an observed value. In particular, a link may only be defined on the mean’s parameter space, while $y_i$ need not belong to that space. The expression for $z_i$ uses the link’s derivative at $\mu_i$, not necessarily the link evaluated at $y_i$.

The connection to Fisher scoring can be stated precisely. Suppose the model has a linear predictor $\eta_i=x_i^\mathsf{T}\beta+o_i$, where $x_i$ is the design vector and $o_i$ is a fixed offset. Let $X$ have rows $x_i^\mathsf{T}$, let $W$ be diagonal with entries $w_i$, and let $z$ collect the working responses. Assume the usual regularity conditions for the GLM score, that the current fitted means are in the interior of the mean parameter space, and that $X^\mathsf{T}WX$ is invertible. The Fisher-scoring update is the solution to the weighted least-squares equations below.

Result: With dispersion $\phi$ treated as fixed, the score and expected information for $\beta$ at the current iterate satisfy $$ U(\beta)=X^\mathsf{T}W(z-X\beta-o), \qquad \mathcal I(\beta)=X^\mathsf{T}WX. $$ Consequently, the Fisher-scoring update is the weighted least-squares fit of $z-o$ on $X$, using weights $W$.

To verify the result, the score contribution from observation $i$, by the chain rule and the GLM mean-variance relationship, is

x_i\frac{y_i-\mu_i}{\phi V(\mu_i)}\frac{d\mu_i}{d\eta_i} = x_i\frac{y_i-\mu_i}{\phi V(\mu_i)g'(\mu_i)}.

On the other hand, the definition of the working response gives

w_i(z_i-\eta_i) = \frac{1}{\phi V(\mu_i)\{g'(\mu_i)\}^2} (y_i-\mu_i)g'(\mu_i) = \frac{y_i-\mu_i}{\phi V(\mu_i)g'(\mu_i)}.

Summing the score contributions yields $U(\beta)=X^\mathsf{T}W(z-X\beta-o)$, since $\eta=X\beta+o$. Under the GLM assumptions, the expected information contribution for observation $i$ is $x_i x_i^\mathsf{T}w_i$; summing gives $\mathcal I(\beta)=X^\mathsf{T}WX$. The Fisher-scoring step adds $\mathcal I(\beta)^{-1}U(\beta)$ to the current coefficient vector, so

\beta_{\mathrm{new}} =\beta+(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W(z-X\beta-o) =(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W(z-o).

The last expression is exactly the weighted least-squares solution. The equivalence concerns the computation of a coefficient update; it does not turn $z$ into a vector of newly observed outcomes. $\square$

Working Responses Need Not Be in the Outcome’s Support

The observed response and its working response answer different questions. The observed value $y_i$ belongs to the support of the assumed response distribution. The working response $z_i$ is a linearized quantity used in an iteration and has no general requirement to remain in that support.

Worked Example: A Logistic Working Response Outside the Bernoulli Support

Consider one Bernoulli observation with $y=0$, a logit link, and current linear predictor $\eta=2$. For a logit link, $\mu=\exp(\eta)/(1+\exp(\eta))$, so

$\mu=1/(1+\exp(-2))\approx 0.880797$, and $g'(\mu)=1/\{\mu(1-\mu)\}\approx 9.5244$. The working response is therefore

$z=2+(0-0.880797)(9.5244)\approx 2-8.3891=-6.3891.$

The observed Bernoulli response, $0$, is valid. The working response, approximately $-6.3891$, is not a possible Bernoulli outcome. There is no contradiction: IRLS uses this value as a local target in a weighted regression, not as a replacement observation.

Worked Example: A Poisson Working Response Can Be Negative

For a Poisson model with log link, $g(\mu)=\log(\mu)$ and $g'(\mu)=1/\mu$. Suppose an observation has $y=0$ and the current fitted mean is $\mu=1$, giving $\eta=\log(1)=0$. Then

$z=0+(0-1)(1)=-1.$

The Poisson response support consists of the nonnegative integers, so $-1$ cannot be an observed count. Nevertheless, it is the correct working response at this iterate. The weight for this observation, when $\phi=1$ and $V(\mu)=\mu$, is $w=1/\{\mu(1/\mu)^2\}=\mu=1$. The weighted regression uses the pair $z=-1$, $w=1$ to calculate an update; the original count remains $y=0$.

Recompute the Working Response After Every Update

Both $z_i$ and $w_i$ depend on $\mu_i$, which changes when the coefficient vector changes. They therefore describe the current iteration, not fixed transformed data. After obtaining $\beta_{\mathrm{new}}$, the model gives new linear predictors and fitted means; those new means must be used to calculate new working responses and weights for the next weighted regression.

Result: At an iteration with current coefficient vector $\beta$, the working quantities are calculated using $\eta_i=x_i^\mathsf{T}\beta+o_i$ and $\mu_i=g^{-1}(\eta_i)$. Once $\beta$ is updated, the next iteration uses the new $\eta_i$ and $\mu_i$ to recalculate $z_i$ and $w_i$. Keeping the old quantities would no longer give the Fisher-scoring update at the new iterate.

This is not just a convention. The weighted-regression equations match the score and information evaluated at the current $\beta$, as shown above. If the coefficients change but $z$ and $W$ are held fixed, the regression being solved is based on the previous iterate’s means and derivatives. Its solution need not equal a Fisher-scoring update from the new coefficients.

Worked Example: The Working Response Changes with the Fitted Mean

Take a Poisson observation with $y=0$ and a log link. At a first iterate with $\mu=1$, the working response is $z=\log(1)+(0-1)/1=-1$. Suppose an update changes the fitted mean to $\mu=2$. The new linear predictor is $\eta=\log(2)$, and the newly calculated working response is

$z_{\mathrm{new}}=\log(2)+(0-2)/2\approx 0.693147-1=-0.306853.$

The observed count is still 0, but the working response changes from $-1$ to approximately $-0.306853$. The Poisson working weight changes as well: it is $1$ at $\mu=1$ and $2$ at $\mu=2$. Reusing the old response and weight would ignore the change in the current fitted mean.

When the Working Response Does Equal the Observed Response

There are special cases where a working response coincides with the observed response. The Gaussian model with identity link is a useful example. It should be treated as a special case, not as evidence that working responses generally represent observed data.

Worked Example: Gaussian Identity Link

For a Gaussian response with identity link, $g(\mu)=\mu$, so $g'(\mu)=1$. If $\operatorname{Var}(Y_i)=\phi$, then $V(\mu_i)=1$. For any current fitted mean,

$z_i=\mu_i+(y_i-\mu_i)(1)=y_i,\qquad w_i=1/\phi.$

Here the working response equals the observed response at every iteration, and the weight is constant. The equality follows from the identity link and constant variance; it does not hold generally for other links or response distributions.

Common Mistake

Wrong reasoning: “A working response is the model’s current estimate of what was observed, so it must be a valid outcome. If a Bernoulli or Poisson working response is outside the support, the calculation must be wrong.”

Correction: The observed outcome is $y_i$; the working response is $z_i$, a local computational target formed using $y_i$ and the current $\mu_i$. Its role is to make the Fisher-scoring coefficient step a weighted least-squares calculation. It need not be a valid outcome. Recalculate it, together with its weight, after updating the coefficients.

A related error is to calculate $z_i$ once and treat it as fixed data for all subsequent iterations. That is valid only when the working quantities happen to remain unchanged, as in the Gaussian identity-link example. In a general GLM, they depend on the current fitted means.

Check Your Understanding

Answer each question by distinguishing the observed response from the quantities used to compute an IRLS update.

For a Bernoulli response, must every working response lie between 0 and 1? Explain.
For a Poisson model with log link, $y=1$, and current mean $\mu=1$, calculate $z$.
Why must working responses and weights usually be recalculated after a coefficient update?
What property of the Gaussian identity-link model makes its working response equal to $y_i$?
In the Fisher-scoring weighted least-squares representation, what role does $z-o$ play?