Offsets in the IRLS Working Regression

Why an Offset Changes the Working Regression

A generalized linear model sometimes includes a known contribution to the linear predictor that should not be estimated. For example, a count model may use log exposure as a known adjustment, so that observations with different exposure times are compared on a rate scale. The regression coefficients still describe the unknown part of the predictor, but the known part must be accounted for at every IRLS iteration.

The central point is simple: the usual working response approximates the full linear predictor. When that predictor is the sum of a regression term and an offset, the weighted regression for the coefficients must either include the offset as a fixed contribution or subtract it from the working response. These are equivalent ways to write the same update.

Definition: For observations \(i=1,\ldots,n\), a GLM with a known offset has linear predictor \(\eta_i=x_i^\mathsf{T}\beta+o_i\), where \(x_i\) is a fixed design vector, \(\beta\) is the unknown coefficient vector, and \(o_i\) is a specified, known number. The offset is held fixed during estimation; it is not an additional coefficient.

Assume independent responses with mean \(\mu_i\), variance \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\), and a differentiable link \(\eta_i=g(\mu_i)\). Here \(\phi\) is a common dispersion parameter treated as fixed for the coefficient update. Write \(\mu_{\eta,i}=d\mu_i/d\eta_i\), evaluated at the current estimate. We assume \(V(\mu_i)>0\) and \(\mu_{\eta,i}\ne 0\) at the current iterate. The derivation does not require a canonical link.

Deriving the Offset-Adjusted Update

For an exponential-dispersion GLM, the derivative of the \(i\)th log-likelihood contribution with respect to \(\mu_i\) is \((y_i-\mu_i)/\{\phi V(\mu_i)\}\). Since \(d\mu_i/d\beta=\mu_{\eta,i}x_i\), the coefficient score is

U(\beta)=\sum_{i=1}^n x_i\frac{\mu_{\eta,i}(y_i-\mu_i)}{\phi V(\mu_i)}.

The offset does not appear as a separate derivative: it is fixed, and \(d\eta_i/d\beta=x_i\). Define the working response and working weight at the current iterate by

z_i=\eta_i+\frac{y_i-\mu_i}{\mu_{\eta,i}}, \qquad w_i=\frac{\mu_{\eta,i}^2}{\phi V(\mu_i)}.

These are evaluated using the current fitted mean and predictor, including the known offset in \(\eta_i=x_i^\mathsf{T}\beta+o_i\). The key identity is \(w_i(z_i-\eta_i)=\mu_{\eta,i}(y_i-\mu_i)/\{\phi V(\mu_i)\}\).

Theorem: Suppose the observations are independent, the mean and variance satisfy the GLM assumptions above, the required derivatives exist, and the expected information matrix is nonsingular at the current iterate. The Fisher-scoring update for \(\beta\) is the solution of \((X^\mathsf{T}WX)\beta_{\mathrm{new}}=X^\mathsf{T}W(z-o)\), where \(X\) has rows \(x_i^\mathsf{T}\), \(W=\operatorname{diag}(w_1,\ldots,w_n)\), \(z=(z_1,\ldots,z_n)^\mathsf{T}\), and \(o=(o_1,\ldots,o_n)^\mathsf{T}\).

To prove the result, first compute the expected information. Independence makes the variance of the score the sum of the individual score variances. Since \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\),

\operatorname{Var}\!\left[ x_i\frac{\mu_{\eta,i}(Y_i-\mu_i)}{\phi V(\mu_i)} \right] =x_ix_i^\mathsf{T}\frac{\mu_{\eta,i}^2}{\phi V(\mu_i)}.

The score has mean zero under the model, so its variance is the expected information. Summing gives \(I(\beta)=X^\mathsf{T}WX\). Also, from the working-response identity,

U(\beta)=X^\mathsf{T}W(z-\eta) =X^\mathsf{T}W\{z-(X\beta+o)\}.

Fisher scoring adds \(I(\beta)^{-1}U(\beta)\) to the current coefficient vector. Equivalently, \(\beta_{\mathrm{new}}\) solves \(I(\beta)\beta_{\mathrm{new}}=I(\beta)\beta+U(\beta)\). Substituting the expressions above yields

X^\mathsf{T}WX\beta_{\mathrm{new}} =X^\mathsf{T}WX\beta+X^\mathsf{T}W\{z-X\beta-o\} =X^\mathsf{T}W(z-o).

Nonsingularity ensures this system has a unique solution. This proves the update. \(\square\)

Equivalent working-regression forms: Regress \(z-o\) on the columns of \(X\) using weights \(w_i\); or regard \(z\) as the response and include \(o\) as a fixed offset in the weighted least-squares problem. Do not estimate a coefficient for \(o\).

The equivalence follows directly from the weighted residual sum of squares. With \(W\) positive and \(X^\mathsf{T}WX\) nonsingular, minimizing \(\sum_i w_i(z_i-x_i^\mathsf{T}b-o_i)^2\) over \(b\) gives the same normal equations as minimizing \(\sum_i w_i\{(z_i-o_i)-x_i^\mathsf{T}b\}^2\). In either form, differentiation with respect to \(b\) gives \(X^\mathsf{T}WXb=X^\mathsf{T}W(z-o)\). Thus the offset changes the target being fitted, not the design matrix or the set of unknown coefficients. \(\square\)

Worked Examples

Worked Example: Gaussian Identity Model With an Offset

Consider a normal model with identity link, variance \(\phi=1\), and an intercept-only regression term. The three responses are \(y=(5,7,8)\), and the known offsets are \(o=(2,4,3)\). Start from \(\beta^{(0)}=0\).

For the identity link, \(\eta_i=\mu_i\), so \(\mu_{\eta,i}=1\) and \(V(\mu_i)=1\). Thus each working weight is \(w_i=1\), and \(z_i=\eta_i+(y_i-\mu_i)=y_i\), regardless of the current fitted value. The offset-adjusted working responses are therefore

z-o=(5-2,\;7-4,\;8-3)=(3,3,5).

The design matrix is a column of ones, so the weighted least-squares update is their ordinary mean:

\beta_{\mathrm{new}}=\frac{3+3+5}{1+1+1}=\frac{11}{3}.

The same result comes from fitting \(z=(5,7,8)\) with the fixed offset \((2,4,3)\): the fitted values for the regression term must be the residual targets \((3,3,5)\). Regressing \(z\) on an intercept alone while ignoring the offset would instead give \(20/3\), which solves a different problem.

Worked Example: Poisson Log-Link With Log Exposure

Suppose two independent Poisson counts have means \(\mu_i=\exp(\beta+o_i)\), with known exposures \(t=(1,3)\) and offsets \(o_i=\log(t_i)\). Take observed counts \(y=(2,6)\), and start from \(\beta^{(0)}=0\). Then \(\eta=(0,\log 3)\) and \(\mu=(1,3)\).

For a Poisson log-link model, \(V(\mu)=\mu\) and \(\mu_\eta=\mu\). With \(\phi=1\), the weights are \(w_i=\mu_i\), giving \(w=(1,3)\). The working responses are

z_1=0+\frac{2-1}{1}=1, \qquad z_2=\log 3+\frac{6-3}{3}=\log 3+1.

Subtract the offsets: \(z-o=(1,1)\). For an intercept-only coefficient, the weighted least-squares update is

\beta_{\mathrm{new}} =\frac{1(1)+3(1)}{1+3}=1.

As a check, the score at the starting value is \((2-1)+(6-3)=4\), and the expected information is \(1+3=4\). The Fisher-scoring step is \(4/4=1\), as obtained from the adjusted working regression. If the offset were omitted from the adjustment, the weighted mean of \(z\) would be \(\{1+3(\log 3+1)\}/4\), not \(1\).

Worked Example: Bernoulli Logit Model With a Known Offset

Consider an intercept-only Bernoulli logit model with two observations, offsets \(o=(0,\log 3)\), outcomes \(y=(1,0)\), and starting coefficient \(\beta^{(0)}=0\). The linear predictors are \(\eta=(0,\log 3)\). Hence the fitted probabilities are \(p_1=1/2\) and \(p_2=3/4\).

For a Bernoulli response with logit link, \(V(\mu)=p(1-p)\) and \(\mu_\eta=p(1-p)\). Therefore \(w_i=p_i(1-p_i)\). The first weight is \(1/4\), and the second is \(3/16\). The working responses are

z_1=0+\frac{1-\frac12}{\frac14}=2, \qquad z_2=\log 3+\frac{0-\frac34}{\frac{3}{16}}=\log 3-4.

After subtracting the offsets, the adjusted responses are \(2\) and \(-4\). The weighted intercept update is

\beta_{\mathrm{new}} =\frac{\frac14(2)+\frac{3}{16}(-4)}{\frac14+\frac{3}{16}} =\frac{\frac12-\frac34}{\frac7{16}} =-\frac47.

The score at the starting value is \((1-\frac12)+(0-\frac34)=-\frac14\); the expected information is \(\frac14+\frac{3}{16}=\frac7{16}\). Their ratio is \(-4/7\), confirming the working-regression calculation.

What the Offset Does—and Does Not Do

An offset contributes to the fitted predictor and therefore affects the current means, working responses, and weights. For example, a Poisson log-link offset of \(\log(t_i)\) gives \(\mu_i=t_i\exp(x_i^\mathsf{T}\beta)\). It does not, however, create a new estimated parameter: the derivative of \(\eta_i=x_i^\mathsf{T}\beta+o_i\) with respect to \(\beta\) remains \(x_i\). Consequently, the coefficient information matrix has the same design-matrix form as without an offset, evaluated at means that do depend on the offset.

If \(\phi\) is a common scalar dispersion, it appears in every \(w_i\) as the same factor \(1/\phi\). It scales the score and expected information consistently and thus cancels from the coefficient solution to the weighted normal equations. Nevertheless, the weights should be defined consistently with the variance model; a dispersion factor must not be placed into the working response.

Common Mistakes

Regressing \(z\) on \(X\) and forgetting \(o\). This treats the whole predictor as \(X\beta\), so it generally produces the wrong update. Use \(z-o\) as the response, or include the known offset as a fixed contribution.
Adding the offset as an ordinary estimated covariate. An offset has a fixed coefficient of one; estimating its coefficient changes the model.
Subtracting the offset from the wrong quantity. The working response \(z\) approximates the full predictor \(\eta\), so subtract \(o\) to obtain the target for \(x^\mathsf{T}\beta\). Do not subtract it from the observed response \(y\).
Ignoring the offset when calculating current weights. The weights depend on the current mean. Since the offset is part of \(\eta\), it can change that mean and hence the weights.

Key takeaway: At each IRLS iteration, compute \(z\) and \(W\) from the current model including the offset, then solve the weighted regression of \(z-o\) on \(X\). Equivalently, fit \(z\) with \(o\) held fixed in the working predictor.

Check Your Understanding

Use the offset-adjusted working regression to answer these questions.

In a GLM with \(\eta_i=x_i^\mathsf{T}\beta+o_i\), which quantity should be regressed on \(X\): \(z\), \(z-o\), or \(y-o\)?
Explain why a known offset does not add a column of unknown coefficients to the Fisher-scoring update.
For a Poisson log-link model with exposure \(t_i\), what offset produces \(\mu_i=t_i\exp(x_i^\mathsf{T}\beta)\)?
If the offset is included in the linear predictor but omitted when calculating the current means, which parts of the IRLS calculation can be affected?
Describe the equivalent weighted least-squares formulation that uses \(z\) directly rather than \(z-o\).