Why Gaussian Identity IRLS Is One-Step Least Squares

Why This Special Case Is Worth Recognizing

Iteratively reweighted least squares (IRLS) is a general method for computing generalized linear model estimates. In many models, its working response and weights change as the fitted means change, so the algorithm must update them repeatedly. The Gaussian model with an identity link is different: its working response is the observed response, and its weights do not depend on the current coefficients. Consequently, a single Fisher-scoring update reaches the least-squares solution.

The word “one-step” needs a precise interpretation. It means one full Fisher-scoring update from any finite starting coefficient vector, provided the design matrix has full column rank. The result is not a claim that every GLM can be fitted in one iteration, nor that ordinary least squares is appropriate when the model has nonconstant observation weights or a nonzero offset.

Model and Working Quantities

Suppose \(Y_1,\ldots,Y_n\) are independent Gaussian observations, with known positive observation weights \(a_i\), and \(\operatorname{Var}(Y_i)=\phi/a_i\), where \(\phi>0\) is the dispersion parameter. Let \(x_i\) be the \(i\)th row of a fixed design matrix \(X\), written as a column vector in formulas involving \(x_i^\top\beta\). A known offset \(o_i\) may be included in the linear predictor.

Definition: The Gaussian identity-link regression model considered here is \(Y_i\sim N(\mu_i,\phi/a_i)\), independently, with \(\mu_i=\eta_i=x_i^\top\beta+o_i\). The coefficient vector is \(\beta\in\mathbb{R}^p\); the offsets \(o_i\) and weights \(a_i>0\) are known. When every \(a_i=1\) and every \(o_i=0\), this reduces to the usual unweighted, no-offset Gaussian linear model.

For a GLM with variance function \(V(\mu)\), the Fisher-scoring working response and weight for observation \(i\) are \(z_i=\eta_i+(Y_i-\mu_i)(d\eta_i/d\mu_i)\) and \(w_i=a_i/\{\phi V(\mu_i)(d\eta_i/d\mu_i)^2\}\). These are the quantities used to form the weighted working regression. In the present model, \(V(\mu_i)=1\), because \(\operatorname{Var}(Y_i)=\phi/a_i\), and the identity link gives \(d\eta_i/d\mu_i=1\).

Result: In the Gaussian identity-link model, \(z_i=Y_i\) and \(w_i=a_i/\phi\) for every \(i\). Thus the working response is fixed at the observed response, and the working weights are constant across coefficient updates.

To see the response calculation directly, substitute \(\eta_i=\mu_i=x_i^\top\beta+o_i\):

z_i =\eta_i+(Y_i-\mu_i) =(x_i^\top\beta+o_i)+Y_i-(x_i^\top\beta+o_i) =Y_i.

The offset has not disappeared from the model. It cancels in the construction of \(z_i\), but it remains in the regression being solved: the working fit compares \(z_i-o_i\) with \(x_i^\top\beta\). This distinction is essential when translating the working response into a coefficient update.

Worked Example: The Working Response and Weights

Take one observation with \(Y_i=7\), \(x_i^\top\beta=2\), offset \(o_i=3\), weight \(a_i=4\), and dispersion \(\phi=2\). Its current linear predictor and mean are both \(2+3=5\).

The working response is \(z_i=5+(7-5)=7\), exactly the observed response. The working weight is \(w_i=4/2=2\). If the current coefficient changes, the predictor and mean change together, but the same cancellation still gives \(z_i=7\), and the weight remains \(2\).

The working regression uses response \(7\), weight \(2\), and offset \(3\). Equivalently, it fits the adjusted response \(7-3=4\) against the design row \(x_i^\top\).

From the Gaussian Likelihood to Least Squares

The same one-step conclusion can be seen from the likelihood. Let \(A=\operatorname{diag}(a_1,\ldots,a_n)\), let \(y\) and \(o\) be the vectors of observations and offsets, and let \(X\) have rows \(x_i^\top\). For fixed \(\phi\), the Gaussian log-likelihood as a function of \(\beta\), up to terms not involving \(\beta\), is

\ell(\beta) =-\frac{1}{2\phi}(y-o-X\beta)^\top A(y-o-X\beta).

Differentiating with respect to \(\beta\) gives the score \(U(\beta)=X^\top A(y-o-X\beta)/\phi\). Differentiating once more gives the Hessian \(-X^\top AX/\phi\), which is constant in \(\beta\). Therefore the expected information is \(X^\top AX/\phi\) as well. If \(X\) has full column rank, \(X^\top AX\) is invertible: for any nonzero \(v\), \(v^\top X^\top AXv=(Xv)^\top A(Xv)>0\), since \(Xv\ne0\) and every \(a_i>0\).

Theorem: Suppose the independent observations follow the Gaussian identity-link model above, \(a_i>0\), \(\phi>0\), and \(X\) has full column rank. For any starting vector \(\beta^{(0)}\), one Fisher-scoring update for \(\beta\) equals \(\hat{\beta}=(X^\top AX)^{-1}X^\top A(y-o)\), the unique weighted least-squares estimate. If all \(a_i=1\) and \(o=0\), this is the ordinary least-squares estimate.

Proof. Fisher scoring adds the inverse expected information times the score to the current coefficient vector. At \(\beta^{(0)}\), the increment is

\begin{aligned} \beta^{(1)}-\beta^{(0)} &=\left(\frac{X^\top AX}{\phi}\right)^{-1} \frac{X^\top A(y-o-X\beta^{(0)})}{\phi}\\ &=(X^\top AX)^{-1}X^\top A(y-o-X\beta^{(0)}). \end{aligned}

The factors of \(\phi\) cancel. Adding \(\beta^{(0)}\) to both sides yields

\begin{aligned} \beta^{(1)} &=\beta^{(0)}+(X^\top AX)^{-1}X^\top A(y-o-X\beta^{(0)})\\ &=(X^\top AX)^{-1}X^\top A(y-o). \end{aligned}

The last equality follows because \((X^\top AX)^{-1}X^\top AX\beta^{(0)}=\beta^{(0)}\). The resulting vector minimizes \((y-o-X\beta)^\top A(y-o-X\beta)\): its derivative with respect to \(\beta\) is zero there, and its Hessian is \(2X^\top AX\), which is positive definite under the full-rank assumption. Hence it is the unique weighted least-squares estimate. The update does not depend on \(\beta^{(0)}\), proving the claim. \(\square\)

This proof also explains why the result is exact rather than an approximation near a solution. The score is linear in \(\beta\), and the information matrix is constant. Fisher scoring therefore solves the coefficient score equation in one update.

Worked Example: One Update With an Intercept and Slope

Observe \((x_i,Y_i)=(0,1),(1,2),(2,2)\), with no offset and all weights equal to one. The design matrix has columns for the intercept and slope. Its cross-products are \(X^\top X=\begin{pmatrix}3&3\\3&5\end{pmatrix}\) and \(X^\top y=\begin{pmatrix}5\\6\end{pmatrix}\). The determinant of \(X^\top X\) is \(3(5)-3(3)=6\), so the matrix is invertible.

The least-squares solution is

\hat{\beta} =\frac{1}{6} \begin{pmatrix}5&-3\\-3&3\end{pmatrix} \begin{pmatrix}5\\6\end{pmatrix} = \begin{pmatrix}7/6\\1/2\end{pmatrix}.

Start Fisher scoring at \(\beta^{(0)}=(0,0)^\top\). Its score is \(X^\top y/\phi=(5,6)^\top/\phi\), and its expected information is \(X^\top X/\phi\). Their product in the scoring update gives the same \((7/6,1/2)^\top\) just calculated. For a direct check, the fitted values are \(7/6,5/3,13/6\); the residuals are \(-1/6,1/3,-1/6\). Their sum is zero, and their \(x\)-weighted sum is \(1/3-2/6=0\), so both intercept and slope score equations hold.

Worked Example: A Weighted Fit With an Offset

Consider a one-coefficient model \(\mu_i=x_i\beta+o_i\), with design values \(x=(1,2,1)^\top\), offsets \(o=(1,0,2)^\top\), observations \(y=(4,8,9)^\top\), and weights \(a=(1,2,1)^\top\). The offset-adjusted responses are \(y-o=(3,8,7)^\top\).

The weighted least-squares coefficient is

\hat{\beta} =\frac{\sum_i a_i x_i(y_i-o_i)}{\sum_i a_i x_i^2} =\frac{1(1)(3)+2(2)(8)+1(1)(7)} {1(1^2)+2(2^2)+1(1^2)} =\frac{42}{10}=4.2.

For a Fisher-scoring check, start at \(\beta^{(0)}=0\). The score is \(42/\phi\), while the expected information is \(10/\phi\), so the increment is \(4.2\). The updated fitted means are \(1+1(4.2)=5.2\), \(0+2(4.2)=8.4\), and \(2+1(4.2)=6.2\). The weighted design-residual sum is \(1(1)(4-5.2)+2(2)(8-8.4)+1(1)(9-6.2)=-1.2-1.6+2.8=0\), confirming the coefficient score equation.

What “One Step” Does and Does Not Mean

The working regression for this model is exactly the weighted least-squares problem

\min_{\beta}\ \sum_{i=1}^n \frac{a_i}{\phi} \{Y_i-o_i-x_i^\top\beta\}^2.

Because \(\phi\) multiplies every weight by the same positive constant, it does not change the minimizing coefficient vector. Thus one may equivalently minimize the same expression without the factor \(1/\phi\). If the \(a_i\) differ, however, the solution is generally weighted least squares, not ordinary least squares.

Worked Example: A Weighted Intercept-Only Fit

Let three observations be \(2,4,8\), with weights \(1,2,1\), no offset, and an intercept-only design. The coefficient is the weighted mean: \(\hat{\beta}=(1(2)+2(4)+1(8))/(1+2+1)=18/4=4.5\).

Starting instead at \(\beta^{(0)}=10\), the score is \(\{1(2-10)+2(4-10)+1(8-10)\}/\phi=-22/\phi\), and the information is \(4/\phi\). The update is \(10+(-22/\phi)/(4/\phi)=10-5.5=4.5\). In contrast, the unweighted mean is \(14/3\), not \(4.5\); ignoring the unequal observation weights would solve a different regression problem.

If \(\phi\) is unknown, the coefficient score and expected information still have the common factor \(1/\phi\), so that factor still cancels in the coefficient update. This statement concerns the coefficient update; estimating dispersion is a separate parameter calculation.

Full column rank guarantees a unique coefficient vector. If the columns of \(X\) are linearly dependent, \(X^\top AX\) is singular, so the displayed inverse and unique-coefficient conclusion do not apply. A least-squares fitted vector may still be defined, but coefficients are not uniquely identified without an additional convention or constraint.

Common Mistakes

Regressing the observations on \(X\) while dropping an offset. With known offsets, fit \(y-o\) on \(X\), or equivalently fit \(y\) with the offset included in the predictor. The offset is part of the mean model even though it cancels from the working response.
Calling the solution ordinary least squares when weights differ. Ordinary least squares follows only when the observation weights are all equal, in particular when \(a_i=1\) for every observation. Unequal \(a_i\) produce a weighted fit.
Thinking that “one step” means starting at the least-squares estimate. The proof shows the update reaches the solution from any finite starting vector because the score is linear and the information is constant.
Claiming that all Gaussian GLMs need only one IRLS update. The result depends on the identity link and constant Gaussian variance function. It does not extend automatically to a different link, where working responses or weights may depend on the current fitted means.
Leaving out the rank condition. If \(X\) is not full column rank, the inverse in the coefficient formula does not exist and a unique coefficient vector is not guaranteed.

Key takeaway: For independent Gaussian observations with an identity link, the working response is \(y\), the working weight is \(a_i/\phi\), and the offset remains in the working regression. With a full-rank design, the first Fisher-scoring update is exactly the weighted least-squares estimate, from any starting coefficient vector.

Check Your Understanding

Use the model assumptions and results above to answer these questions.

For a Gaussian identity-link model with \(a_i=3\) and \(\phi=6\), what is the working weight for observation \(i\), and does it depend on the current coefficient vector?
With a known offset \(o_i\), explain why the working response is still \(Y_i\) but the offset must still appear in the coefficient fit.
Why does multiplying every working weight by \(1/\phi\) leave the weighted least-squares coefficient estimate unchanged?
What does the full-column-rank assumption ensure in the one-step theorem?
Starting at a coefficient vector different from the least-squares solution, why does one Fisher-scoring update nevertheless reach that solution?