From Starting Coefficients to a Logistic Update
A logistic IRLS update turns the current fit into a weighted least-squares problem. The arithmetic has a clear sequence: use the starting coefficients to calculate fitted probabilities, convert the binary observations into working responses, assign weights, and solve for new coefficients. The working responses are calculated quantities; they need not be zero or one.
Suppose \(Y_i\) are independent Bernoulli observations, with \(y_i\in\{0,1\}\), and the model is \(\operatorname{logit}(p_i)=\eta_i=\mathbf{x}_i^\top\boldsymbol{\beta}\), where \(p_i=P(Y_i=1)\) and \(\operatorname{logit}(p)=\log\{p/(1-p)\}\). At a current coefficient vector \(\boldsymbol{\beta}^{(0)}\), write \(\eta_i^{(0)}=\mathbf{x}_i^\top\boldsymbol{\beta}^{(0)}\) and \(p_i^{(0)}=1/\{1+\exp(-\eta_i^{(0)})\}\).
For finite starting linear predictors, each \(p_i^{(0)}\) is strictly between zero and one, so each weight is positive. If the design matrix has full column rank, the weighted least-squares solution is unique. In hand calculations, keep enough precision in probabilities and weights until after solving the coefficient update; early rounding can change the answer.
Why These Are the Working Quantities
The formulas come from locally expressing the mean response as a function of the linear predictor. For the logistic mean \(p(\eta)=1/\{1+\exp(-\eta)\}\), differentiation gives \(dp/d\eta=p(1-p)\). Thus a first-order Taylor approximation around the current predictor is \(p(\eta)\approx p_i^{(0)}+w_i(\eta-\eta_i^{(0)})\). Substituting the observed response \(y_i\) for the mean in this local relation and solving for the predictor gives \(z_i\). This constructs a working response for the local weighted regression; it does not assert that \(y_i\) is close to \(p_i^{(0)}\).
The weighted least-squares update also matches the Fisher-scoring step. Let \(X\) have rows \(\mathbf{x}_i^\top\), let \(W\) be diagonal with entries \(w_i\), and let \(\mathbf{p}\) and \(\mathbf{y}\) collect the fitted probabilities and observed outcomes at the current coefficients. The Bernoulli-logit score is \(X^\top(\mathbf{y}-\mathbf{p})\), and the expected information is \(X^\top W X\). If \(X\) has full column rank, the Fisher-scoring update solves \(\boldsymbol{\beta}^{(1)}=\boldsymbol{\beta}^{(0)}+(X^\top W X)^{-1}X^\top(\mathbf{y}-\mathbf{p})\).
Worked Logistic IRLS Updates
Worked Example: A Two-Coefficient Update From Four Observations
Consider a model with an intercept and one numeric predictor: \(\eta_i=\beta_0+\beta_1x_i\). The data are \((x_i,y_i)=(0,0),(1,1),(2,0),(3,1)\), and the starting coefficients are \((\beta_0^{(0)},\beta_1^{(0)})=(0,0)\). Calculate the first update.
At the start, every linear predictor is zero, so each fitted probability is \(p_i^{(0)}=1/(1+\exp(0))=0.5\). The weight for each observation is \(w_i=0.5(1-0.5)=0.25\). The working response is \(z_i=0+(y_i-0.5)/0.25\), which is \(-2\) for \(y_i=0\) and \(2\) for \(y_i=1\).
| \(i\) | \(x_i\) | \(y_i\) | \(\eta_i^{(0)}\) | \(p_i^{(0)}\) | \(w_i\) | \(z_i\) |
|---|---|---|---|---|---|---|
| 1 | 0 | 0 | 0 | 0.5 | 0.25 | -2 |
| 2 | 1 | 1 | 0 | 0.5 | 0.25 | 2 |
| 3 | 2 | 0 | 0 | 0.5 | 0.25 | -2 |
| 4 | 3 | 1 | 0 | 0.5 | 0.25 | 2 |
The design rows are \((1,x_i)\). The weighted normal equations have matrix and right-hand side
Thus \(\beta_0+1.5\beta_1=0\) and \(1.5\beta_0+3.5\beta_1=1\). Substituting \(\beta_0=-1.5\beta_1\) into the second equation gives \(1.25\beta_1=1\), so \(\beta_1^{(1)}=0.8\) and \(\beta_0^{(1)}=-1.2\). As a check, the determinant of the normal-equation matrix is \(1(3.5)-1.5^2=1.25>0\), so the solution is unique.
The updated linear predictors at \(x=0,1,2,3\) are \(-1.2,-0.4,0.4,1.2\). Their corresponding logistic probabilities are approximately \(0.2315,0.4013,0.5987,0.7685\). These probabilities describe the fitted model after one update; the calculated coefficient vector is not asserted to be a converged estimate.
Worked Example: An Intercept-Only Update
Suppose three independent binary observations are \(y=(1,1,0)\), with an intercept-only logistic model and starting intercept \(\beta_0^{(0)}=0\). Each fitted probability is \(0.5\), each weight is \(0.25\), and the working responses are \(2,2,-2\).
With only an intercept, the weighted least-squares update is the weighted average of the working responses. Since the weights are equal, \(\beta_0^{(1)}=\{0.25(2)+0.25(2)+0.25(-2)\}/\{3(0.25)\}=2/3\). The updated probability is \(1/\{1+\exp(-2/3)\}\approx0.6608\). The observed proportion is also \(2/3\), but the updated probability is not \(2/3\): the updated coefficient is \(2/3\), and it must be passed through the inverse-logit function to obtain the probability. The coefficient that would give probability \(2/3\) is \(\log\{(2/3)/(1/3)\}=\log 2\approx0.6931\), not \(2/3\).
Worked Example: A Binary Predictor With Nonzero Starting Values
Consider three observations at each of two predictor levels. Let \(x=0\) have outcomes \(0,0,1\), and let \(x=1\) have outcomes \(0,1,1\). Use \(\eta=\beta_0+\beta_1x\) and start at \((\beta_0^{(0)},\beta_1^{(0)})=(-1,0.5)\). This example shows that weights depend on the current fitted probability.
At \(x=0\), \(\eta^{(0)}=-1\), \(p^{(0)}\approx0.268941\), and \(w\approx0.196612\). The response for \(y=0\) is \(z=-1-0.268941/0.196612\approx-2.367879\); for \(y=1\), it is \(z=-1+0.731059/0.196612\approx2.718282\). The weighted mean of the three responses in this group is their ordinary mean, because all three share the same weight: \((-2.367879-2.367879+2.718282)/3\approx-0.672492\).
At \(x=1\), \(\eta^{(0)}=-0.5\), \(p^{(0)}\approx0.377541\), and \(w\approx0.235004\). The responses for \(y=0\) and \(y=1\) are, respectively, \(-0.5-0.377541/0.235004\approx-2.106531\) and \(-0.5+0.622459/0.235004\approx2.148721\). Their group mean is \((-2.106531+2.148721+2.148721)/3\approx0.730304\).
With an intercept and a binary indicator, the weighted least-squares fitted value at each predictor level is that group’s weighted mean response. Thus the new intercept is the \(x=0\) mean, \(\beta_0^{(1)}\approx-0.672492\), and the new \(x=1\) fitted value is the \(x=0\) mean plus the slope. Hence \(\beta_1^{(1)}\approx0.730304-(-0.672492)=1.402796\). The updated probabilities at \(x=0\) and \(x=1\) are approximately \(0.3379\) and \(0.6749\), respectively.
Reading the Update Correctly
The update is a local calculation at the starting coefficients. Each observation contributes a working response and a weight, and the weighted regression finds the coefficient vector that best fits those working responses in the design space. A high weight means the observation has greater influence on this particular weighted least-squares calculation; it does not mean that its binary outcome is more likely to be correct.
For the Bernoulli logit model, weights are largest when \(p_i^{(0)}\) is near \(0.5\) and become small when it is near zero or one. Working responses can be outside the interval \([0,1]\), as the examples show. That is expected: they are on the linear-predictor scale and are built to make the local weighted regression reproduce the Fisher-scoring step.
Common mistake: Treating \(z_i\) as a replacement observed outcome and interpreting it as a probability. For example, the value \(z_i=2\) in the first example is not a probability of two. It is a working response on the linear-predictor scale. The correction is to keep the scales distinct: solve the weighted least-squares problem using \(z_i\), then interpret the resulting coefficients through \(\eta_i=\mathbf{x}_i^\top\boldsymbol{\beta}\) and \(p_i=1/\{1+\exp(-\eta_i)\}\).
Check Your Understanding
Use the definitions and worked calculations above to answer these questions.
- For a Bernoulli logit observation with current predictor \(\eta^{(0)}=0\) and observed outcome \(y=0\), calculate its fitted probability, weight, and working response.
- Why can a logistic working response be negative or greater than one without creating a problem for the model?
- In an intercept-only weighted least-squares update, how do you obtain the new intercept from the working responses and weights?
- For a finite current linear predictor, why is the logistic working weight positive?
- What condition on the design matrix ensures the weighted least-squares coefficient update is unique when all logistic working weights are positive?