Bernoulli Logit Working Responses and Weights

From a Fitted Probability to a Working Response

In a Bernoulli logit model, each observation has outcome $Y_i\in\{0,1\}$, fitted probability $p_i=P(Y_i=1)$, and linear predictor $\eta_i=\log\{p_i/(1-p_i)\}$. During an IRLS iteration, the current fitted probability determines two quantities for each observation: a working response $z_i$ and a working weight $w_i$. They are calculated from the current fit, not recorded as new data.

For the Bernoulli distribution, $E[Y_i]=p_i$ and $\operatorname{Var}(Y_i)=p_i(1-p_i)$. The logit link is $g(p)=\log\{p/(1-p)\}$, whose derivative is $g'(p)=1/\{p(1-p)\}$. Substitution into the general IRLS definitions gives the specific quantities used in this tutorial.

Definition: For a Bernoulli observation with $0<p_i<1$, current linear predictor $\eta_i=\log\{p_i/(1-p_i)\}$, and observed outcome $y_i\in\{0,1\}$, the logit working response and working weight are $$ z_i=\eta_i+\frac{y_i-p_i}{p_i(1-p_i)},\qquad w_i=p_i(1-p_i). $$

The weight is positive for every strictly interior probability. It is largest at $p_i=1/2$, where it equals $1/4$, and tends toward zero as $p_i$ approaches either boundary. The working response adds a residual adjustment to the current linear predictor. The adjustment is positive when $y_i=1$ and negative when $y_i=0$, but its size depends strongly on the fitted probability.

Deriving the Bernoulli Logit Formulas

For a GLM with mean $\mu_i$, variance function $V(\mu_i)$, and link $g$, the working response is $z_i=\eta_i+(y_i-\mu_i)g'(\mu_i)$, and the corresponding weight, with dispersion equal to one, is $w_i=1/\{V(\mu_i)[g'(\mu_i)]^2\}$. Here $\mu_i=p_i$, $V(p_i)=p_i(1-p_i)$, and $g'(p_i)=1/\{p_i(1-p_i)\}$. Therefore $$ z_i=\eta_i+\frac{y_i-p_i}{p_i(1-p_i)},\qquad w_i=\frac{1}{p_i(1-p_i)\{1/[p_i(1-p_i)]\}^2} =p_i(1-p_i). $$

The same formulas can be checked directly against the Bernoulli log-likelihood. For one observation, ignoring terms that do not depend on the model parameter, $\ell_i=y_i\log(p_i)+(1-y_i)\log(1-p_i)$. Since $d p_i/d\eta_i=p_i(1-p_i)$, differentiation gives $$ \frac{\partial \ell_i}{\partial \eta_i} =\left\{\frac{y_i}{p_i}-\frac{1-y_i}{1-p_i}\right\}p_i(1-p_i) =y_i-p_i. $$

This residual is the Bernoulli logit score with respect to the linear predictor. The weight $p_i(1-p_i)$ is the variance of that score, and hence the expected information for $\eta_i$. The working response and weight are the form this score and information take in the weighted least-squares representation of Fisher scoring.

Worked Example: One Success With Fitted Probability 0.8

Suppose the observed outcome is $y=1$ and the current fitted probability is $p=0.8$. First calculate the current linear predictor: $$ \eta=\log\left(\frac{0.8}{1-0.8}\right)=\log(4)\approx 1.3863. $$

The Bernoulli variance factor is $$ p(1-p)=0.8(0.2)=0.16. $$

The observed-minus-fitted residual is $y-p=1-0.8=0.2$. Thus the working response and weight are $$ z=1.3863+\frac{0.2}{0.16}=1.3863+1.25=2.6363,\qquad w=0.16. $$

The positive residual moves the working response above the current linear predictor. The weight is relatively small because a fitted probability of 0.8 is away from one-half.

Worked Example: One Failure With Fitted Probability 0.25

Now let the observed outcome be $y=0$ and the current fitted probability be $p=0.25$. The current linear predictor is $$ \eta=\log\left(\frac{0.25}{0.75}\right)=\log(1/3)\approx -1.0986. $$

The variance factor and residual are $$ p(1-p)=0.25(0.75)=0.1875,\qquad y-p=0-0.25=-0.25. $$

Therefore $$ z=-1.0986+\frac{-0.25}{0.1875} =-1.0986-\frac{4}{3} \approx -2.4319,\qquad w=0.1875. $$

Because the observed outcome is zero despite a positive fitted probability, the residual adjustment is negative. The working response is consequently below the current predictor.

Why These Quantities Give a Weighted Least-Squares Step

Let $X$ be the design matrix with rows $x_i^\mathsf{T}$, and suppose $\eta_i=x_i^\mathsf{T}\beta$, with no offset. At a current coefficient vector $\beta$, collect the fitted probabilities and working quantities into vectors $p,z$, and let $W$ be diagonal with entries $w_i=p_i(1-p_i)$. Fisher scoring for the coefficients uses the Bernoulli logit score $X^\mathsf{T}(y-p)$ and expected information $X^\mathsf{T}WX$. The following identity shows exactly how the working response encodes that score.

Result: For a Bernoulli logit model with current probabilities $0<p_i<1$, the working quantities satisfy $$ W(z-X\beta)=y-p. $$ If $X^\mathsf{T}WX$ is nonsingular, the Fisher scoring update is equivalently the weighted least-squares solution $$ \beta_{\mathrm{new}}=(X^\mathsf{T}WX)^{-1}X^\mathsf{T}Wz. $$

Proof. The current predictor vector is $X\beta$, whose $i$th entry is $\eta_i$. By the definition of $z_i$, $$ w_i(z_i-\eta_i) =p_i(1-p_i)\frac{y_i-p_i}{p_i(1-p_i)} =y_i-p_i. $$

This holds for each observation, so in vector form $W(z-X\beta)=y-p$. The Fisher scoring equation for an increment $\delta$ is $$ X^\mathsf{T}WX\delta=X^\mathsf{T}(y-p). $$

Using the identity just proved, its right-hand side is $X^\mathsf{T}W(z-X\beta)$. If $X^\mathsf{T}WX$ is nonsingular, solve for the increment: $$ \delta=(X^\mathsf{T}WX)^{-1}X^\mathsf{T}Wz-\beta. $$

Since $\beta_{\mathrm{new}}=\beta+\delta$, it follows that $\beta_{\mathrm{new}}=(X^\mathsf{T}WX)^{-1}X^\mathsf{T}Wz$, as claimed. $\square$

Worked Example: Working Quantities for Four Observations

Consider four Bernoulli observations with outcomes $y=(1,0,1,0)$ and current fitted probabilities $p=(0.2,0.5,0.8,0.9)$. Calculate each predictor using $\eta_i=\log\{p_i/(1-p_i)\}$, then calculate $z_i$ and $w_i$.

For observation 1, $\eta_1=\log(0.2/0.8)=\log(0.25)\approx-1.3863$. Its residual is $1-0.2=0.8$, and its variance factor is $0.2(0.8)=0.16$. Hence $$ z_1=-1.3863+\frac{0.8}{0.16}=3.6137,\qquad w_1=0.16. $$

For observation 2, $\eta_2=\log(0.5/0.5)=0$, its residual is $0-0.5=-0.5$, and its variance factor is $0.25$. Thus $$ z_2=0+\frac{-0.5}{0.25}=-2,\qquad w_2=0.25. $$

For observation 3, $\eta_3=\log(0.8/0.2)=\log(4)\approx1.3863$. Its residual is $1-0.8=0.2$, and its variance factor is $0.16$. Thus $$ z_3=1.3863+\frac{0.2}{0.16}=2.6363,\qquad w_3=0.16. $$

For observation 4, $\eta_4=\log(0.9/0.1)=\log(9)\approx2.1972$. Its residual is $0-0.9=-0.9$, and its variance factor is $0.9(0.1)=0.09$. Thus $$ z_4=2.1972+\frac{-0.9}{0.09}=-7.8028,\qquad w_4=0.09. $$

The results are summarized below. The largest weight belongs to the observation with fitted probability $0.5$; the observation with probability $0.9$ has the smallest weight in this set.

Observation	$y_i$	$p_i$	$\eta_i$	$z_i$	$w_i$
1	1	0.2	-1.3863	3.6137	0.16
2	0	0.5	0	-2.0000	0.25
3	1	0.8	1.3863	2.6363	0.16
4	0	0.9	2.1972	-7.8028	0.09

What the Working Response and Weight Mean

A useful way to understand the working response is to regard it as a locally adjusted response on the linear-predictor scale. Its adjustment is the observed residual divided by the current variance factor. Thus, when the current fitted probability is near zero or one, the denominator is small, and a disagreement between observation and fit can produce a large adjustment. This does not mean the original observation has changed: $y_i$ remains either zero or one.

For fixed current $p_i$ and $\eta_i$, the working response also has a simple exact mean and variance under the Bernoulli model: $$ E[z_i]=\eta_i+\frac{E[Y_i]-p_i}{p_i(1-p_i)}=\eta_i, \qquad \operatorname{Var}(z_i)=\frac{\operatorname{Var}(Y_i)}{\{p_i(1-p_i)\}^2} =\frac{1}{p_i(1-p_i)}=\frac{1}{w_i}. $$

These calculations use $E[Y_i]=p_i$ and $\operatorname{Var}(Y_i)=p_i(1-p_i)$, treating the current fitted probability as fixed. They explain why $w_i$ is the inverse variance of the working response and why the weighted least-squares step gives less weight to observations with fitted probabilities near the boundaries. The working response and weight are recomputed at each iteration because the fitted probabilities change.

The formulas require $0<p_i<1$. At exactly zero or one, the logit and the working-response denominator are not finite. In ordinary finite-coefficient logit models, fitted probabilities are strictly between zero and one, although they can be extremely close to a boundary.

Common Mistake

Wrong reasoning: “The working response is the observed binary outcome on a different scale, so I can use the same $z_i$ at every iteration.”

Correction: $z_i$ is calculated from both $y_i$ and the current fitted probability $p_i$, including the current predictor $\eta_i$. If the coefficients change, then $p_i$, $\eta_i$, and generally $z_i$ and $w_i$ change as well. The original observed response remains $y_i$; $z_i$ is an iteration-specific quantity used to express the current Fisher scoring step as weighted least squares.

A second practical check is to keep the residual sign straight: $y_i-p_i$ is positive for an observed success and negative for an observed failure. The weight is $p_i(1-p_i)$, not its reciprocal. The reciprocal appears in the working-response adjustment and in the variance of that response.

Key takeaway: For individual Bernoulli observations with a logit link, $z_i=\eta_i+(y_i-p_i)/\{p_i(1-p_i)\}$ and $w_i=p_i(1-p_i)$. Together they encode the Fisher scoring update as a weighted least-squares step, and both must be recalculated from the current fit.

Check Your Understanding

Use the current fitted probability in each question, and keep the outcome $y_i$ distinct from the working response $z_i$.

For $y=1$ and $p=0.5$, calculate $\eta$, $z$, and $w$.
For $y=0$ and $p=0.2$, calculate the residual adjustment $(y-p)/\{p(1-p)\}$ and the weight.
Why is the weight largest when the fitted probability is one-half?
State the vector identity connecting $W$, $z$, $X\beta$, and the Bernoulli residual vector $y-p$.
Explain why the working response for an observation with $y=0$ and $p$ near one can be far below its current linear predictor.