Weights in the IRLS Working Regression

Why Weights Appear in the Working Regression

A GLM is fitted by repeatedly replacing its nonlinear mean structure with a locally appropriate weighted least-squares problem. The response in that problem is not generally the observed response: it is a working response constructed from the current fitted mean. The working weights describe how much information each observation contributes to that local approximation.

Frequency and variance weights both multiply the contribution of an observation to the working regression, but they have different meanings. A frequency weight counts identical replicated records in a weighted likelihood. A variance weight represents relative precision in a weighted estimating equation or working variance. Their algebraic effects on an IRLS step can match, without making their interpretations interchangeable. Binomial trial counts, in particular, belong to the binomial response model rather than being generic weights.

Setup: Suppose independent observations \(Y_i\) have means \(\mu_i\), with \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\) before weights are applied. The GLM has link \(g\), so \(\eta_i=g(\mu_i)=o_i+x_i^\mathsf{T}\beta\), where \(o_i\) is a known offset. Assume \(g\) is differentiable and locally invertible, \(V(\mu_i)>0\), and the means are in the interior of their allowed range. Let \(c_i\) be a frequency weight and \(v_i\) a variance weight, with \(c_i\) a nonnegative integer and \(v_i>0\). When both apply, write \(a_i=c_i v_i\); if only one applies, the other factor is 1.

For a frequency weight, the weighted log-likelihood contribution is \(c_i\) copies of the contribution for that record. For a variance weight, the weighted score and information contribution are multiplied by \(v_i\), equivalently giving working variance \(\phi V(\mu_i)/v_i\). The derivation below uses their product \(a_i\) to show the common IRLS algebra. It does not assert that the two kinds of weights have the same statistical meaning.

The Working Response and Working Weight

At the current parameter value, let \(\eta_i\) and \(\mu_i=g^{-1}(\eta_i)\) denote the current linear predictor and fitted mean. A first-order expansion of the inverse link around \(\eta_i\) gives \(g^{-1}(\eta_i+\delta)\approx\mu_i+(d\mu_i/d\eta_i)\delta\). Solving this local relation for the predictor value corresponding to the observed response produces the working response.

Definition: The working response and unscaled working weight for observation \(i\) are

z_i=\eta_i+(y_i-\mu_i)\frac{d\eta_i}{d\mu_i}, \qquad W_i=a_i\frac{(d\mu_i/d\eta_i)^2}{V(\mu_i)}.

The corresponding weighted least-squares criterion is proportional to \(\sum_i W_i\{z_i-o_i-x_i^\mathsf{T}\beta_{\mathrm{new}}\}^2\). A common dispersion factor \(1/\phi\) can be included in every working weight; it does not change the coefficient update when \(\phi\) is common to all observations.

The weight can also be written \(W_i=a_i/\{V(\mu_i)(d\eta_i/d\mu_i)^2\}\). The two forms are identical because \(d\mu_i/d\eta_i\) and \(d\eta_i/d\mu_i\) are reciprocal wherever the inverse link is differentiable with nonzero derivative. The working response itself has no \(a_i\) multiplier: weights affect the least-squares importance of a response, not the local link-scale correction used to construct that response.

Derivation from Fisher Scoring

The following result connects the local least-squares calculation to a GLM parameter update. Assume the mean-variance specification above and a differentiable link, and that the weighted score has the usual GLM form. This includes frequency-weighted likelihood contributions and variance-weighted quasi-likelihood contributions. For a unique coefficient update, also assume \(X^\mathsf{T}WX\) is nonsingular.

Result: One Fisher-scoring step for \(\beta\) is the minimizer of

$$ \sum_i W_i\{z_i-o_i-x_i^\mathsf{T}\beta_{\mathrm{new}}\}^2, \qquad W_i=a_i\frac{(d\mu_i/d\eta_i)^2}{V(\mu_i)}. $$ Thus the frequency and variance weights multiply the working weight, while the working response retains its ordinary GLM formula.

Proof. By the chain rule, the weighted score for \(\beta\) at the current fit is

U(\beta)=\frac{1}{\phi}\sum_i a_i \frac{(y_i-\mu_i)}{V(\mu_i)} \frac{d\mu_i}{d\eta_i}x_i.

The expected information matrix is

I(\beta)=\frac{1}{\phi}\sum_i a_i \frac{(d\mu_i/d\eta_i)^2}{V(\mu_i)}x_i x_i^\mathsf{T}.

The working-response definition gives \(z_i-\eta_i=(y_i-\mu_i)d\eta_i/d\mu_i\). Since \(\eta_i=o_i+x_i^\mathsf{T}\beta\), it follows that

U(\beta)=\frac{1}{\phi}\sum_i W_i x_i(z_i-\eta_i), \qquad I(\beta)=\frac{1}{\phi}\sum_i W_i x_i x_i^\mathsf{T}.

Fisher scoring updates the coefficients by \(I(\beta)^{-1}U(\beta)\). Therefore \(\beta_{\mathrm{new}}=\beta+I(\beta)^{-1}U(\beta)\) satisfies

\sum_i W_i x_i\{z_i-o_i-x_i^\mathsf{T}\beta_{\mathrm{new}}\}=0.

These are the normal equations for weighted least squares of \(z_i-o_i\) on the rows \(x_i^\mathsf{T}\) with weights \(W_i\). Nonsingularity gives a unique solution. The common factor \(1/\phi\) cancels from the update. \(\square\)

This also explains the role of an offset. It is held fixed, so the regression target is \(z_i-o_i\), not \(z_i\). The offset does not receive a coefficient update.

What Frequency Weights Change

If \(c_i\) identical records have the same response, predictors, and offset, their log-likelihood contributions add to \(c_i\) times the single-record contribution. Their score and expected information are likewise multiplied by \(c_i\). In IRLS this leaves the single-record working response unchanged and multiplies its working weight by \(c_i\).

Replication result: Repeating a record \(c_i\) times gives the same coefficient update as retaining one copy with frequency weight \(c_i\), provided the repeated records are identical in response and design and the model contributions are independent. In the working least-squares criterion, both contribute \(c_i W_i^{(0)}(z_i-o_i-x_i^\mathsf{T}\beta_{\mathrm{new}})^2\), where \(W_i^{(0)}=(d\mu_i/d\eta_i)^2/V(\mu_i)\).

To verify the result, write the contribution of the replicated records as a sum. Each of the \(c_i\) identical terms has the same current mean, working response, offset, and unweighted working weight. Their sum is exactly \(c_i\) times the one-record term. Consequently the total weighted least-squares criterion, and hence its minimizer, is unchanged by replacing the copies with one frequency-weighted record. This reasoning does not apply to records with different observed responses merely because their predictors match; aggregation of such data requires an appropriate model and response representation.

Worked Examples

Worked Example: Poisson Log Link With Two Kinds of Weight

Suppose a Poisson observation has \(y=3\) and current mean \(\mu=2\). Use the log link, frequency weight \(c=4\), and variance weight \(v=2\). Find the working response and working weight.

For a Poisson GLM, \(V(\mu)=\mu\). The log link gives \(\eta=\log\mu\), so \(d\eta/d\mu=1/\mu=1/2\), and \(d\mu/d\eta=\mu=2\). Thus

z=\log(2)+(3-2)\frac{1}{2} =0.693147+0.5=1.193147.

The unweighted working weight is \(2^2/2=2\). The combined multiplier is \(cv=4(2)=8\), so the final working weight is \(8(2)=16\). Neither weight changes \(z\); together they make this working response contribute eight times as much as the same unweighted record.

Worked Example: Binomial Trials Are Part of the Response Model

Suppose \(Y\) successes are observed in \(m=10\) Bernoulli trials, and the response is recorded as the proportion \(\bar y=Y/m=0.5\). At a current fitted probability \(p=0.4\), use a logit link. Show how the ten trials enter the working weight.

For a Bernoulli proportion, the unit-level variance function is \(V(p)=p(1-p)=0.4(0.6)=0.24\). The link-scale predictor is \(\eta=\log\{p/(1-p)\}\), with \(d\eta/dp=1/\{p(1-p)\}=1/0.24\). The working response is therefore

z=\log(0.4/0.6)+(0.5-0.4)/0.24 =-0.405465+0.416667 =0.011202.

The derivative \(dp/d\eta=p(1-p)=0.24\), so the unit-trial working weight is \(0.24^2/0.24=0.24\). The binomial likelihood for ten trials gives the proportion an information multiplier of \(m=10\), hence working weight \(10(0.24)=2.4\). The response here is a proportion with a binomial trial structure; this multiplier arises from that likelihood and variance, not from declaring the trial count to be an arbitrary frequency or variance weight.

Worked Example: Variance Weights With a Gaussian Identity Link and Offset

Consider two independent Gaussian observations with identity link and constant variance function \(V(\mu)=1\). Let the offsets be \(o_1=1\) and \(o_2=2\), the observed values be \(y_1=6\) and \(y_2=10\), and the variance weights be \(v_1=3\) and \(v_2=1\). Fit an intercept \(\beta\) for one IRLS step.

With the identity link, \(d\mu/d\eta=1\), and the working response is \(z_i=\eta_i+(y_i-\mu_i)=y_i\). The working weights are \(W_1=3\) and \(W_2=1\). Since the linear predictor is \(o_i+\beta\), the offset-adjusted targets are \(z_1-o_1=5\) and \(z_2-o_2=8\). The weighted least-squares estimate is

\beta_{\mathrm{new}} =\frac{3(5)+1(8)}{3+1} =\frac{23}{4}=5.75.

The fitted predictors are \(1+5.75=6.75\) and \(2+5.75=7.75\). The working response remains the observed value in this Gaussian identity-link case; the variance weights affect its relative influence, and the offset is subtracted before fitting the intercept.

Common Mistake: Multiplying the Working Response by the Weight

A tempting but incorrect move is to replace \(z_i\) with \(a_i z_i\) and then use it in an ordinary regression. That changes the target value rather than its influence. Weighted least squares instead minimizes \(\sum_i W_i(z_i-o_i-x_i^\mathsf{T}\beta)^2\): the weight multiplies the squared residual. Equivalently, one may multiply both the response and design row by \(\sqrt{W_i}\) to form an unweighted regression, but multiplying only the response by \(W_i\) is not equivalent.

A second error is to treat every count as a generic weight. A binomial trial count affects the binomial likelihood and the variance of the observed proportion. It must be represented consistently with the response: for a proportion, it supplies the trial-count multiplier; for a success count, the binomial mean and variance already include the number of trials. Adding that count again as an unrelated weight would count its information twice.

Key takeaway: At each IRLS iteration, compute \(z_i\) from the current mean and link, and compute \(W_i\) from the current variance function and link derivative. Frequency and variance weights multiply \(W_i\); a known offset is subtracted from \(z_i\) in the weighted regression. Trial counts must instead be handled through the binomial response model.

Check Your Understanding

Use the current fitted means when calculating working quantities.

For a Poisson log-link model, derive \(z_i\) and the unweighted \(W_i\) in terms of \(\mu_i\) and \(y_i\).
Why does a frequency weight multiply the working weight but not the working response?
In an offset model, what is the response used in the weighted least-squares regression?
For a binomial proportion, explain why the trial count contributes information without being a generic variance weight.
Why is multiplying the working response by \(W_i\), without transforming the design row, not weighted least squares?