Dispersion Scaling in Fisher Scoring

Why Dispersion Changes the Scale, Not the Direction, of an Update

In a generalized linear model, the variance of the response determines how much information each observation contributes about the mean. When every observation shares the same dispersion parameter, that parameter rescales all the information by the same amount. Fisher scoring therefore takes a coefficient step in the same direction and of the same size for any fixed positive common dispersion, even though the likelihood curvature changes.

The distinction matters in computation and interpretation. A weighted least-squares calculation may give the same coefficient update after all its weights are multiplied by a common constant, but that does not make the constant irrelevant: it changes the scale of the score and information, and hence the curvature used to describe uncertainty.

Model and Notation

Suppose \(Y_1,\ldots,Y_n\) are independent responses in an exponential-dispersion model with a common, fixed dispersion \(\phi>0\). Write \(\mu_i=E[Y_i]\), \(\eta_i=x_i^\mathsf{T}\beta\), and \(\eta_i=g(\mu_i)\), where \(x_i^\mathsf{T}\) is row \(i\) of the design matrix \(X\). Assume \(\operatorname{Var}(Y_i)=\phi V(\mu_i)\), with \(V(\mu_i)>0\). Assume the link and mean-variance function are differentiable at the current fitted means, and that the model has the usual regularity conditions for differentiating the log-likelihood and taking the expected information. In particular, the response support does not depend on \(\beta\).

Definition: At a current coefficient value \(\beta\), let \(d_i=d\mu_i/d\eta_i=1/g'(\mu_i)\). The unscaled IRLS working weight is \(W_i=d_i^2/V(\mu_i)\), and the working response is \(z_i=\eta_i+(y_i-\mu_i)/d_i\), provided \(d_i\ne 0\). With common dispersion \(\phi\), the actual working weight is \(W_i/\phi\).

The working response is a local linearization of the mean relationship, not a replacement for the observed response. The focus here is the extra common factor \(1/\phi\) in the weights. To make coefficient updates unique, assume the weighted design matrix has full column rank at the current iterate.

Information and Working Weights

For an exponential-dispersion likelihood, the coefficient score contribution from observation \(i\) is \(\phi^{-1}(y_i-\mu_i)(d_i/V(\mu_i))x_i\). Summing the contributions gives

U(\beta)=\frac{1}{\phi}X^\mathsf{T} \operatorname{diag}\!\left(\frac{d_i}{V(\mu_i)}\right)(y-\mu).

Taking the expected negative derivative of this score gives the Fisher information. Equivalently, the information contribution from observation \(i\) is \(\phi^{-1}W_i x_i x_i^\mathsf{T}\).

Theorem: Under the stated model and regularity conditions, the Fisher information for \(\beta\), with \(\phi\) held fixed, is

\mathcal I(\beta)=\frac{1}{\phi}X^\mathsf{T} \operatorname{diag}(W_i)X.

Derivation. Let \(s_i(\beta)=\phi^{-1}(y_i-\mu_i)(d_i/V(\mu_i))x_i\). In the expected negative derivative, the term obtained by differentiating the factors \(d_i/V(\mu_i)\) is multiplied by \(E[Y_i-\mu_i]=0\). The remaining term uses \(d\mu_i/d\beta=d_i x_i\):

-E\!\left[\frac{\partial s_i}{\partial\beta^\mathsf{T}}\right] =\frac{1}{\phi}\frac{d_i^2}{V(\mu_i)} x_i x_i^\mathsf{T}.

Adding these contributions over independent observations proves the formula. Thus \(\phi\) divides the information, while each \(W_i\) describes the relative contribution of observation \(i\) before that common rescaling.

Why the Common Factor Cancels from the Coefficient Step

Fisher scoring adds the information-weighted score to the current coefficient vector. With \(D=\operatorname{diag}(d_i/V(\mu_i))\) and \(W=\operatorname{diag}(W_i)\), its increment \(\Delta\beta\) solves

\mathcal I(\beta)\Delta\beta=U(\beta).

Theorem: If \(\phi>0\) is common to all observations and the weighted information matrix is invertible, the Fisher-scoring coefficient increment is independent of \(\phi\). It is

\Delta\beta=(X^\mathsf{T}WX)^{-1}X^\mathsf{T}D(y-\mu).

Proof. Substitute the score and information into the scoring equation:

\frac{1}{\phi}X^\mathsf{T}WX\Delta\beta =\frac{1}{\phi}X^\mathsf{T}D(y-\mu).

Because \(\phi>0\), multiply both sides by \(\phi\). The resulting equation is \(X^\mathsf{T}WX\Delta\beta=X^\mathsf{T}D(y-\mu)\). Invertibility gives the stated increment, with no \(\phi\) remaining. \(\square\)

The same cancellation appears in the weighted least-squares form. At the current iterate, the Fisher-scoring update minimizes \(\sum_i (W_i/\phi)(z_i-x_i^\mathsf{T}\beta_{\text{new}})^2\). Multiplying this whole objective by \(\phi\) does not change its minimizer, so the update can instead be found using weights \(W_i\). This equivalence is for the coefficient update, not for the scale of the objective or its curvature.

Worked Example: Gaussian Identity Link at Two Dispersion Values

Consider independent \(Y_i\sim N(\mu_i,\phi)\), with identity link and \(\mu_i=\beta_0+\beta_1x_i\). Use \((x_i,y_i)=(0,1),(1,2),(2,4)\), and start at \(\beta=(0,1)^\mathsf{T}\). Here \(V(\mu_i)=1\), \(d_i=1\), and all unscaled weights equal 1.

The current fitted means are \((0,1,2)\), so the residual vector is \(y-\mu=(1,1,2)^\mathsf{T}\). With design rows \((1,0),(1,1),(1,2)\), the unscaled score and information are

X^\mathsf{T}(y-\mu)= \begin{pmatrix}4\\5\end{pmatrix}, \qquad X^\mathsf{T}X= \begin{pmatrix}3&3\\3&5\end{pmatrix}.

For \(\phi=1\), the increment solves \(\begin{pmatrix}3&3\\3&5\end{pmatrix}\Delta\beta= \begin{pmatrix}4\\5\end{pmatrix}\). The determinant is \(15-9=6\), so

\Delta\beta =\frac{1}{6} \begin{pmatrix}5&-3\\-3&3\end{pmatrix} \begin{pmatrix}4\\5\end{pmatrix} = \begin{pmatrix}5/6\\1/2\end{pmatrix}.

For \(\phi=4\), the score and information are instead \((1,5/4)^\mathsf{T}\) and \(\begin{pmatrix}3/4&3/4\\3/4&5/4\end{pmatrix}\). Both are one quarter of their \(\phi=1\) values, so the increment remains \((5/6,1/2)^\mathsf{T}\). The updated coefficients are \((5/6,3/2)^\mathsf{T}\) in either case.

Dispersion and Likelihood Curvature

Although a common \(\phi\) cancels from the coefficient increment, it does not cancel from the curvature. The expected negative Hessian is the Fisher information, so its matrix is \(1/\phi\) times the information matrix computed with unit dispersion. In the usual exponential-dispersion likelihood, the coefficient-dependent log-likelihood has the same \(1/\phi\) scaling; with the mean and link fixed, its observed coefficient Hessian is also scaled by \(1/\phi\). A larger \(\phi\) therefore means less curvature in the coefficient likelihood, not a different coefficient step for that iteration.

For instance, in the Gaussian example, changing \(\phi\) from 1 to 4 divides every eigenvalue of the information matrix by 4. The score is divided by 4 as well. Their ratio in the scoring equation is unchanged, but the likelihood surface is flatter. Under the standard local quadratic approximation, coefficient covariance is approximately \(\mathcal I(\beta)^{-1}\), which scales as \(\phi\). Thus an unchanged point update does not imply unchanged inferential precision.

Worked Example: Gamma Log Link and the Working Regression

Suppose \(Y_1,Y_2\) are independent Gamma responses with \(E[Y_i]=\mu_i\) and \(\operatorname{Var}(Y_i)=\phi\mu_i^2\). Use the log link and an intercept-only model, so \(\eta_i=\beta\), \(\mu_i=e^\beta\), \(V(\mu_i)=\mu_i^2\), and \(d_i=\mu_i\). Take responses \(y_1=1,y_2=4\), and current value \(\mu_1=\mu_2=2\), or \(\beta=\log 2\).

For each observation the unscaled working weight is \(W_i=d_i^2/V(\mu_i)=4/4=1\). The working responses are \(z_1=\log 2+(1-2)/2=\log 2-1/2\) and \(z_2=\log 2+(4-2)/2=\log 2+1\).

The weighted least-squares update for an intercept is the weighted mean of the working responses. Both actual weights are \(1/\phi\), so

\beta_{\text{new}} =\frac{(1/\phi)z_1+(1/\phi)z_2}{2/\phi} =\frac{z_1+z_2}{2} =\log 2+\frac{1}{4}.

Equivalently, the score is \(\sum_i(y_i-\mu_i)/(\phi\mu_i)=1/(2\phi)\), and the information is \(2/\phi\). Their ratio gives increment \(1/4\), agreeing with the working regression. The dispersion changes both score and information scales, but not this coefficient step.

When the Cancellation Fails

Cancellation requires a single positive factor multiplying every observation’s score and information contribution. If the responses have different dispersions, the observation-specific factors alter their relative weights, and there is no common multiplier to remove. The same warning applies if weights are otherwise modified by observation-specific quantities: only a factor shared by all observations cancels from the coefficient solution.

Worked Example: Unequal Dispersions Change the Update

Let \(Y_1\sim N(\mu,\phi_1)\) and \(Y_2\sim N(\mu,\phi_2)\) independently, with an intercept-only mean model and observations \(y_1=0\), \(y_2=10\). Take \(\phi_1=1\) and \(\phi_2=4\). The Fisher-scoring information is \(1/\phi_1+1/\phi_2=1+1/4=5/4\). At current mean \(\mu\), the score is

\frac{0-\mu}{1}+\frac{10-\mu}{4} =\frac{5}{2}-\frac{5\mu}{4}.

The increment is the score divided by the information, namely \((5/2-5\mu/4)/(5/4)=2-\mu\). Thus the new mean is \(2\), the precision-weighted average \((1\cdot0+(1/4)\cdot10)/(1+1/4)\). Treating the dispersions as if they shared one common factor would instead give the unweighted average \(5\), which is incorrect. The difference arises because \(1\) and \(1/4\) do not multiply the two observations equally.

Common Mistake and Practical Interpretation

Mistake: “Dispersion has no effect on Fisher scoring.” The coefficient increment is unchanged only when \(\phi\) is a common positive factor and the rest of the current model quantities are held fixed. The score, information, and likelihood curvature still depend on \(\phi\).
Mistake: “Any dispersion factors can be dropped.” The unequal-dispersion example shows why this fails: observation-specific factors change relative precision and therefore change the update.
Mistake: “The weights are always \(W_i\).” For the likelihood-based working regression with common dispersion, the weights are \(W_i/\phi\). They may be replaced by \(W_i\) when computing the coefficient minimizer because the removed factor is common—not because it was absent from the model.

In practice, it is useful to keep the actual weights \(W_i/\phi\) when discussing information or curvature, and to recognize that the common scale can be omitted when solving the coefficient-only weighted least-squares system. If dispersion is updated between scoring iterations, each coefficient step still has this cancellation when its current dispersion is common and fixed for that step; the curvature and the model’s uncertainty scale nevertheless reflect the dispersion value.

Key takeaway: Common dispersion divides the coefficient score, Fisher information, and working weights by the same positive number. It cancels from the coefficient increment, but not from likelihood curvature or information. Unequal observation-specific dispersions generally change the coefficient update.

Check Your Understanding

Use the common-factor argument and the working-weight formulas to answer the following.

If a common dispersion is multiplied by 3, what happens to the coefficient score and Fisher information, and what happens to the Fisher-scoring increment?
For a model with \(d_i=2\), \(V(\mu_i)=4\), and common dispersion \(\phi=5\), calculate the unscaled and actual working weights.
Why does larger common dispersion imply lower expected information even when the current coefficient increment is unchanged?
In an intercept-only Gaussian model with two unequal variances, explain why the update is a precision-weighted rather than unweighted average.
What full-rank condition is needed to write the coefficient increment using an inverse information matrix?