Why the Choice of Update Matters
Finding a random-effects conditional mode is an optimization problem: for fixed model parameters and observed data, maximize the log integrand over the random effects. The earlier material on finding the conditional mode by Newton iteration introduced a curvature-based update. Here the focus is on how that update compares with Fisher scoring, and on what to do when a full proposed step is unreliable.
A mode-finding algorithm must do more than produce a plausible number. A poor step can lower the objective, leave the region where a local approximation is useful, or encounter curvature that is singular or points in the wrong direction. Fisher scoring changes the curvature used to calculate the proposal; damping changes how far the algorithm moves. Neither choice removes the need to check the objective and the numerical calculations.
Objective, Score, and Curvature
For cluster \(i\), let \(b_i\) be its random-effects vector, with Gaussian distribution \(N(0,D)\), where \(D\) is positive definite. At fixed model parameters and observed responses \(y_i\), define the conditional log integrand, up to terms not depending on \(b_i\), by
where \(\ell_i(b_i)\) is the conditional log likelihood. The Gaussian density also has a constant term in \(b_i\), which does not affect the mode. The score is \(U_i(b_i)=\nabla h_i(b_i)\). The observed negative Hessian is \(J_i(b_i)=-\nabla^2 h_i(b_i)\); it measures the local curvature of the negative log integrand. A conditional mode is a point where \(U_i(b_i)=0\), and it is a local maximum when the Hessian of \(h_i\) there is negative definite.
“Expected” here means expectation under the response distribution at the current parameter and random-effects value, holding that value fixed. It does not mean averaging over the random-effects distribution. If the likelihood curvature is data-dependent, observed and expected information can differ. In some canonical-link GLMMs, including the binary logit example below, they are identical for the random-effects coefficients, so Newton and Fisher scoring give the same proposal.
Why the Updates Are Ascent Directions
The update equations arise by approximating the objective locally. Newton uses a second-order Taylor expansion around the current value \(b_i\). Writing a trial change as \(d\), the approximation is \(h_i(b_i+d)\approx h_i(b_i)+U_i(b_i)^\mathsf{T}d-\frac12d^\mathsf{T}J_i(b_i)d\). Maximizing this quadratic gives \(J_i(b_i)d=U_i(b_i)\), provided the system is solvable.
The same local argument motivates Fisher scoring with \(I_i\) in place of \(J_i\). It also explains an important safeguard. If the matrix \(A_i\) used by either update is positive definite and \(d_i=A_i^{-1}U_i\), then
whenever \(U_i\ne 0\). Thus \(d_i\) is a local ascent direction. This does not guarantee that the full step increases the actual, non-quadratic \(h_i\). For \(0<\lambda\leq1\), a damped step is \(b_i+\lambda d_i\). Its initial directional derivative is \(\lambda U_i^\mathsf{T}d_i>0\), but a sufficiently large step can still reduce the objective. A line search tests the actual objective and reduces \(\lambda\) if needed.
Worked Example: Newton and Fisher Scoring for a Binary Random Intercept
Suppose one hypothetical cluster has 10 binary responses, 7 equal to 1, and a random intercept \(b\sim N(0,1)\). At fixed covariates and fixed parameters, its log integrand, omitting constants, is \(h(b)=7b-10\log(1+\exp(b))-b^2/2\). The likelihood is binary logit, so its negative second derivative at \(b\) is \(10p(b)\{1-p(b)\}\), where \(p(b)=\exp(b)/\{1+\exp(b)\}\). This curvature does not depend on the observed response values; observed and expected likelihood information agree. The prior contributes 1 to the negative Hessian.
At the current value \(b=0\), \(p(0)=0.5\). The score, observed negative Hessian, and Fisher information are
Both methods therefore give \(d=2/3.5=0.5714286\), and the full proposal is \(b_{\text{new}}=0.5714286\). As an objective check, \(h(0)=-10\log(2)\approx-6.9315\), while substitution of the unrounded update gives \(h(2/3.5)\approx-6.3546\). The objective increases, as desired. If instead the displayed rounded value \(b=0.5714\) is substituted, the corresponding value is \(h(0.5714)\approx-6.3546\). These are evaluations at different inputs; retain sufficient precision when checking an update.
When Fisher Scoring Changes the Proposal
For a general GLMM, Fisher scoring can differ from Newton because it substitutes expected likelihood information for the observed likelihood curvature. The prior precision remains part of both curvature matrices when \(D\) is fixed. A practical benefit is that expected information is often positive definite under regular conditions, even when an observed curvature at a particular iterate is indefinite. That can yield a better-defined local direction. It is not a universal guarantee: the expected information may also be ill-conditioned, and the resulting full step may still decrease the objective.
Worked Example: Computing a Fisher-Scoring Proposal
Consider a hypothetical one-dimensional GLMM at a current random-effects value \(b=1.2\). Suppose evaluation of its conditional log integrand gives score \(U(b)=2.5251\) and Fisher information \(I(b)=1.4386\), including the prior precision. These are the evaluated quantities supplied by the model at this iterate; Fisher scoring uses them to solve \(I(b)d_{\mathrm F}=U(b)\).
Substitution gives
The score and information give a proposal, not a certificate that moving to \(2.9552\) improves the objective. The algorithm should evaluate \(h(2.9552)\) and compare it with \(h(1.2)\). If the objective decreases or the evaluation is numerically unsafe, it can test a damped proposal such as \(1.2+0.5d_{\mathrm F}\), then shorten the step further if necessary. The rounded division matters: \(2.5251/1.4386\) rounds to \(1.7552\), not \(1.7553\).
A useful interpretation of the comparison is that Newton responds to the curvature actually observed at the current point, whereas Fisher scoring uses curvature averaged over responses under the current model. For canonical exponential-family likelihoods with linear predictors in \(b_i\), the likelihood’s negative Hessian often depends on fitted means and design values but not directly on the observed responses. In such cases the two curvature matrices agree. When they do agree, calling one update “Fisher” does not make it numerically different from Newton.
Damping and Objective-Based Safeguards
Damping scales a computed direction rather than changing the score. Given a proposal direction \(d_i\), choose \(0<\lambda\leq1\) and update \(b_i\leftarrow b_i+\lambda d_i\). A common backtracking rule starts at \(\lambda=1\), evaluates the objective, and repeatedly halves \(\lambda\) until the step yields a sufficient increase and all computations remain valid. The precise sufficient-increase rule is an algorithmic choice; the essential check is against the actual objective, not just its quadratic approximation.
Worked Example: Halving a Proposal to Control Step Size
Return to the binary random-intercept example. Its full Newton proposal from \(b=0\) is \(d=0.5714286\). Suppose an implementation is configured to take a half step for added conservatism, so \(\lambda=0.5\). Then
Direct evaluation gives \(h(0)=-6.9315\) and \(h(0.2857143)\approx-6.5026\), so this half step also increases the objective. It moves less than the full step, which reached an even higher objective in this example. Because this particular objective is well behaved near the starting value and the full step was successful, damping was not necessary here. In a difficult fit, the same check can reject an unsafe full step and retain a shorter one instead.
Damping is most useful when a proposal is large, when curvature changes quickly, or when the objective has regions that are numerically unstable. It cannot repair a wrong score, an incorrectly assembled Hessian, a singular model, or a genuinely poor local target. If no tested step produces an acceptable objective increase, inspect the derivatives, scaling, covariance parameterization, and starting value rather than continuing to shrink the step indefinitely.
Diagnostics, Alternatives, and Limitations
Monitor both progress and numerical health. Useful checks include the score norm, the change in \(h_i\), the size of the step, and whether the linear system for \(d_i\) can be solved reliably. For a positive-definite curvature matrix, a Cholesky factorization is a standard check. A failed factorization or a very large step can indicate non-positive curvature, poor scaling, or near-singularity. A solver reporting a tiny step is not by itself evidence of convergence: the score should also be small, and the objective should have stabilized.
| Approach | Curvature used | Main benefit | Important caution |
|---|---|---|---|
| Newton | Observed negative Hessian | Uses the local curvature of the actual objective | May be indefinite or produce an unsafe full step |
| Fisher scoring | Expected information, including prior precision | Can give a stable, positive-definite system under suitable conditions | May differ from actual local curvature and still needs an objective check |
| Damped Newton or scoring | Either method’s direction, multiplied by \(\lambda\) | Limits step length and supports backtracking | Does not correct an incorrect direction or a defective model |
These methods are local optimization procedures, not proofs that a unique mode exists or that every starting value reaches it. A random-effects log integrand need not be globally concave. In a multimodal or poorly identified problem, different starting values can reach different stationary points. Even at a well-behaved mode, accurate convergence matters: a residual score can affect subsequent curvature calculations and any approximation based on the local mode. The diagnostics here help assess the mode-finding iteration; they do not replace model assessment or guarantee that an approximation is adequate.
Common mistake: “The curvature matrix is positive definite, so the full Newton or Fisher step must increase the objective.” Positive definiteness makes the computed direction an ascent direction locally, as shown by \(U_i^\mathsf{T}d_i>0\). It does not guarantee that a finite step increases the actual objective. The correction is to evaluate \(h_i\) after the proposal and damp or reject the step when the objective or numerical checks indicate a problem.
Check Your Understanding
Answer each question, then compare with the solutions.
- For \(U=1.8\), observed negative Hessian \(J=2.4\), and expected information \(I=3.0\), calculate the Newton and Fisher-scoring directions.
- If \(A\) is positive definite and \(d=A^{-1}U\), why is \(d\) an ascent direction for a nonzero score?
- In the binary logit example, why did Newton and Fisher scoring give the same update?
- A full proposal increases the objective but produces an unexpectedly large step. What can damping do, and what can it not establish?
Solutions. (1) Newton gives \(d_{\mathrm N}=1.8/2.4=0.75\); Fisher scoring gives \(d_{\mathrm F}=1.8/3.0=0.60\). (2) Since \(d=A^{-1}U\), \(U^\mathsf{T}d=U^\mathsf{T}A^{-1}U>0\) for nonzero \(U\), so the directional derivative is positive. (3) In that canonical-link example, the binary logit likelihood’s negative Hessian equals its expected information at the current value; the prior precision is also included in both. (4) Damping takes a fraction of the direction and allows an objective-based check at shorter step lengths. A successful damped step does not prove global optimality, uniqueness of the mode, or correctness of the model.