Scaled Gradient Checks

Why Check the Gradient?

An optimizer can report convergence even when its final parameter vector is not close enough to a stationary point for reliable inference. A gradient check asks whether the objective still has a meaningful local direction of decrease. In a GLMM, this is a numerical diagnostic of the fitted optimization problem: it does not by itself establish that the model is appropriate, that its parameters are identifiable, or that the optimizer found the best solution.

The raw gradient is informative but depends on the scale used to express each parameter. A component can be numerically large because its parameter is measured on a small scale, or numerically small because its parameter is measured on a large scale. Scaling the gradient by local curvature provides a way to assess components on a more comparable basis, provided that the curvature calculation is reliable and the solution is locally interior.

A raw gradient component has units of objective per unit of its parameter. Thus, comparing \(|g_j|\) directly across parameters with different units or scales can be misleading. A scaled gradient adjusts for the local curvature of the objective. For the definitions below, let \(\mathbf{H}=\nabla^2Q(\boldsymbol{\theta})\) be the Hessian at the candidate solution. When \(\mathbf{H}\) is positive definite, the symmetric curvature-scaled gradient is \(\mathbf{H}^{-1/2}\mathbf{g}\), and its length is \(\sqrt{\mathbf{g}^{\mathsf T}\mathbf{H}^{-1}\mathbf{g}}\). These are local diagnostics, not universal pass-or-fail tests: software and optimization methods may use different scaling conventions or tolerances.

What Curvature Scaling Measures

Curvature indicates how rapidly the objective changes as a parameter moves locally. A direction with high curvature is steep even for a small displacement; a direction with low curvature may require a larger displacement to produce a comparable change. A curvature-scaled gradient expresses the gradient in relation to this local geometry. The scaling should not be confused with simply dividing each component by an arbitrary constant or by the parameter estimate.

An implementation may instead use a Cholesky factor. If \(\mathbf{H}=\mathbf{R}^{\mathsf T}\mathbf{R}\), with \(\mathbf{R}\) upper triangular, define \(\mathbf{z}_{\mathrm{chol}}\) by solving \(\mathbf{R}^{\mathsf T}\mathbf{z}_{\mathrm{chol}}=\mathbf{g}\). This gives \(\mathbf{z}_{\mathrm{chol}}=\mathbf{R}^{-\mathsf T}\mathbf{g}\). It generally does not give the same component vector as \(\mathbf{H}^{-1/2}\mathbf{g}\) when \(\mathbf{H}\) is non-diagonal. However, the two vectors have the same squared norm:

This distinction matters when interpreting individual scaled components. The norm summarizes the size of the gradient in the local curvature metric, but the component values depend on the scaling transformation and parameterization. When reporting or interpreting a component-wise check, state the convention used if it is known. A small norm can support a conclusion that the local gradient is small; it does not guarantee that each component under every possible scaling convention will look identical.

Derivation: Link to the Local Newton Step

The curvature-scaled gradient is motivated by the local quadratic approximation. Assume \(Q\) has continuous second derivatives near the candidate point and that the Hessian there is positive definite. For a small displacement \(\boldsymbol{\delta}\), the second-order Taylor approximation is

The quadratic approximation is minimized by setting its gradient with respect to \(\boldsymbol{\delta}\) to zero. Differentiating the linear and quadratic terms gives \(\mathbf{g}+\mathbf{H}\boldsymbol{\delta}=\mathbf{0}\), hence the Newton displacement is \(\boldsymbol{\delta}_{N}=-\mathbf{H}^{-1}\mathbf{g}\). Substituting this displacement into the quadratic approximation gives the predicted decrease:

Thus the scaled-gradient norm is related to the amount by which a local quadratic approximation predicts the objective could decrease. This is a justification for checking it, not a promise that the actual objective will decrease by that amount: the approximation may be poor, numerical derivatives may be inaccurate, and a proposed displacement may not be feasible. At an unconstrained interior minimum, the gradient is zero. At a constrained minimum on a boundary, the ordinary gradient need not be zero; feasibility and the relevant constrained optimality conditions also matter.

Checking Components in Practice

A useful check begins by confirming what the reported gradient refers to: the objective being minimized, the parameterization, the parameter ordering, and whether the components are raw or scaled. Then inspect both the component pattern and an overall norm if available. A single large scaled component may indicate that the corresponding coordinate has not settled, but a component can also be inflated by a noisy derivative or an unreliable curvature estimate. Conversely, small components do not rule out problems elsewhere in the fit.

The Cholesky convention illustrates why a component check needs care. With a non-diagonal Hessian, the Cholesky-scaled components can mix information across coordinates. The symmetric and Cholesky vectors have equal lengths, but an individual entry in one vector is not generally the same as the corresponding entry in the other.

Assumptions, Diagnostics, and Failure Modes

The calculations above presume a smooth objective and derivatives evaluated at the reported parameter vector. Curvature scaling as written also requires a positive-definite Hessian. If the Hessian is indefinite, singular, or numerically unreliable, its inverse square root or Cholesky factor may not exist or may be unstable. In that situation, a scaled-gradient result may be unavailable or misleading; it is not a reason to declare convergence. Checking the Hessian and its numerical properties is a separate diagnostic.

In practice, gradients may be analytic, automatic-differentiation, or finite-difference approximations. Finite differences can be sensitive to step size and rounding, especially when parameters have very different scales. If a gradient looks suspicious, useful follow-up checks include evaluating it again with a suitable derivative method or step size, checking that the objective is stable near the estimate, and considering a refit from a nearby starting point. Optimizer convergence messages provide context, but a success message is not a substitute for checking the returned parameter values and numerical diagnostics.

Boundary constraints require additional care. Variance parameters in GLMMs may be represented using constrained scales or transformed coordinates. A raw gradient in one parameterization is not directly comparable to a gradient in another: under a change of coordinates, gradient components change. At a boundary, the unconstrained condition \(\mathbf{g}=\mathbf{0}\) need not characterize a constrained optimum. Interpret the check in the parameterization actually optimized, and do not use an unconstrained scaled-gradient calculation as a stand-in for constrained optimality conditions.

Common Pitfall and Practical Comparison

Wrong reasoning: “One raw gradient component is much larger than the others, so that parameter must be the one that failed to converge.” This conclusion ignores parameter units and curvature. Correction: examine the scaled diagnostic and the underlying numerical calculation, while remembering that even scaled components depend on parameterization and scaling convention. A large raw component can be compatible with modest curvature-scaled movement; a small raw component can still matter in a low-curvature direction.

Raw-gradient checks are simple and can help reveal gross derivative problems, but their components are difficult to compare across differently scaled parameters. Curvature-scaled checks adjust for local curvature and are often more useful for judging local stationarity, but require a reliable curvature estimate and a stated convention. Optimizer convergence messages summarize the optimizer's own stopping rules; they are complementary evidence, not alternatives that make gradient inspection unnecessary. No single check diagnoses every issue, and a favorable gradient check does not imply that the model is identifiable or that its estimates are substantively trustworthy.