Verifying Gradients And Hessians

Why Derivative Checks Matter

Likelihood optimization depends on derivatives, not just on the value of the likelihood. A gradient that is coded with a sign error can send an update in the wrong direction; a Hessian with a small but consequential error can change the Newton step or the estimated standard errors. Software that returns derivatives through automatic differentiation can reduce algebraic mistakes, but it differentiates the function as implemented. It does not establish that the implementation matches the intended likelihood.

Derivative verification compares a proposed analytic or automatic derivative with a numerical approximation calculated from likelihood values or nearby gradients. The comparison is a diagnostic, not a replacement for deriving the score and Hessian. A numerical approximation also has error, so the step size and the scale of the parameters matter.

Definitions and Numerical Approximations

For a small nonzero step \(h\), the forward-difference approximation to a gradient component is

where \(e_j\) is the vector whose \(j\)th entry is one and whose other entries are zero. A central difference uses evaluations on both sides:

For a sufficiently smooth function, the forward difference has leading truncation error of order \(h\), while the central difference has leading truncation error of order \(h^2\). Central differences are therefore often more accurate for a comparable step size, though they require more function evaluations.

A Hessian can be approximated by differencing gradient vectors, one column at a time:

Alternatively, second differences of the log-likelihood approximate diagonal Hessian entries:

Mixed entries can also be computed from four function evaluations. For distinct coordinates \(j\) and \(k\), a central approximation is

When \(\ell\) is twice continuously differentiable, its exact Hessian is symmetric: \(H_{jk}=H_{kj}\). A numerically estimated Hessian may not be exactly symmetric because of approximation error. A large asymmetry is a warning to investigate, not a reason to conceal the discrepancy by symmetrizing the matrix immediately.

Directional Checks

Coordinate-by-coordinate checks are useful, but a multivariate derivative can also be tested along a direction that changes several parameters at once. For a direction \(v\), the directional derivative of \(\ell\) at \(\theta\) is \(v'g(\theta)\). If \(v\) is a unit vector, this is the rate of change per unit distance in that direction. A central numerical check is

The second directional derivative is \(v'H(\theta)v\). It can be checked using

Directional checks can reveal errors that are less obvious in individual components, particularly when a parameter vector has many coordinates. Several directions can be tried, including unit directions and directions with mixed signs. A successful check in one direction is not proof that every component is correct, so directional checks complement rather than replace coordinate checks.

Choosing a Step Size

A step that is too large creates truncation error: the difference quotient averages behavior over a region where curvature may matter. A step that is too small can also be inaccurate. In floating-point arithmetic, two nearly equal log-likelihood values are subtracted; rounding error or numerical noise in the likelihood evaluation can then overwhelm their small difference. For a first difference, this error often grows roughly as the evaluation error divided by \(h\). For a second difference, it can grow roughly as the evaluation error divided by \(h^2\).

These competing errors explain why reducing \(h\) indefinitely does not make a numerical derivative more accurate. In practice, compare results over a sensible range of step sizes. A central-difference gradient should often become more stable as \(h\) is reduced initially, then become less stable when cancellation or evaluation noise dominates. Hessian approximations can be especially sensitive because they divide by \(h^2\).

Parameter scaling matters. If one parameter is naturally near \(1000\) and another near \(0.01\), using the same absolute step for both may perturb one too little and the other too much. Choose steps relative to parameter magnitudes or meaningful parameter scales, and check that the perturbed values remain in a region where the likelihood can be evaluated. For likelihoods with numerical integration, simulation, or other noisy calculations, the evaluation noise must also be considered; an extremely small step may make the approximation unusable.

A useful diagnostic reports both absolute and relative discrepancies. For a proposed gradient \(g\) and numerical approximation \(\tilde g\), one possible componentwise comparison is \(\lvert g_j-\tilde g_j\rvert/\max(1,\lvert g_j\rvert,\lvert\tilde g_j\rvert)\). The maximum with one avoids an unstable relative error when both values are close to zero. The acceptable discrepancy depends on the scale, precision, and downstream use; there is no universal threshold that certifies correctness.

What Derivative Errors Do to Estimation

For a log-likelihood maximization problem, a Newton–Raphson update uses the gradient and Hessian, conventionally in the form \(\theta_{\text{new}}=\theta-H(\theta)^{-1}g(\theta)\) when the inverse and update are well-defined. An incorrect gradient changes the proposed direction and size of the step. An incorrect Hessian changes the curvature adjustment. A line search or trust-region procedure can limit a poor step, but it cannot make incorrect derivatives correct.

At a converged interior maximum, the observed-information estimate is \(J(\hat\theta)=-H(\hat\theta)\). Under the usual likelihood regularity conditions, an inverse of this matrix is used as an estimated covariance matrix, and its diagonal entries determine estimated standard errors. Thus, an erroneous Hessian can affect both the optimizer and reported uncertainty. The effect can be amplified when the information matrix is poorly conditioned: a small perturbation to its entries can produce a much larger change in its inverse.

Automatic differentiation is valuable because it applies the chain rule to the computational operations defining the likelihood, often more accurately than finite differences. It is still important to compare its output with independent checks at representative parameter values. Such comparisons can identify incorrect indexing, omitted likelihood contributions, inconsistent parameter transformations, or mistakes in the function being differentiated. Finite differences cannot tell which implementation is substantively correct, but disagreement is a reason to investigate before trusting the fit.

Common Pitfalls and a Verification Workflow

• Making the step arbitrarily tiny. Smaller steps reduce truncation error only up to a point; cancellation and evaluation noise can then dominate. Check stability across several scaled step sizes.
• Checking only at a stationary point. A gradient near zero can make some errors hard to see. Verify derivatives at more than one valid parameter value, including points away from the optimum.
• Comparing only one directional derivative. Agreement in one direction does not verify all gradient components or Hessian entries. Combine coordinate checks with multiple directional checks.
• Assuming automatic differentiation validates the model. It differentiates the implemented computation. Independently confirm that the likelihood code represents the intended model.
• Treating a numerical Hessian's asymmetry as harmless. Small asymmetry can result from numerical error, but substantial asymmetry calls for investigation of the derivative calculations, step sizes, or likelihood evaluations.

A practical sequence is to evaluate the likelihood at the test point, compare analytic or automatic gradient components with central differences, compare Hessian columns with differences of gradients, and check symmetry and several directional derivatives. Repeat at different step sizes and parameter values. Investigate any discrepancy before interpreting an optimizer's convergence or its standard errors.