Analytic and Finite-Difference Gradients

Why Gradient Computation Matters

An optimizer uses a gradient to determine how an objective changes when model parameters move. In nonlinear mixed-effects estimation, that objective may require substantial computation: evaluating a likelihood can involve integrating over random effects, locating conditional modes, or using a numerical approximation. A gradient method can therefore spend much of its time not in the parameter-update rule, but in repeatedly evaluating or differentiating that objective.

This tutorial compares two ways to obtain the gradient: differentiating the objective analytically, and approximating its derivatives by finite differences. The comparison concerns both computational cost and reliability. A derivative formula may be exact on paper but difficult to implement correctly through an integral or iterative inner calculation. A finite difference is easy to define, but it can be inaccurate when objective evaluations are noisy or when the perturbation is poorly scaled.

Objective and Gradient Definitions

If the estimation target is a log-likelihood \(\ell(\boldsymbol\theta)\) to be maximized, one can instead minimize \(Q(\boldsymbol\theta)=-\ell(\boldsymbol\theta)\). The gradient then has the opposite sign: \(\nabla Q=-\nabla\ell\). This tutorial uses the minimization convention. A gradient is a local quantity; a small gradient alone does not establish that the objective has a well-behaved minimum or that the statistical model is adequate.

An analytic gradient is obtained by differentiating the stated objective using calculus and implementing the resulting derivatives. “Analytic” does not necessarily mean a short closed-form expression: derivatives can be evaluated by automatic differentiation or by carefully implemented derivative recursions. A finite-difference gradient approximates each partial derivative using changes in objective values at perturbed parameter vectors.

Finite-Difference Formulas and Their Error

Let \(\mathbf e_j\) be the vector whose \(j\)th entry is one and whose other entries are zero. For a nonzero perturbation \(h\), the forward difference for coordinate \(j\) is

The central difference uses objective values on both sides:

The formulas approximate \(\partial Q/\partial\theta_j\), not the whole gradient at once. Repeating the calculation for all \(p\) coordinates gives an approximate gradient vector. The forward formula can reuse the unperturbed objective value; the central formula uses two perturbed evaluations per coordinate.

These are truncation-error statements: they describe how the formulas behave when objective values can be evaluated exactly and \(h\) is small. In computation, making \(h\) smaller also makes the numerator a difference between increasingly similar numbers. Rounding, approximate integration, stochastic evaluation, and incomplete convergence of an inner solver can then dominate. The displayed orders do not promise that reducing \(h\) always improves an implemented gradient.

Worked Example: Forward and Central Differences for a Likelihood Objective

Computational Cost as the Parameter Dimension Grows

If one objective evaluation costs \(C_Q\), a forward-difference gradient requires approximately \(p+1\) objective evaluations when the baseline value \(Q(\boldsymbol\theta)\) is not already available. A central-difference gradient requires \(2p\) perturbed evaluations. If the optimizer has already computed the baseline objective, the incremental forward cost is about \(p\) evaluations. These counts concern objective evaluations; they are not direct counts of arithmetic operations or wall-clock time.

An analytic gradient may be much cheaper than \(p\) separate objective evaluations when derivatives can be propagated alongside the calculations. It may also be comparable in cost to, or more expensive than, finite differences if differentiating the objective requires substantial additional work. The actual comparison depends on the model, implementation, and objective-evaluation cost. A single analytic-gradient call is not automatically “one objective evaluation” in computational effort.

Why Mixed-Effects Likelihoods Make the Comparison Important

A marginal likelihood contribution for one individual can be written schematically as \(L_i(\boldsymbol\theta)=\int f(y_i\mid\boldsymbol\eta,\boldsymbol\theta)g(\boldsymbol\eta\mid\boldsymbol\theta)\,d\boldsymbol\eta\), where \(y_i\) is that individual’s data and \(\boldsymbol\eta\) denotes random effects. If the integrand and its parameter derivative satisfy conditions that permit differentiation under the integral sign, then

This identity illustrates the analytic route: differentiate the likelihood contribution and evaluate the resulting expression. In practice, an estimation method may instead define an approximate objective involving conditional modes, quadrature, or other numerical calculations. The gradient must correspond to the objective actually supplied to the optimizer. Differentiating a theoretical likelihood while evaluating a different approximation can produce a gradient that is internally inconsistent with the objective values.

Finite differences automatically target the implemented objective as a black box, provided each evaluation computes that same objective accurately and reproducibly. But they may be expensive because each perturbation can repeat the inner calculations. If the evaluation includes Monte Carlo variability, varying numerical integration error, or an inner optimization that stops at different precision, the apparent change in \(Q\) may reflect evaluation noise rather than the parameter perturbation.

Diagnostics, Alternatives, and Limitations

A useful diagnostic is to compare analytic and finite-difference gradients at selected parameter vectors, using more than one coordinate and more than one perturbation scale. Agreement supports consistency between the implemented derivative and objective in those locations; disagreement is a signal to investigate, not proof that one method is correct. Check that the same data, objective definition, integration settings, parameter scale, and convergence criteria are used in each evaluation. Large discrepancies can indicate a derivative coding error, a discontinuity or kink, inaccurate inner calculations, or finite-difference cancellation.

Common warning signs include a gradient component that changes erratically under small parameter perturbations, a numerical derivative that changes greatly when the perturbation is modestly altered, and a derivative that disagrees in sign with observed objective changes. Such checks do not by themselves identify a single cause. In nonlinear mixed-effects calculations, also inspect whether conditional modes or numerical integration results are stable across the perturbed evaluations. A finite difference through an incompletely converged inner solver can measure changes in solver behavior as well as changes in the intended objective.

No method removes all numerical or modeling limitations. Analytic derivatives can be wrong because of implementation mistakes or because they differentiate a different target than the objective being optimized. Finite differences can be unreliable near non-smooth behavior, parameter boundaries, or regions where objective evaluations are noisy. When the objective is stochastic, a derivative estimate may vary between runs; when the objective is an approximation, its gradient is the gradient of that approximation, not necessarily of the exact likelihood.

A common mistake is to assume that central differences must be more accurate because their formal truncation error is of higher order, and therefore to make \(h\) as small as possible. The formal order assumes smoothness and sufficiently accurate objective values. If \(h\) is too small, rounding or integration error can dominate the numerator; if it is too large, curvature can dominate the approximation. The correction is to treat gradient comparison as a diagnostic under controlled objective evaluations, rather than as a guarantee based on the formula alone. Detailed selection of the perturbation scale is a separate issue.