Finite-Difference Gradient Noise

Why Finite-Difference Gradients Become Noisy

An optimizer may report a gradient, but that number can be an estimate formed by subtracting two nearly equal objective values. If the objective is a numerically evaluated pharmacometric likelihood, small integration or floating-point errors in those values can become large relative to their difference. The symptom is a gradient that changes markedly when the perturbation is changed, even though the parameter and model have not changed.

The central diagnostic is to calculate a derivative estimate at several perturbation sizes, using the same parameter coordinate and a controlled evaluation procedure. Large steps may show truncation error because the local slope changes across the step. Very small steps may amplify errors in objective evaluation. A useful estimate usually lies between these regimes, if such a range exists.

In model fitting, \(F\) might be the negative log-likelihood and \(\theta\) one component of a parameter vector. The formulas estimate a partial derivative while holding the other parameters fixed. For a vector parameter, apply the same calculation separately to each coordinate; the resulting components form an approximate gradient. The step \(h\) has the units of the parameter being perturbed, while the derivative has units of objective per parameter unit.

Truncation Error and Evaluation Error

First consider an exact, smoothly evaluated objective. If \(F\) has a continuous second derivative near \(\theta\), Taylor expansion gives \(F(\theta+h)=F(\theta)+hF'(\theta)+\frac{h^2}{2}F''(\theta)+o(h^2)\). Dividing the difference by \(h\) shows that the forward estimate is \(F'(\theta)+\frac{h}{2}F''(\theta)+o(h)\). Its leading truncation error is generally proportional to \(h\).

For the central estimate, assume \(F\) has a continuous third derivative near \(\theta\). The expansions on the two sides have remainders \(o(h^3)\), not necessarily \(O(h^4)\):

Subtracting the minus expansion from the plus expansion cancels the even-power terms. Dividing by \(2h\) gives the central estimate \(F'(\theta)+\frac{h^2}{6}F'''(\theta)+o(h^2)\). Thus its leading truncation error is generally proportional to \(h^2\), under the stated smoothness condition. The stronger claim of an \(O(h^4)\) remainder for the quotient requires stronger smoothness assumptions; continuous third derivative alone does not justify it.

Actual objective evaluations are not exact. Write the computed value as \(\widetilde F(x)=F(x)+e(x)\), where \(e(x)\) is evaluation error. If \(|e(x)|\leq\varepsilon\) at each perturbed point, the error contribution to a central difference is bounded by

The analogous bound for the forward difference is \(2\varepsilon/h\), since it uses errors at \(\theta+h\) and \(\theta\). These bounds explain the basic tradeoff: truncation tends to decrease as \(h\) shrinks, while evaluation error can grow like \(1/h\). If errors at the two central evaluations are independent with standard deviation \(\sigma\), the standard deviation of their difference divided by \(2h\) is \(\sigma/(\sqrt{2}h)\). This stochastic calculation is a model for variability, not a guarantee that errors from a deterministic solver are independent.

Parameter Scale Changes the Meaning of a Step

A fixed numerical step is not equally informative for every parameter. A perturbation of 0.001 may be tiny for a parameter near 100 and substantial for one near 0.002. The earlier material on parameter scaling and ill-conditioning motivates making the coordinate scale explicit here: gradient diagnostics should use perturbations that are meaningful relative to each parameter’s scale, while retaining the units of the coordinate used in the calculation.

If \(u=\theta/s\), where \(s\) is a positive scale with the same units as \(\theta\), then the chain rule gives \(dF/du=s\,dF/d\theta\). A step \(h_u\) in \(u\) corresponds exactly to a step \(h_\theta=s h_u\) in \(\theta\). Rescaling does not remove objective-evaluation error for the same physical perturbation. It does make step selection and comparisons across parameters more interpretable, and prevents confusing gradients expressed in different units.

Testing Smoothness and Evaluation Accuracy

Finite-difference formulas rely on local smoothness. A central estimate can be especially useful when the objective behaves smoothly over both sides of the perturbation, but its nominal second-order truncation behavior does not apply if the function lacks the required derivatives. Integration error, changes in solver behavior, and event handling can also make computed objective values depart from a smooth local approximation. A set of estimates that fails to settle as \(h\) is reduced may reflect such evaluation behavior, rather than simply an unfortunate step size.

For a central difference with per-evaluation error bounded by \(\varepsilon\), a useful conservative accounting is truncation of approximately \(|F'''(\theta)|h^2/6\) plus evaluation error of at most \(\varepsilon/h\). If a local bound \(M_3\) on \(|F'''|\) is available, balancing these two terms suggests \(M_3h^2/6\approx\varepsilon/h\), or \(h\approx(6\varepsilon/M_3)^{1/3}\). This is a guide, not a universal prescription: in model fitting, \(M_3\) and \(\varepsilon\) are rarely known accurately, and parameter scaling affects what \(h\) means.

A Practical Diagnostic Workflow

Forward differences require fewer evaluations if \(F(\theta)\) is already available, but have first-order truncation error and use a baseline evaluation. Central differences require two perturbed evaluations and have second-order truncation error under greater smoothness. Analytic derivatives avoid finite-difference step selection but can contain implementation errors. Automatic differentiation can provide accurate derivatives of the operations it differentiates, but does not automatically eliminate errors from approximate numerical solvers or unsupported operations. None of these alternatives makes an invalid or inaccurate objective evaluation reliable.

Finite-difference noise is a diagnostic about derivative calculation, not by itself a diagnosis of the model. A noisy gradient does not prove that a parameter is unidentified, at a boundary, or governed by a nonsmooth likelihood. Conversely, an apparently stable finite difference does not prove that a solver is accurate over the full parameter space. When the function does not satisfy the local smoothness conditions, the Taylor-based error analysis above no longer applies; the immediate task is to identify and understand the evaluation behavior rather than force a step-size rule to fit.

Common Mistakes and Limitations

• Making \(h\) as small as possible. This can magnify objective-evaluation error by a factor proportional to \(1/h\). Compare a range of steps instead.
• Using one absolute step for all parameters. The same increment can be negligible for one coordinate and enormous for another. Choose steps with reference to parameter scale and units.
• Calling every changing estimate “noise.” A large-step discrepancy may be truncation error, while a small-step discrepancy may be evaluation noise. The direction and pattern across steps matter.
• Interpreting a rescaled gradient without conversion. If \(u=\theta/s\), then \(dF/du=s\,dF/d\theta\). The numerical values differ because their units differ.
• Assuming smoothness from a successful fit. An optimizer stopping successfully does not establish that the objective has the derivatives required for the finite-difference accuracy claim.

These checks are local and coordinate-wise. They do not establish that the optimizer will behave well in every direction, nor do they replace investigation of integration failures or model behavior over a broader parameter region. Their value is narrower: they show whether a reported numerical gradient is plausibly stable to reasonable changes in its finite-difference step.