Truncation and Roundoff in Numerical Differentiation

Why the Step Size Matters

A finite-difference derivative replaces a limit by a calculation at a nonzero step size \(h\). Choosing \(h\) creates a trade-off. A larger step can make the approximation formula less accurate, while a very small step can make the arithmetic less reliable.

These effects have different sources. Truncation error comes from replacing a limiting process with a finite-step formula. Roundoff error comes from finite-precision arithmetic, especially when nearby function values are subtracted. The total error can therefore decrease at first and then increase as \(h\) shrinks.

Finite-Difference Formulas and Truncation Error

For a function \(f\) and a point \(x\), the forward-difference estimate for the first derivative is \((f(x+h)-f(x))/h\), where \(h>0\). The central-difference estimate is \((f(x+h)-f(x-h))/(2h)\). A common central estimate for the second derivative is \((f(x+h)-2f(x)+f(x-h))/h^2\).

The following Taylor expansions describe the error in these formulas. The bounds are local: the required derivatives must exist and be bounded on an interval containing \(x-h\), \(x\), and \(x+h\).

Thus the forward formula has leading truncation error proportional to \(h\), provided \(f''\) is bounded near \(x\). For the central first-derivative formula, if \(f\) has a bounded third derivative, the error is \(O(h^2)\). With additional smoothness, its leading term is:

For the central second-derivative formula, the corresponding expansion is:

In both central formulas, the leading truncation error is proportional to \(h^2\), when the indicated leading derivative is nonzero. The order notation describes the size of the omitted terms as \(h\) approaches zero; it does not say that the error is exactly a fixed constant times that power for every \(h\).

Why the Central Second-Derivative Error Is Order \(h^2\)

The central second-derivative expansion needs enough Taylor terms to justify its remainder after division by \(h^2\). Assume \(f\) has a continuous sixth derivative near \(x\), with \(|f^{(6)}(t)|\leq M_6\) throughout the interval from \(x-h\) to \(x+h\). Taylor's theorem, expanded through degree five, gives each of the two function values a remainder bounded by \(M_6h^6/720\).

Adding these expansions and subtracting \(2f(x)\) cancels the odd-degree terms. The constant terms cancel too. The remaining fourth-degree term is \(f^{(4)}(x)h^4/12\), and the sum of the two remainders has magnitude at most \(M_6h^6/360\). Consequently:

This also explains why stopping Taylor expansions too early can give an invalid remainder claim. For example, remainders of size \(O(h^5)\) in the function values only guarantee a numerator remainder of \(O(h^5)\), hence \(O(h^3)\) after division by \(h^2\). They do not justify an \(O(h^4)\) remainder for the quotient.

How Roundoff Enters

In finite-precision arithmetic, a computed value of \(f(t)\) may differ slightly from its exact value. A useful simplified model is that this perturbation has size at most about \(\epsilon S\), where \(\epsilon\) measures the relative precision and \(S\) represents the scale of the function values involved. Additional rounding in the arithmetic can be included in the constant multiplying this estimate.

For a first-derivative formula, errors in the function values are divided by a quantity proportional to \(h\), so the resulting roundoff contribution is commonly modeled as \(B\epsilon/h\). For the central second-derivative formula, the denominator is \(h^2\), and a useful model is \(B\epsilon/h^2\). Here \(B\) is a positive constant that absorbs the scale of the function values and details of the computation. These are estimates, not universal exact bounds; the actual constants depend on the problem and implementation.

The reason subtraction is a concern is that \(f(x+h)\) and \(f(x)\), or the three values in a second-difference numerator, can be close to one another. Their difference may be much smaller than the values being subtracted. Errors already present in those values can then be large relative to the small difference. Dividing that difference by \(h\) or \(h^2\) can magnify the effect.

Balancing the Two Error Contributions

A simple total-error model has the form \(E(h)=Ah^p+B\epsilon h^{-q}\). The first term represents truncation error, with \(A>0\) and \(p>0\); the second represents roundoff, with \(B>0\) and \(q>0\). Differentiating this model and setting its derivative to zero gives a candidate minimizing step:

For positive \(A\), \(B\), and \(\epsilon\), the error model decreases before this critical step and increases after it, so the candidate is its minimum. In practice, \(A\) and \(B\) are often estimated rather than known precisely. The formula gives a scale for a useful step, not an exact promise of the most accurate computed answer.

Worked Step-Size Estimates

Choosing a Step in Practice

The balance formula is most useful when the leading truncation coefficient and the scale of roundoff are reasonably estimated. For example, the first-derivative central estimate has leading coefficient \(A=|f'''(x)|/6\) when that derivative is nonzero. The second-derivative central estimate has \(A=|f^{(4)}(x)|/12\) when \(f^{(4)}(x)\neq0\). If these derivatives vanish or are poorly estimated, the leading error may have a different size, and the simple model may be conservative or misleading.

Several practical checks help:

• Keep the units of \(h\) consistent with the units of \(x\); a step size is not unitless unless the input is.
• Use the smoothness and derivative scales near \(x\) when estimating truncation error.
• Remember that \(B\epsilon\) depends on the scale of the computed function values, not just on the precision parameter.
• Compare estimates at a few step sizes near the predicted scale. A plateau or a rise in error as \(h\) shrinks can indicate the growing effect of roundoff.
• Do not apply a Taylor-based error estimate across a point where the required derivatives fail to exist or are not bounded.

When the input values themselves contain measurement error, the same amplification principle applies: perturbations in function values are divided by \(h\) or \(h^2\). In that setting, reducing arithmetic roundoff alone cannot eliminate the effects of noisy data.

A Common Mistake: Assuming Smaller Is Always Better

A tempting argument is that reducing \(h\) always improves a finite difference because its truncation error tends to zero. That argument ignores roundoff. For the model \(E(h)=Ah^2+B\epsilon/h\), halving \(h\) reduces the truncation term by a factor of four, but doubles the roundoff term.

For example, suppose \(B\epsilon=10^{-12}\) in this model. At \(h=10^{-4}\), the roundoff contribution is \(10^{-12}/10^{-4}=10^{-8}\). At the smaller step \(h=5\times10^{-5}\), it is \(10^{-12}/(5\times10^{-5})=2\times10^{-8}\). The roundoff contribution has doubled, not decreased. The correct response is to balance both terms, rather than to make \(h\) as small as possible.