Why Bound the Remainder?
A linearization or quadratic Taylor polynomial gives a simple local approximation to a function. But an approximation is useful only if we also know how far it may be from the true value. Derivative bounds provide a way to answer that question without having to calculate the exact error at every point.
The central principle is that a small displacement from the base point, combined with bounded higher derivatives nearby, limits the approximation error. A bound is a guarantee, not necessarily a close prediction of the actual error: it may overestimate the error substantially.
Neighborhoods and Derivative Bounds
Let \(f\) be a real-valued function of \(n\) variables, let \(\mathbf{a}\) be a base point, and let \(\mathbf{h}\) be a displacement. The target point is \(\mathbf{a}+\mathbf{h}\). We will use the Euclidean norm \(\|\mathbf{h}\|\), and write \(D^k f(\mathbf{x})\) for the \(k\)th derivative at \(\mathbf{x}\), viewed as a \(k\)-linear form. For example, \(D^2 f(\mathbf{x})[\mathbf{u},\mathbf{v}]\) is the Hessian applied to two vectors.
The relevant path from the base point to the target is the line segment \(\mathbf{a}+t\mathbf{h}\), where \(0\le t\le1\). A convenient sufficient condition is that this entire segment lies in an open set on which the required derivatives exist and are continuous. A ball or box centered at \(\mathbf{a}\) is convex, so if both endpoints lie in it, the whole segment does too.
The First-Order Remainder Bound
The first-order Taylor approximation at \(\mathbf{a}\) is \(L(\mathbf{a}+\mathbf{h})=f(\mathbf{a})+Df(\mathbf{a})[\mathbf{h}]\), which is the linearization. If the second derivative has operator norm at most \(M_2\) along the segment, then
For a function of two variables, \(Df(\mathbf{a})[\mathbf{h}]=f_x(\mathbf{a})h_1+f_y(\mathbf{a})h_2\). In matrix notation the second derivative is the Hessian, and its action on \(\mathbf{h}\) twice is \(\mathbf{h}^{\top}H_f(\mathbf{x})\mathbf{h}\).
Worked Example: A Linearization Bound on a Square
Let \(f(x,y)=x^2+3xy+2y^2\), use the base point \((1,-1)\), and estimate the error in its linearization at \((1.02,-1.03)\). Consider the square \(|x-1|\le0.1,\ |y+1|\le0.1\), which contains the segment between these points.
The displacement is \(\mathbf{h}=(0.02,-0.03)\), so \(|h_1|+|h_2|=0.05\). The second partial derivatives are \(f_{xx}=2,\ f_{xy}=3,\ f_{yy}=4\). Thus every entry of the Hessian has absolute value at most \(4\) throughout the square. For any displacement \(\mathbf{u}\), this gives \(\left|\mathbf{u}^{\top}H_f\mathbf{u}\right|\le4(|u_1|+|u_2|)^2\). The one-variable Taylor formula along the segment therefore gives an error bound of \(\frac{4}{2}(0.05)^2=0.005\).
To check the approximation and the actual error, \(f(1,-1)=0\), \(f_x=2x+3y=-1\) there, and \(f_y=3x+4y=-1\) there. The linearization is \(L=0+(-1)(0.02)+(-1)(-0.03)=0.01\). Direct substitution gives \(f(1.02,-1.03)=1.0404-3.1518+2.1218=0.0104\). The error is \(0.0104-0.01=0.0004\), whose absolute value is at most \(0.005\), as guaranteed.
The Second-Order Remainder Bound
The second-order Taylor polynomial at \(\mathbf{a}\) is
If the third derivative exists, is continuous along the segment, and has operator norm at most \(M_3\) there, then the second-order remainder satisfies
For two variables, the quadratic term is \(\frac12(f_{xx}h_1^2+2f_{xy}h_1h_2+f_{yy}h_2^2)\), with the second partial derivatives evaluated at the base point. The factor \(2\) on the mixed term reflects the two equal mixed-derivative contributions when the Hessian form is expanded.
Why the Bounds Hold
The multivariable result follows by considering the function of one variable \(g(t)=f(\mathbf{a}+t\mathbf{h})\) on \(0\le t\le1\). The chain rule gives \(g'(t)=Df(\mathbf{a}+t\mathbf{h})[\mathbf{h}]\), \(g''(t)=D^2f(\mathbf{a}+t\mathbf{h})[\mathbf{h},\mathbf{h}]\), and \(g'''(t)=D^3f(\mathbf{a}+t\mathbf{h})[\mathbf{h},\mathbf{h},\mathbf{h}]\). The ordinary one-variable Taylor formulas with integral remainder apply because the specified derivatives are continuous along the segment.
The integral remainder is \(g(1)-g(0)-g'(0)=\int_0^1(1-t)g''(t)\,dt\). Since \(|g''(t)|\le M_2\|\mathbf{h}\|^2\), its absolute value is at most \(M_2\|\mathbf{h}\|^2\int_0^1(1-t)\,dt=\frac12M_2\|\mathbf{h}\|^2\).
The integral remainder is \(g(1)-g(0)-g'(0)-\frac12g''(0)=\frac12\int_0^1(1-t)^2g'''(t)\,dt\). The derivative bound gives an absolute value at most \(\frac12M_3\|\mathbf{h}\|^3\int_0^1(1-t)^2\,dt=\frac16M_3\|\mathbf{h}\|^3\).
This derivation shows why the displacement appears to the second or third power: each derivative applied to the path contributes a factor of \(\mathbf{h}\). The constants \(1/2\) and \(1/6\) come from integrating the corresponding remainder weights.
Using Coordinate Partial Derivatives
It is often easier to bound individual partial derivatives than to compute an operator norm. Suppose \(f\) has two continuous variables and \(|f_{ij}|\le K\) throughout the segment for \(i,j\in\{x,y\}\). Expanding the Hessian form gives \(|D^2f[\mathbf{h},\mathbf{h}]|\le K(|h_1|+|h_2|)^2\). Thus the first-order error is at most \(\frac K2(|h_1|+|h_2|)^2\).
Likewise, if all four third-order partial derivatives in two variables have absolute value at most \(K\) on the segment, expansion of the third derivative gives \(|D^3f[\mathbf{h},\mathbf{h},\mathbf{h}]|\le K(|h_1|+|h_2|)^3\). The second-order error is then at most \(\frac K6(|h_1|+|h_2|)^3\). These coordinate bounds can be convenient, but the resulting estimates may be looser than using the exact operator norm.
Worked Example: A Quadratic Taylor Error for an Exponential-Trigonometric Function
Let \(f(x,y)=e^x\cos y\), approximate at \((0,0)\), and estimate the error at \((0.1,-0.1)\). The segment lies in the square \(|x|\le0.1,\ |y|\le0.1\). On this square, \(e^x\le e^{0.1}\), while \(|\sin y|\le1\) and \(|\cos y|\le1\).
The value at the base point is \(1\), the first derivatives there are \(f_x=1\) and \(f_y=0\), and the second derivatives there are \(f_{xx}=1,\ f_{xy}=0,\ f_{yy}=-1\). Therefore the quadratic polynomial is \(Q(x,y)=1+x+\frac12(x^2-y^2)\). At \((0.1,-0.1)\), the quadratic term is \(\frac12(0.01-0.01)=0\), so \(Q=1.1\).
The third partial derivatives are \(f_{xxx}=e^x\cos y,\ f_{xxy}=-e^x\sin y,\ f_{xyy}=-e^x\cos y,\ f_{yyy}=e^x\sin y\). Each has absolute value at most \(e^{0.1}\) on the square. Since \(|h_1|+|h_2|=0.2\), the coordinate bound on the error is \(\frac{e^{0.1}}6(0.2)^3=\frac{0.008e^{0.1}}6\), approximately \(0.001474\).
Direct substitution gives \(f(0.1,-0.1)=e^{0.1}\cos(0.1)\), approximately \(1.099650\). The actual error is approximately \(1.099650-1.1=-0.000350\). Its absolute value is less than \(0.001474\), so the bound checks.
Worked Example: A Three-Variable Linearization Bound
Let \(f(x,y,z)=\sin(x+y+z)\), linearize at \((0,0,0)\), and estimate the error at \((0.1,-0.05,0.02)\). The segment lies in the cube \(|x|,|y|,|z|\le0.1\). Every second partial derivative is either \(-\sin(x+y+z)\) or its identical mixed-partial counterpart, so each has absolute value at most \(1\).
The linearization at the origin is \(L(x,y,z)=x+y+z\), because \(f(0,0,0)=0\) and all three first partial derivatives equal \(1\) there. The displacement has sum of absolute coordinates \(0.1+0.05+0.02=0.17\). Expanding the Hessian form and using the bound \(K=1\), the error is at most \(\frac12(0.17)^2=0.01445\).
At the target, \(x+y+z=0.07\), so \(f=\sin(0.07)\), approximately \(0.069943\), whereas \(L=0.07\). The actual error is approximately \(-0.000057\), whose absolute value is smaller than \(0.01445\). The derivative estimate is valid, though much less sharp than the actual error.
Common Pitfall: Forgetting the Remainder Factor
A frequent error is to claim that a first-order error is at most \(M_2\|\mathbf{h}\|^2\), or that a second-order error is at most \(M_3\|\mathbf{h}\|^3\), and then treat these as the standard Taylor bounds. They are safe but unnecessarily larger bounds if the stated constants really bound the derivative norms. The sharper standard estimates include the factors \(1/2\) and \(1/6\), respectively.
For example, if \(M_2=4\) and \(\|\mathbf{h}\|=0.1\), the first-order Taylor bound is \(\frac12(4)(0.1)^2=0.02\), not \(4(0.1)^2=0.04\). The latter is still an upper bound, but it misses the integral remainder factor and should not be presented as the derived Taylor estimate.
A more serious mistake is to bound derivatives only at the base point and use that value as if it controlled the whole segment. The estimates require a bound everywhere along the segment. A bound on a containing neighborhood is a convenient way to ensure this condition.
Choosing and Interpreting a Bound
To apply a remainder estimate reliably, first identify the order of approximation and the derivative order needed to control its remainder. Then verify that the required derivatives are continuous along the segment and find a valid bound there. Finally, calculate the displacement and apply the corresponding formula.
| Approximation | Derivative bound on the segment | Error bound |
|---|---|---|
| First order | \(\|D^2f\|\le M_2\) | \(\frac12M_2\|\mathbf{h}\|^2\) |
| Second order | \(\|D^3f\|\le M_3\) | \(\frac16M_3\|\mathbf{h}\|^3\) |
These estimates explain why quadratic approximations often improve rapidly as the target approaches the base point: their error is bounded by a constant times the cube of the displacement, rather than its square. The constants and the neighborhood still matter. A large derivative bound can make an estimate uninformative, while a smaller neighborhood may permit a tighter bound.
Check Your Understanding
Answer each question using the conditions and estimates developed above.
- What derivative bound controls the error in a first-order Taylor approximation, and what power of \(\|\mathbf{h}\|\) appears?
- Why must a derivative bound hold along the segment from the base point to the target, rather than only at the base point?
- If all second partial derivatives in two variables are bounded in absolute value by \(K\), what coordinate-based bound follows for the linearization error?
- What third-derivative condition gives a bound for the error in a second-order Taylor polynomial?
- Can a valid remainder bound be much larger than the actual error? Explain briefly.