The Hessian Form of a Quadratic Approximation

From Coordinate Formula to Hessian Form

A second-order Taylor approximation improves on a tangent-plane approximation by also accounting for curvature near the base point. In two variables, the quadratic terms involve the second partial derivatives, including the mixed partial derivatives that describe how changes in the two variables interact.

The Hessian packages all these second partial derivatives into one matrix. This is more than a compact way to write the formula: its quadratic form makes the roles of the pure-square terms and the mixed term visible.

Definition and Notation

Let \(f\) be a function of two variables, and fix a base point \((a,b)\). Write a nearby point as \((a+h,b+k)\), where \(h\) and \(k\) are the changes in the two coordinates. The gradient is a column vector of first partial derivatives, and the Hessian is the matrix of second partial derivatives, both evaluated at the base point.

\nabla f(a,b) = \begin{bmatrix} f_x(a,b)\\ f_y(a,b) \end{bmatrix}, \qquad \mathbf{H}_f(a,b) = \begin{bmatrix} f_{xx}(a,b) & f_{xy}(a,b)\\ f_{yx}(a,b) & f_{yy}(a,b) \end{bmatrix}, \qquad \mathbf{d} = \begin{bmatrix} h\\ k \end{bmatrix}.

Definition: The second-order Taylor polynomial of \(f\) at \((a,b)\), written in Hessian form, is

T_2(a+h,b+k) = f(a,b) + \nabla f(a,b)^{\top}\mathbf{d} + \frac{1}{2}\mathbf{d}^{\top}\mathbf{H}_f(a,b)\mathbf{d}.

The formula gives a local approximation when \(f\) has continuous second partial derivatives in a neighborhood of \((a,b)\). Under that condition, as \((h,k)\) approaches \((0,0)\),

$$ f(a+h,b+k) = T_2(a+h,b+k)+o(h^2+k^2). $$

The notation \(o(h^2+k^2)\) means that the error, divided by \(h^2+k^2\), approaches zero as \((h,k)\) approaches \((0,0)\), with \((h,k)\ne(0,0)\). The polynomial is the approximation; it need not equal \(f\) exactly unless the function has no higher-order terms.

How the Matrix Formula Expands

The Hessian formula is exactly the familiar coordinate formula written using matrix multiplication. First, the gradient term is

\nabla f(a,b)^{\top}\mathbf{d} = \begin{bmatrix}f_x(a,b)&f_y(a,b)\end{bmatrix} \begin{bmatrix}h\\k\end{bmatrix} = f_x(a,b)h+f_y(a,b)k.

For the quadratic term, multiply the Hessian by the displacement vector, then multiply on the left by the transposed displacement vector:

\begin{aligned} \mathbf{d}^{\top}\mathbf{H}_f(a,b)\mathbf{d} &= \begin{bmatrix}h&k\end{bmatrix} \begin{bmatrix}f_{xx}(a,b)&f_{xy}(a,b)\\ f_{yx}(a,b)&f_{yy}(a,b)\end{bmatrix} \begin{bmatrix}h\\k\end{bmatrix}\\ &= f_{xx}(a,b)h^2+ \bigl(f_{xy}(a,b)+f_{yx}(a,b)\bigr)hk+ f_{yy}(a,b)k^2. \end{aligned}

Therefore, before making any symmetry assumption about the mixed partials, the Hessian expression expands to

\begin{aligned} T_2(a+h,b+k) ={}&f(a,b)+f_x(a,b)h+f_y(a,b)k\\ &+\frac12 f_{xx}(a,b)h^2 +\frac12\bigl(f_{xy}(a,b)+f_{yx}(a,b)\bigr)hk +\frac12 f_{yy}(a,b)k^2. \end{aligned}

If the mixed partial derivatives are continuous near \((a,b)\), then \(f_{xy}(a,b)=f_{yx}(a,b)\). In that case the two contributions to the mixed term combine, giving the usual coordinate form:

T_2(a+h,b+k) = f(a,b)+f_x(a,b)h+f_y(a,b)k +\frac12 f_{xx}(a,b)h^2 +f_{xy}(a,b)hk +\frac12 f_{yy}(a,b)k^2.

Why Continuous Mixed Partials Agree

Here is a justification of the equality \(f_{xy}(a,b)=f_{yx}(a,b)\) under the continuity condition. Suppose both mixed partial derivatives are continuous in a neighborhood of \((a,b)\). For small nonzero \(h\) and \(k\), form the change across the corners of a rectangle:

\Delta = f(a+h,b+k)-f(a+h,b)-f(a,b+k)+f(a,b).

Apply the one-variable Fundamental Theorem of Calculus first in the \(y\)-direction to the difference \(f(a+h,y)-f(a,y)\), and then in the \(x\)-direction to the resulting difference of \(f_y\). This gives

\Delta = \int_a^{a+h}\int_b^{b+k} f_{yx}(x,y)\,dy\,dx.

Reversing the roles of the coordinates gives the other iterated-integral representation:

\Delta = \int_b^{b+k}\int_a^{a+h} f_{xy}(x,y)\,dx\,dy.

Divide each representation by \(hk\). For the first, the result is the average value of \(f_{yx}\) over the rectangle; for the second, it is the average value of \(f_{xy}\). As \(h\) and \(k\) approach zero, continuity means each average approaches the value of its integrand at \((a,b)\). Both averages are the same number \(\Delta/(hk)\) for every such rectangle, so their limits agree. Thus \(f_{yx}(a,b)=f_{xy}(a,b)\).

This is why, when the mixed partials are continuous, the Hessian is symmetric and the single coordinate cross term is \(f_{xy}(a,b)hk\). The factor \(1/2\) in the matrix formula does not leave a factor of \(1/2\) on that cross term: the two off-diagonal entries both contribute.

Worked Examples

Worked Example: A Quadratic Function at a Shifted Base Point

Find the Hessian-form quadratic approximation for \(f(x,y)=x^2+3xy+2y^2\) at \((1,-1)\), and check its value at \((1.1,-0.8)\).

First evaluate the function and its first partial derivatives at the base point:

\begin{aligned} f(1,-1)&=1^2+3(1)(-1)+2(-1)^2=1-3+2=0,\\ f_x(x,y)&=2x+3y, &f_x(1,-1)&=2-3=-1,\\ f_y(x,y)&=3x+4y, &f_y(1,-1)&=3-4=-1. \end{aligned}

The second partial derivatives are \(f_{xx}=2\), \(f_{xy}=3\), \(f_{yx}=3\), and \(f_{yy}=4\). Therefore,

\nabla f(1,-1)= \begin{bmatrix}-1\\-1\end{bmatrix}, \qquad \mathbf{H}_f(1,-1)= \begin{bmatrix}2&3\\3&4\end{bmatrix}.

The Hessian formula gives

\begin{aligned} T_2(1+h,-1+k) &=0+ \begin{bmatrix}-1&-1\end{bmatrix} \begin{bmatrix}h\\k\end{bmatrix} +\frac12 \begin{bmatrix}h&k\end{bmatrix} \begin{bmatrix}2&3\\3&4\end{bmatrix} \begin{bmatrix}h\\k\end{bmatrix}\\ &=-h-k+h^2+3hk+2k^2. \end{aligned}

At \((1.1,-0.8)\), the increments are \(h=0.1\) and \(k=0.2\), so

$$ T_2 =-0.1-0.2+(0.1)^2+3(0.1)(0.2)+2(0.2)^2 =-0.3+0.01+0.06+0.08 =-0.15. $$

Check by direct substitution: \(f(1.1,-0.8)=1.21+3(1.1)(-0.8)+2(0.64)=1.21-2.64+1.28=-0.15\). The approximation is exact here because \(f\) is already a polynomial of degree two.

Worked Example: An Approximation for an Exponential-Trigonometric Function

Find the quadratic Taylor polynomial for \(f(x,y)=e^x\cos y\) at \((0,0)\), then use it to approximate \(f(0.1,0.2)\).

The function and first derivatives at the base point are

\begin{aligned} f(0,0)&=e^0\cos 0=1,\\ f_x&=e^x\cos y, &f_x(0,0)&=1,\\ f_y&=-e^x\sin y, &f_y(0,0)&=0. \end{aligned}

The second derivatives are \(f_{xx}=e^x\cos y\), \(f_{xy}=-e^x\sin y\), \(f_{yx}=-e^x\sin y\), and \(f_{yy}=-e^x\cos y\). At \((0,0)\), these are \(1,0,0,-1\), respectively. Thus

\mathbf{H}_f(0,0)= \begin{bmatrix}1&0\\0&-1\end{bmatrix}, \qquad T_2(h,k)=1+h+\frac12(h^2-k^2).

Substitute \(h=0.1\) and \(k=0.2\):

T_2(0.1,0.2) =1+0.1+\frac12(0.01-0.04) =1.1-0.015 =1.085.

As a check against the original function, \(f(0.1,0.2)=e^{0.1}\cos(0.2)\approx 1.08314\), rounded to five decimal places. The value differs from \(1.085\), as expected: this function has terms beyond degree two, so its Taylor polynomial is a local approximation rather than an exact identity.

Worked Example: A Nonzero Mixed Term

For \(f(x,y)=x^2y\), construct the quadratic Taylor polynomial at \((1,2)\), then approximate \(f(1.1,2.2)\).

At the base point,

\begin{aligned} f(1,2)&=1^2(2)=2,\\ f_x=2xy,\quad &f_x(1,2)=4,\\ f_y=x^2,\quad &f_y(1,2)=1. \end{aligned}

The second partial derivatives are \(f_{xx}=2y\), \(f_{xy}=2x\), \(f_{yx}=2x\), and \(f_{yy}=0\). Their values at the base point are \(4,2,2,0\), so

\nabla f(1,2)= \begin{bmatrix}4\\1\end{bmatrix}, \qquad \mathbf{H}_f(1,2)= \begin{bmatrix}4&2\\2&0\end{bmatrix}.

Writing the increments as \(h\) and \(k\), the Hessian formula becomes

\begin{aligned} T_2(1+h,2+k) &=2+4h+k+ \frac12 \begin{bmatrix}h&k\end{bmatrix} \begin{bmatrix}4&2\\2&0\end{bmatrix} \begin{bmatrix}h\\k\end{bmatrix}\\ &=2+4h+k+2h^2+2hk. \end{aligned}

For \((1.1,2.2)\), \(h=0.1\) and \(k=0.2\). Therefore,

$$ T_2 =2+4(0.1)+0.2+2(0.1)^2+2(0.1)(0.2) =2+0.4+0.2+0.02+0.04 =2.66. $$

Direct substitution gives \(f(1.1,2.2)=(1.1)^2(2.2)=1.21(2.2)=2.662\). The difference is \(2.662-2.66=0.002\), consistent with the omitted higher-order terms.

A Common Mistake: Losing Half the Cross Term

A frequent error is to see the factor \(\frac12\) in the Hessian formula and carry it into the mixed term without accounting for both off-diagonal entries. For the simple function \(f(x,y)=xy\), the Hessian is

\mathbf{H}_f= \begin{bmatrix}0&1\\1&0\end{bmatrix}.

If the displacement is \(\mathbf{d}=\begin{bmatrix}2\\3\end{bmatrix}\), the incorrect calculation would keep just one off-diagonal contribution and report \(\frac12(2)(3)=3\). But the full quadratic form is

\frac12 \begin{bmatrix}2&3\end{bmatrix} \begin{bmatrix}0&1\\1&0\end{bmatrix} \begin{bmatrix}2\\3\end{bmatrix} = \frac12\bigl((2)(3)+(3)(2)\bigr) =6.

This agrees with \(f(2,3)=6\). The correction is to multiply out the entire Hessian quadratic form before simplifying. With equal mixed partials, its two off-diagonal entries contribute \(\frac12 f_{xy}hk+\frac12 f_{yx}hk=f_{xy}hk\).

Why the Hessian Form Is Useful

The Hessian form separates the local approximation into a value, a linear change, and a quadratic correction. The gradient controls the first-order change; the Hessian records second-order curvature in both coordinate directions and their interaction. Packaging the second derivatives as a matrix also makes the structure of the formula easier to recognize and reuse.

When applying the formula, evaluate every derivative at the same base point, keep the displacement vector in the order \((h,k)\), and include the factor \(\frac12\) on the whole quadratic form. If the mixed partials are continuous, the matrix is symmetric and the expanded cross term has coefficient \(f_{xy}(a,b)\), not \(\frac12 f_{xy}(a,b)\).

Check Your Understanding

Use the Hessian definition and the expanded coordinate form to answer these questions.

For a function of two variables, which four second partial derivatives appear in its Hessian matrix?
Expand \(\frac12\mathbf{d}^{\top}\mathbf{H}_f(a,b)\mathbf{d}\) and identify the coefficient of \(hk\) before assuming the mixed partials are equal.
What continuity condition ensures that \(f_{xy}(a,b)=f_{yx}(a,b)\)?
Why does the factor \(\frac12\) in the Hessian formula not remain on the mixed term when the mixed partials agree?
For \(f(x,y)=x^2+y^2\) at \((0,0)\), write its Hessian and its quadratic Taylor polynomial.