Curvature in Opposite Directions
A stationary point can look like a local high point in one direction and a local low point in another. This is the characteristic behavior of a saddle: nearby function values occur both above and below the value at the point. For a twice-differentiable function of two variables, an indefinite Hessian at a stationary point guarantees this behavior.
The key is to examine curvature along selected directions. At a stationary point, the first-order change vanishes. The second-order change is then determined by the Hessian’s quadratic form. If that form is positive in one direction and negative in another, the function rises along one nearby line and falls along another.
For a function \(f(x,y)\), the Hessian at a point \((a,b)\) is the matrix of second partial derivatives \(\mathbf{H}_f(a,b)=\begin{pmatrix} f_{xx}(a,b)&f_{xy}(a,b)\\ f_{yx}(a,b)&f_{yy}(a,b)\end{pmatrix}\). When \(f\) has continuous second partial derivatives near the point, the mixed partials agree, so this Hessian is symmetric. Its quadratic form measures the second-order change in a direction.
In two variables, a symmetric Hessian with negative determinant is indefinite. That criterion can help identify the situation, but the geometric reason for the classification is the existence of directions where the quadratic form has opposite signs.
Why an Indefinite Hessian Rules Out Extrema
Suppose \(f\) has continuous second partial derivatives near an interior point \(\mathbf{p}=(a,b)\), and \(\nabla f(\mathbf{p})=\mathbf{0}\). The second-order Taylor expansion at \(\mathbf{p}\) says that for a small displacement \(\mathbf{h}\), \[ f(\mathbf{p}+\mathbf{h})=f(\mathbf{p})+\nabla f(\mathbf{p})^{\top}\mathbf{h} +\frac12\mathbf{h}^{\top}\mathbf{H}_f(\mathbf{p})\mathbf{h} +o(\|\mathbf{h}\|^2). \] Here the remainder, divided by \(\|\mathbf{h}\|^2\), tends to zero as \(\mathbf{h}\) tends to \(\mathbf{0}\). The stationary-point condition makes the linear term zero.
Now choose a fixed direction \(\mathbf{u}\) and move by \(\mathbf{h}=t\mathbf{u}\). The expansion becomes \(f(\mathbf{p}+t\mathbf{u})=f(\mathbf{p})+\frac12t^2\mathbf{u}^{\top}\mathbf{H}_f(\mathbf{p})\mathbf{u}+o(t^2)\). The sign of the quadratic-form value controls the change for sufficiently small nonzero \(t\). The following argument makes that sign claim precise.
Indefiniteness gives a vector \(\mathbf{u}\) with \(c=\mathbf{u}^{\top}\mathbf{H}_f(\mathbf{p})\mathbf{u}>0\). Along \(\mathbf{p}+t\mathbf{u}\), the change is \(\frac12ct^2+r_{\mathbf{u}}(t)\), where \(r_{\mathbf{u}}(t)=o(t^2)\).
Because \(r_{\mathbf{u}}(t)=o(t^2)\), for all sufficiently small nonzero \(t\), \(|r_{\mathbf{u}}(t)|<\frac14ct^2\). Thus the change is greater than \(\frac14ct^2>0\).
Indefiniteness also gives a vector \(\mathbf{v}\) with \(d=\mathbf{v}^{\top}\mathbf{H}_f(\mathbf{p})\mathbf{v}<0\). For sufficiently small nonzero \(t\), its remainder has magnitude less than \(\frac14|d|t^2\), so the change along \(\mathbf{p}+t\mathbf{v}\) is less than \(-\frac14|d|t^2<0\).
There are points arbitrarily close to \(\mathbf{p}\) where \(f\) is larger, and points arbitrarily close where it is smaller. Therefore \(\mathbf{p}\) is neither a local maximum nor a local minimum.
This proves the result using the Taylor expansion and the definition of an indefinite quadratic form. The condition that the point be stationary matters: without it, the linear term need not vanish and can dominate the second-order term near the point.
Worked Examples
Worked Example: Opposite Curvature Along the Coordinate Axes
Classify the stationary point at the origin for \(f(x,y)=x^2-y^2\).
First calculate the gradient: \(\nabla f(x,y)=(2x,-2y)\), so \(\nabla f(0,0)=(0,0)\). The origin is stationary.
The second partial derivatives are \(f_{xx}=2\), \(f_{xy}=0\), \(f_{yx}=0\), and \(f_{yy}=-2\). Therefore, \(\mathbf{H}_f(0,0)=\begin{pmatrix}2&0\\0&-2\end{pmatrix}\).
For \(\mathbf{u}=(1,0)^{\top}\), the quadratic-form value is \(\mathbf{u}^{\top}\mathbf{H}_f(0,0)\mathbf{u}=2>0\). For \(\mathbf{v}=(0,1)^{\top}\), it is \(\mathbf{v}^{\top}\mathbf{H}_f(0,0)\mathbf{v}=-2<0\). The Hessian is indefinite.
Check directly along the two directions. On \((x,y)=(t,0)\), \(f(t,0)=t^2>0=f(0,0)\) for \(t\ne0\). On \((x,y)=(0,t)\), \(f(0,t)=-t^2<0=f(0,0)\) for \(t\ne0\). Both kinds of values occur arbitrarily close to the origin, so it is a saddle point.
Worked Example: Mixed Partial Derivatives and Direction Tests
Classify the stationary point at the origin for \(f(x,y)=3x^2+4xy-y^2\).
The first partial derivatives are \(f_x=6x+4y\) and \(f_y=4x-2y\). Both are zero at \((0,0)\), so the origin is stationary.
The Hessian is \(\mathbf{H}_f(0,0)=\begin{pmatrix}6&4\\4&-2\end{pmatrix}\). Using the coordinate directions, \(\mathbf{u}=(1,0)^{\top}\) gives \(\mathbf{u}^{\top}\mathbf{H}_f(0,0)\mathbf{u}=6>0\), while \(\mathbf{v}=(0,1)^{\top}\) gives \(\mathbf{v}^{\top}\mathbf{H}_f(0,0)\mathbf{v}=-2<0\). Thus the Hessian is indefinite.
Along \((t,0)\), the function value is \(3t^2>0=f(0,0)\) when \(t\ne0\). Along \((0,t)\), it is \(-t^2<0=f(0,0)\). These substitutions verify the two signs directly. The origin is neither a local maximum nor a local minimum, so it is a saddle point.
Worked Example: A Saddle Point Away from the Origin
Classify the point \((1,-2)\) for \(f(x,y)=(x-1)^2-2(y+2)^2+(x-1)^3\).
Set \(f_x=2(x-1)+3(x-1)^2\) and \(f_y=-4(y+2)\). At \((1,-2)\), both derivatives are zero, so this is a stationary point.
The second partial derivatives are \(f_{xx}=2+6(x-1)\), \(f_{xy}=f_{yx}=0\), and \(f_{yy}=-4\). Hence \(\mathbf{H}_f(1,-2)=\begin{pmatrix}2&0\\0&-4\end{pmatrix}\). The directions \(\mathbf{u}=(1,0)^{\top}\) and \(\mathbf{v}=(0,1)^{\top}\) give quadratic-form values \(2>0\) and \(-4<0\), respectively.
To check the function values, first note that \(f(1,-2)=0\). Along \((1+t,-2)\), the value is \(t^2+t^3=t^2(1+t)>0\) for \(0<|t|<1\). Along \((1,-2+t)\), the value is \(-2t^2<0\) for every \(t\ne0\). The point is therefore a saddle.
Using the Test Carefully
An indefinite Hessian gives a decisive conclusion only when it is evaluated at a stationary point and the second-order Taylor expansion applies. In particular, the function must have the required second-order regularity near an interior point. The conclusion is local: it describes function values in every sufficiently small neighborhood, not the global shape of the surface.
One common mistake is to look at just one second partial derivative and treat its sign as a classification. For \(f(x,y)=x^2-y^2\), for instance, \(f_{xx}(0,0)=2>0\). A mistaken calculation might conclude, “the curvature is positive, so the origin is a local minimum.” But the \(y\)-direction gives \(f(0,t)=-t^2<0=f(0,0)\), disproving that conclusion. The \(x\)-direction gives \(f(t,0)=t^2>f(0,0)\), so the origin is not a local maximum either. The correction is to examine the full Hessian’s quadratic form in at least one direction of each sign.
Another error is to call a stationary point a saddle merely because the Hessian is indefinite without checking the hypotheses. A point on the boundary of a domain does not have all nearby directions available, and at a point that is not stationary, the first-order term can change the behavior. Check the point, the gradient, the domain, and the Hessian before applying the argument.
Check Your Understanding
Use the stationary-point condition and directional curvature to answer these questions.
- What does it mean for a Hessian’s quadratic form to be indefinite?
- Why does the Taylor expansion at a stationary point have no linear term?
- For \(f(x,y)=x^2-3y^2\), identify two directions that show the Hessian is indefinite at the origin.
- Why does finding positive and negative changes arbitrarily close to a point rule out both a local maximum and a local minimum?
- Name two hypotheses that should be checked before using the indefinite-Hessian argument.