Why Tangent Directions Matter
A constraint restricts which directions of motion are allowed. At a constrained local maximum or minimum, the objective cannot increase or decrease to first order along any feasible motion through the point. The geometry of the constraint tells us which directions are feasible, and gradients describe how the objective changes in those directions.
This tutorial develops that geometric connection for one smooth equality constraint. The key conclusion is that, at a regular constrained extremum, the objective gradient is perpendicular to the constraint’s tangent directions. Since the constraint gradient is perpendicular to those same directions, the two gradients must be parallel.
Level Sets and Regular Points
For a function \(g:\mathbb{R}^n\to\mathbb{R}\), the level set \(g=c\) is the set of points \(\mathbf{x}\) satisfying \(g(\mathbf{x})=c\). It is the feasible set when the constraint is \(g(\mathbf{x})=c\). Throughout the geometric argument, assume \(f\) and \(g\) are continuously differentiable on an open neighborhood of the point under consideration.
For a regular level set, the implicit function theorem gives a local smooth surface of dimension \(n-1\). Its tangent space at \(\mathbf{p}\) consists of the velocities of differentiable feasible curves through \(\mathbf{p}\). In particular, every vector in the nullspace of the derivative of \(g\) at \(\mathbf{p}\) is such a velocity.
The dot product condition follows by differentiating \(g(\boldsymbol{\gamma}(t))=c\) along a differentiable feasible curve \(\boldsymbol{\gamma}\) with \(\boldsymbol{\gamma}(0)=\mathbf{p}\). The chain rule gives \(\nabla g(\mathbf{p})\cdot\boldsymbol{\gamma}'(0)=0\). Conversely, regularity and the implicit function theorem ensure that every vector satisfying this condition is realized as the velocity of a local feasible curve. The regularity assumption matters: without it, the level set may not have this smooth local description.
How the Objective Changes Along a Feasible Curve
Suppose \(\mathbf{p}\) is a constrained local extremum of \(f\) on \(g=c\). This means that among feasible points sufficiently close to \(\mathbf{p}\), the value \(f(\mathbf{p})\) is either no smaller than all nearby values or no larger than all nearby values.
Take any differentiable feasible curve \(\boldsymbol{\gamma}(t)\) through \(\mathbf{p}\), defined for \(t\) on an interval around zero. The one-variable function \(F(t)=f(\boldsymbol{\gamma}(t))\) has a local extremum at \(t=0\). Therefore \(F'(0)=0\). By the chain rule,
Since every tangent vector is the velocity of such a feasible curve at a regular point, the objective gradient is perpendicular to every vector in \(T_{\mathbf{p}}\). The constraint gradient is also perpendicular to every vector in \(T_{\mathbf{p}}\), by the definition of the tangent space.
The tangent space has dimension \(n-1\), so its orthogonal complement has dimension one. The nonzero vector \(\nabla g(\mathbf{p})\) lies in that orthogonal complement. Thus every vector perpendicular to the tangent space is a scalar multiple of \(\nabla g(\mathbf{p})\). This gives the geometric reason for parallel gradients.
Derivation Through Tangent Directions
The argument can be organized as a short proof. It uses differentiability, the regularity of the constraint, and the one-variable fact that a differentiable function at an interior local extremum has derivative zero.
At a regular point, each tangent vector \(\mathbf{v}\) satisfies \(\nabla g(\mathbf{p})\cdot\mathbf{v}=0\), and each such vector is the velocity of a differentiable feasible curve.
Along any such feasible curve, \(f\) has a one-variable local extremum at the parameter value corresponding to \(\mathbf{p}\). The derivative along the curve is therefore zero.
For every tangent vector \(\mathbf{v}\), the chain rule gives \(\nabla f(\mathbf{p})\cdot\mathbf{v}=0\). Thus \(\nabla f(\mathbf{p})\) is perpendicular to the tangent space.
The tangent space has codimension one, and its orthogonal complement is spanned by the nonzero normal \(\nabla g(\mathbf{p})\). Hence \(\nabla f(\mathbf{p})=\lambda\nabla g(\mathbf{p})\) for some scalar \(\lambda\).
If \(\nabla f(\mathbf{p})=\mathbf{0}\), the conclusion still holds, with \(\lambda=0\). Otherwise, both gradients are nonzero normal vectors to the same tangent space, so they point along the same line, though they need not point in the same direction.
Worked Example: Maximizing a Coordinate on a Circle
Worked Example: Maximizing a Coordinate on a Circle
Let \(f(x,y)=x\), subject to \(g(x,y)=x^2+y^2=1\). Consider \(\mathbf{p}=(1,0)\). First, \(\mathbf{p}\) is feasible because \(1^2+0^2=1\). It is a global maximum of \(f\) on the circle: \(x^2+y^2=1\) implies \(x^2\le 1\), so \(x\le 1\), with equality at \((1,0)\).
The gradients are \(\nabla f=(1,0)\) and \(\nabla g=(2x,2y)\). At \(\mathbf{p}\),
The tangent vectors \((a,b)\) satisfy \((2,0)\cdot(a,b)=2a=0\), so they have the form \((0,b)\). The objective gradient has zero dot product with each: \((1,0)\cdot(0,b)=0\).
A feasible curve that checks this tangent direction is \(\boldsymbol{\gamma}(t)=(\cos t,\sin t)\). It lies on the circle since \(\cos^2t+\sin^2t=1\), and \(\boldsymbol{\gamma}'(0)=(0,1)\). Along it, \(f(\boldsymbol{\gamma}(t))=\cos t\), whose derivative at zero is \(-\sin 0=0\). The curve, the tangent condition, and the parallel gradients agree.
Worked Example: A Product on a Circle
Worked Example: A Product on a Circle
Let \(f(x,y)=xy\) and constrain \(g(x,y)=x^2+y^2=2\). Consider \(\mathbf{p}=(1,1)\). It is feasible because \(1^2+1^2=2\). It is a global maximum: \((x-y)^2\ge0\) implies \(2xy\le x^2+y^2=2\), so \(xy\le1\), and equality holds at \((1,1)\).
The gradients are \(\nabla f=(y,x)\) and \(\nabla g=(2x,2y)\). At the point,
The tangent condition for \(\mathbf{v}=(a,b)\) is \((2,2)\cdot(a,b)=2a+2b=0\), or \(b=-a\). For every such direction, \((1,1)\cdot(a,-a)=a-a=0\).
To check with a feasible curve, take \(\boldsymbol{\gamma}(t)=(\sqrt{2}\cos(\frac{\pi}{4}+t),\sqrt{2}\sin(\frac{\pi}{4}+t))\). The sum of the squared coordinates is \(2\), and \(\boldsymbol{\gamma}(0)=(1,1)\). Along the curve, \(xy=\sin(\frac{\pi}{2}+2t)=\cos(2t)\), whose derivative is \(-2\sin(2t)\), equal to zero at \(t=0\). Thus the objective has no first-order change in its feasible tangent direction.
Worked Example: A Nonzero Tangential Change Rules Out an Extremum
Worked Example: A Nonzero Tangential Change Rules Out an Extremum
Let \(f(x,y)=x+y\), subject to \(g(x,y)=x=0\), and consider \(\mathbf{p}=(0,0)\). The point is feasible, and the constraint is regular because \(\nabla g=(1,0)\ne(0,0)\). Its tangent vectors satisfy \((1,0)\cdot(a,b)=a=0\), so the feasible tangent directions are vertical.
The objective gradient is \(\nabla f=(1,1)\). For the tangent vector \(\mathbf{v}=(0,1)\), the directional change is
This correctly signals that \(\mathbf{p}\) is not a constrained extremum. In fact, the feasible curve \(\boldsymbol{\gamma}(t)=(0,t)\) satisfies \(f(\boldsymbol{\gamma}(t))=t\), which is larger for positive \(t\) and smaller for negative \(t\). The gradients \((1,1)\) and \((1,0)\) are not parallel, consistent with the necessary condition failing.
What the Condition Does—and Does Not—Say
The gradient condition is necessary at a regular constrained extremum, but parallel gradients alone do not establish a maximum or minimum. A candidate could be a constrained local minimum, a constrained local maximum, or neither. Further analysis is needed to classify candidates or compare their values.
It is also important that “tangent direction” means a direction compatible with the constraint, not an arbitrary vector in the ambient space. At a regular point, the constraint gradient is normal to the feasible surface; its dot product with a tangent vector must vanish. The objective must have zero first-order change along all those tangent directions at an extremum.
A Common Mistake: Testing a Direction That Violates the Constraint
Return to \(f(x,y)=x\) on the unit circle at \((1,0)\). A mistaken test might choose the direction \((1,0)\) and calculate
From this nonzero value, one might wrongly conclude that the point cannot be an extremum. But \((1,0)\) is not tangent to the circle at \((1,0)\): it fails the tangent condition, since \(\nabla g(1,0)\cdot(1,0)=(2,0)\cdot(1,0)=2\ne0\). It points normal to the circle, not along a feasible first-order motion.
The correct tangent directions are \((0,b)\). For them, \(\nabla f(1,0)\cdot(0,b)=0\), as required. The error was not in the dot-product calculation; it was in using a direction that does not respect the constraint.
Why Regularity Cannot Be Dropped
At a singular point, the gradient of the constraint may fail to describe the feasible set’s local geometry. For example, take \(g(x,y)=x^2+y^2\), \(c=0\), and \(f(x,y)=x\). The feasible set consists only of \((0,0)\), because a sum of squares is zero only when both coordinates are zero. Therefore \((0,0)\) is both a constrained maximum and minimum of \(f\) on this one-point set.
However, \(\nabla g(0,0)=(0,0)\), while \(\nabla f(0,0)=(1,0)\). The constraint is not regular there, and its zero gradient does not supply a normal direction that captures the isolated feasible set. This is why the parallel-gradient conclusion requires \(\nabla g(\mathbf{p})\ne\mathbf{0}\).
Check Your Understanding
Use tangent directions and the gradient interpretation to answer these questions.
- For a regular constraint \(g=c\), what equation must a tangent vector \(\mathbf{v}\) satisfy at \(\mathbf{p}\)?
- Why does a constrained local extremum imply that \(\nabla f(\mathbf{p})\cdot\mathbf{v}=0\) for every tangent vector \(\mathbf{v}\)?
- What role does the assumption \(\nabla g(\mathbf{p})\ne\mathbf{0}\) play in the geometric argument?
- On the unit circle at \((1,0)\), is \((1,0)\) a tangent direction? Verify using the tangent condition.
- Does parallelism of \(\nabla f\) and \(\nabla g\) alone prove that a constrained point is a maximum or minimum? Explain.