The Gradient

A Vector That Represents a Scalar Derivative

For a scalar-valued function on \(\mathbb{R}^m\), the derivative at a point is a linear map from \(\mathbb{R}^m\) to \(\mathbb{R}\). In coordinates, such a map can be represented by a vector using the Euclidean dot product. The vector that represents the derivative is the gradient. It gathers the coordinate partial derivatives into one object and lets us evaluate the first-order change in any input direction with a single dot product.

Throughout, \(U\subseteq\mathbb{R}^m\) is open and \(f:U\to\mathbb{R}\) is scalar-valued. We use the Euclidean inner product \(x\cdot y\) and norm \(\|\cdot\|_2\). As established in the tutorials on Partial Derivatives and Differentiability in \(\mathbb{R}^n\), a partial derivative measures change along one coordinate direction, while differentiability means that a linear map approximates the function's total change to first order.

The gradient is written as a vector in the input space, even though \(Df(a)\) is a linear map from the input space to \(\mathbb{R}\). These are two ways to encode the same first-order information when the Euclidean inner product has been chosen. The gradient's \(j\)-th coordinate is the partial derivative in the \(j\)-th coordinate direction; its dot product with an arbitrary increment combines all those coordinate contributions.

The Derivative Is the Dot Product with the Gradient

Proof. Differentiability at \(a\) means that there is a linear map \(L:\mathbb{R}^m\to\mathbb{R}\) and a remainder \(r(h)\) such that, for \(h\) near \(0\),

Fix a coordinate index \(j\), and let \(e_j\) be the \(j\)-th standard basis vector. Since \(U\) is open, \(a+te_j\in U\) for all sufficiently small \(t\). For nonzero such \(t\), apply the differentiability formula with \(h=te_j\) and divide by \(t\):

Because \(\|te_j\|_2=|t|\), the remainder term satisfies \(\left|r(te_j)/t\right|=|r(te_j)|/\|te_j\|_2\to 0\) as \(t\to 0\). Thus the partial derivative exists and \(\partial_j f(a)=L(e_j)\). By the Unique Standard-Coordinate Representation of vectors, \(h=\sum_{j=1}^m h_j e_j\). Linearity of \(L\) now gives

This proves both the existence of the partial derivatives and the asserted representation. \(\square\)

The theorem shows why differentiability is the key hypothesis connecting the gradient to the total first-order change. The gradient's coordinates are obtained by testing the derivative on the standard basis vectors, and linearity then determines its action on every input vector.

The Size of the Gradient

The operator norm of \(Df(a)\) measures the largest absolute first-order response to a unit input. For a scalar-valued function, the gradient makes that norm especially simple: it equals the Euclidean length of the gradient. This gives a precise link between a vector's length and the size of the linear map it represents.

Proof. By the Gradient Representation of the Derivative, \(Df(a)(h)=\nabla f(a)\cdot h\). For every \(h\) with \(\|h\|_2=1\), the Cauchy–Schwarz inequality gives

Taking the supremum over all unit \(h\) shows \(\|Df(a)\|_{\mathrm{op}}\leq\|\nabla f(a)\|_2\). If \(\nabla f(a)=0\), then \(Df(a)(h)=0\) for every \(h\), so both norms are zero. If \(\nabla f(a)\neq 0\), choose the unit vector \(h=\nabla f(a)/\|\nabla f(a)\|_2\). Then

Thus the supremum over unit vectors is at least \(\|\nabla f(a)\|_2\), proving the reverse inequality and hence equality. \(\square\)

The equality depends on the Euclidean norm and inner product being used. It also covers the zero-gradient case: when \(\nabla f(a)=0\), the derivative is the zero linear map and has no nonzero first-order response in any direction.

Using the Gradient for First-Order Changes

The derivative approximation can now be written directly with the gradient. As \(h\to 0\), with \(a+h\in U\), differentiability and the representation theorem give

The dot product is the linear prediction; the remainder is negligible relative to the size of the input increment. In particular, Cauchy–Schwarz bounds the magnitude of the linear prediction by \(\|\nabla f(a)\|_2\|h\|_2\). The operator-norm theorem shows this is the best possible uniform bound for unit increments.

What the Gradient Does Not Guarantee

The gradient is defined from partial derivatives, but the existence of all partial derivatives at a point does not by itself guarantee differentiability there. Without differentiability, the partial derivatives may fail to combine into a linear approximation, and the gradient need not represent the total change. The representation theorem has differentiability as a hypothesis for this reason.

When \(f\) is differentiable at every point of a region, the gradient also gives a convenient way to express a derivative bound: the Operator Norm of a Scalar Derivative theorem identifies \(\|Df(z)\|_{\mathrm{op}}\) with \(\|\nabla f(z)\|_2\). Consequently, the Mean Value Estimate Along a Segment from the previous tutorial applies whenever this gradient norm is bounded along the segment. That estimate controls the total change in \(f\) by the bound times the segment length.

The central distinction is therefore precise: partial derivatives supply the coordinates of a candidate gradient, while differentiability ensures that this vector represents the full derivative. Once that representation is justified, the dot product describes the linear response and the gradient's norm measures the largest such response among unit increments.