Second Differentials and Matrix Hessians

From First-Order Change to Curvature

A first differential describes how a scalar function changes under a small matrix perturbation. To understand local curvature, we need the next-order change: how the first-order rate itself varies as the matrix moves. For a scalar matrix function, that second-order information is naturally a bilinear function of two perturbation directions. A Hessian matrix is one coordinate representation of this bilinear object.

The distinction matters because the second differential takes two matrix directions as inputs and returns a scalar, whereas a Hessian matrix takes vectorized coordinates as its representation. Stating the vectorization convention makes that representation unambiguous.

Here \(D^2f(X)[U,V]\) is a scalar. In particular, it is not itself a matrix. We use column-wise vectorization: \(\operatorname{vec}(X)\) stacks the columns of \(X\) into a vector in \(\mathbb{R}^{mn}\). The matrix gradient \(\nabla f(X)\) is defined by \(df_X[U]=\operatorname{tr}(\nabla f(X)^T U)\).

The Hessian has \(mn\) rows and \(mn\) columns, regardless of the shape of \(X\). One may also describe the same second-order information as a four-index array, but that representation depends more heavily on how indices are ordered. The vectorized definition fixes the ordering explicitly.

Directional Derivatives and Hessian Symmetry

A practical way to calculate curvature in a direction \(U\) is to restrict \(f\) to the line through \(X\) in that direction: define \(g(t)=f(X+tU)\) wherever \(X+tU\in\mathcal{D}\). The ordinary one-variable second derivative \(g''(0)\) is exactly \(d^2f_X[U,U]\). This gives a direct interpretation of the second differential as the rate of change of the first directional derivative.

Proof. Write \(x=\operatorname{vec}(X)\), \(u=\operatorname{vec}(U)\), and \(v=\operatorname{vec}(V)\). Since \(\mathcal{D}\) is open, the points \(x+su+tv\) remain in the coordinate domain for sufficiently small \(s,t\). Define \(\phi(s,t)=f(\operatorname{unvec}(x+su+tv))\), where \(\operatorname{unvec}\) reverses the column-wise vectorization. The chain rule gives \(\frac{\partial^2\phi}{\partial s\,\partial t}(0,0)=d^2f_X[U,V]\) and \(\frac{\partial^2\phi}{\partial t\,\partial s}(0,0)=d^2f_X[V,U]\). Because the second partial derivatives of \(f\), and therefore those of \(\phi\), are continuous, the equality of mixed partial derivatives gives \(d^2f_X[U,V]=d^2f_X[V,U]\). Taking \(U\) and \(V\) to be coordinate basis matrices shows that the corresponding Hessian entries satisfy \(\mathbf{H}_{ij}=\mathbf{H}_{ji}\). \(\square\)

Second-Order Taylor Approximation

The second differential describes the quadratic term in a local approximation to \(f\). The following result makes that statement precise. The remainder is smaller than the squared size of the perturbation, measured in Frobenius norm.

Proof. Since \(\mathcal{D}\) is open, for all sufficiently small \(E\) the segment \(X+tE\), \(0\le t\le1\), lies in \(\mathcal{D}\). Let \(g(t)=f(X+tE)\). Then \(g'(t)=df_{X+tE}[E]\) and \(g''(t)=d^2f_{X+tE}[E,E]\). Applying the fundamental theorem of calculus twice gives \[ f(X+E)=f(X)+df_X[E]+\int_0^1(1-t)d^2f_{X+tE}[E,E]\,dt. \] After subtracting \(\frac{1}{2}d^2f_X[E,E]\), the remainder is \[ \int_0^1(1-t)\big(d^2f_{X+tE}[E,E]-d^2f_X[E,E]\big)\,dt. \] Continuity of the second derivative at \(X\) implies that the operator norm of \(D^2f(X+tE)-D^2f(X)\) tends uniformly to zero over \(0\le t\le1\) as \(\|E\|_F\to0\). The absolute remainder is therefore at most \[ \frac{1}{2}\|E\|_F^2\sup_{0\le t\le1}\|D^2f(X+tE)-D^2f(X)\|_{\mathrm{op}} =o(\|E\|_F^2). \] This proves the expansion. \(\square\)

Common Pitfall: Mixing Up the Objects

A common mistake is to call \(d^2f_X[U,U]\) “the Hessian.” It is only one scalar curvature value, for one direction. The Hessian \(\mathbf{H}_f(X)\) records all coordinate second derivatives; inserting a direction into both slots of the second differential gives the scalar \(\operatorname{vec}(U)^T\mathbf{H}_f(X)\operatorname{vec}(U)\). Another source of errors is silently changing vectorization order. A Hessian computed under column-wise stacking cannot be combined with coordinates stacked in a different order without changing its entries.

The directional form is often the clearest way to state what curvature means, while the Hessian matrix is useful for coordinate calculations and software implementations. Both describe the same second-order information when the coordinate convention is fixed. In later optimization work, Hessians will be used to study objectives; here the essential point is how to define and interpret that object.