The Frobenius Inner Product and Matrix Gradients

Introduction

For a scalar function of a matrix, the differential describes how the function changes under a small matrix perturbation. The differential is a linear function of that perturbation; a matrix gradient is a convenient way to represent this linear function using an inner product. The representation is useful only when its convention is clear: in this tutorial, the gradient has the same dimensions as the matrix argument.

The key idea is that ordinary Euclidean geometry extends to the space of matrices. Its inner product is the Frobenius inner product, and it gives a precise meaning to the convention \(df=\operatorname{tr}((\nabla f)^T dX)\). The earlier material on matrix differentials as first-order linear maps supplies the notion of differential used here.

The Frobenius Inner Product

Let \(A,B\in\mathbb{R}^{m\times n}\). Write \(a_{ij}\) and \(b_{ij}\) for their entries. The following definition gives a scalar measure of how the two matrices align.

The trace expression and the entrywise expression are equal because the diagonal entries of \(A^TB\) are \((A^TB)_{jj}=\sum_{i=1}^{m}a_{ij}b_{ij}\), for \(j=1,\ldots,n\). Summing those diagonal entries gives \(\sum_{j=1}^{n}\sum_{i=1}^{m}a_{ij}b_{ij}\), which is the stated double sum. Thus the trace is a compact notation for the same ordinary dot product of the matrices’ entries.

This inner product is symmetric and bilinear, and it is positive definite: \(\langle A,A\rangle_F=\sum_{i,j}a_{ij}^2\geq 0\), with equality only when every entry of \(A\) is zero. Consequently, the space \(\mathbb{R}^{m\times n}\), with this inner product, behaves like a Euclidean space of dimension \(mn\). The Frobenius norm is the corresponding Euclidean length.

Differentials and the Gradient Convention

Suppose \(U\) is an open subset of \(\mathbb{R}^{m\times n}\) and \(f:U\to\mathbb{R}\). At \(X\in U\), differentiability means that there is a linear map \(L_X:\mathbb{R}^{m\times n}\to\mathbb{R}\) such that, for perturbations \(H\) tending to zero, \(f(X+H)=f(X)+L_X(H)+o(\|H\|_F)\). The linear map \(L_X\) is the differential at \(X\), written \(df_X\); its value at a perturbation \(H\) is \(df_X[H]\). The remainder condition means that the remainder divided by \(\|H\|_F\) tends to zero.

The differential is a linear map that takes a perturbation matrix to a scalar. The gradient is a matrix whose inner product with any perturbation returns that scalar. In particular, \(dX\) represents an arbitrary perturbation of the same dimensions as \(X\); it is not an independent matrix-valued output. The transpose in the convention ensures that \((\nabla f)^T dX\) is square and its trace is scalar, including when \(X\) is rectangular.

Why a Unique Gradient Exists

The gradient convention does not place an extra restriction on a differentiable scalar function. Every linear differential on the finite-dimensional matrix space has exactly one matrix representation under the Frobenius inner product.

Proof. Let \(E_{ij}\) be the \(m\times n\) matrix with a 1 in position \((i,j)\) and zeros elsewhere. Define \(g_{ij}=L(E_{ij})\), and let \(G\) be the matrix with entries \(g_{ij}\). Every \(H\in\mathbb{R}^{m\times n}\) can be written as \(H=\sum_{i=1}^{m}\sum_{j=1}^{n}h_{ij}E_{ij}\). By linearity, \(L(H)=\sum_{i=1}^{m}\sum_{j=1}^{n}h_{ij}L(E_{ij})=\sum_{i=1}^{m}\sum_{j=1}^{n}h_{ij}g_{ij}=\langle G,H\rangle_F\). This proves existence. If \(G_1\) and \(G_2\) both represent \(L\), then \(\langle G_1-G_2,H\rangle_F=0\) for every \(H\). Set \(H=G_1-G_2\). It follows that \(\|G_1-G_2\|_F^2=0\), so \(G_1=G_2\). For differentiable \(f\), apply the result to its linear differential \(df_X\) at each \(X\). \(\square\)

Translating Coordinate Differentials into Gradients

The theorem gives a direct calculation method. First express the differential in terms of the entries of a general perturbation \(H\). Then read off the coefficient of each \(h_{ij}\): that coefficient is the \((i,j)\) entry of the gradient. Each example below follows this procedure.

Directional Change and Steepest Increase

For any direction \(H\), the directional derivative at \(X\) is the first-order change per unit step in that direction: \(\left.\frac{d}{dt}f(X+tH)\right|_{t=0}=df_X[H]=\langle\nabla f(X),H\rangle_F\). If \(H\) is a unit direction, so that \(\|H\|_F=1\), the differential measures the change per unit Frobenius length.

Proof. Write \(G=\nabla f(X)\). The Cauchy–Schwarz inequality for the Frobenius inner product gives \(\langle G,H\rangle_F\leq|\langle G,H\rangle_F|\leq\|G\|_F\|H\|_F=\|G\|_F\) for every unit \(H\). When \(G\neq0\), choosing \(H=G/\|G\|_F\) gives \(\langle G,H\rangle_F=\|G\|_F\), so the upper bound is attained. When \(G=0\), \(\langle G,H\rangle_F=0\) for every \(H\). \(\square\)

This result explains the geometric role of the gradient: it points in the direction of greatest first-order increase, with its norm giving the rate of that increase per unit Frobenius length. It is a local statement about first-order change, not a claim that taking a step in that direction will always increase the function by the same amount.

Common Mistake: Transposing the Gradient

A frequent error is to confuse the gradient with its transpose. Under the convention here, if \(X\) is \(m\times n\), then \(\nabla f(X)\) is also \(m\times n\). The transpose appears inside the inner product: \(\operatorname{tr}((\nabla f)^T dX)\). It does not change the dimensions assigned to the gradient itself.

For example, when \(X\) is \(2\times3\), the expression \((\nabla f)^T dX\) is a \(3\times3\) matrix and its trace is scalar. Writing a \(3\times2\) matrix as the gradient instead would make its Frobenius inner product with the \(2\times3\) perturbation undefined. A related pitfall is to read coefficients in the wrong positions: the coefficient of \(h_{ij}\) belongs in gradient position \((i,j)\), not \((j,i)\).