Matrix Differentials as First-Order Linear Maps

Why a Differential Is a Linear Approximation

A matrix-valued function can change in many directions at once: its input may contain several entries, and its output may contain several more. The differential gives one precise way to describe that local change. At a fixed input, it is a linear map that predicts the function’s change for any sufficiently small perturbation. The approximation is useful only when its error is smaller than the perturbation itself in a precise limiting sense.

This viewpoint avoids treating a matrix as if it had just one scalar input. It also keeps two ideas distinct: the differential is a linear map, while a particular perturbation is an input to that map. We will use the Euclidean norm for vectors and the Frobenius norm for matrices. In either case, write $\|\cdot\|$ for the relevant norm.

Definition: Let $U$ be an open subset of a finite-dimensional vector space $E$, and let $F:U\to G$, where $G$ is another finite-dimensional vector space. The function $F$ is differentiable at $X\in U$ if there is a linear map $L:E\to G$ such that $$ F(X+H)=F(X)+L[H]+R(H), \qquad \lim_{\|H\|\to 0}\frac{\|R(H)\|}{\|H\|}=0. $$ The unique map $L$, when it exists, is the differential at $X$, written $DF(X)$. Its value on a perturbation $H$ is $DF(X)[H]$. For a scalar-valued function $f$, the corresponding notation is $Df(x)[h]$.

The condition on $R(H)$ says that the error is $o(\|H\|)$: after dividing by the size of the perturbation, the error tends to zero. The differential itself is linear in $H$; the remainder need not be linear. Because $U$ is open, sufficiently small perturbations of $X$ remain in the domain.

For instance, when $F$ takes matrices to matrices, both $F(X)$ and $DF(X)[H]$ are matrices of the same size. The map $DF(X)$, however, takes a perturbation matrix $H$ as input. This distinction matters: it is not generally meaningful to identify the differential with a single matrix before specifying how that matrix acts on perturbations.

The Differential Is Unique

The definition would be ambiguous if two different linear maps could both satisfy its approximation condition. They cannot. This uniqueness lets us refer to “the” differential whenever differentiability holds.

Theorem: If $F$ is differentiable at $X$, its differential is unique.

Proof. Suppose linear maps $L_1$ and $L_2$ both satisfy the definition. For $i=1,2$, write $$ F(X+H)=F(X)+L_i[H]+R_i(H), \qquad \frac{\|R_i(H)\|}{\|H\|}\longrightarrow 0. $$ Subtract the two expressions. For every sufficiently small nonzero $H$, $$ (L_1-L_2)[H]=R_2(H)-R_1(H). $$ Fix any nonzero direction $V$, and set $H=tV$ for nonzero $t$ tending to zero. Linearity and homogeneity of the norm give $$ \frac{\|(L_1-L_2)[tV]\|}{\|tV\|} = \frac{\|(L_1-L_2)[V]\|}{\|V\|} \leq \frac{\|R_1(tV)\|+\|R_2(tV)\|}{|t|\|V\|}. $$ The right-hand side tends to zero by the two remainder conditions. The left-hand side is constant in $t$, so it must be zero. Thus $(L_1-L_2)[V]=0$. This holds for every nonzero $V$, and it also holds for $V=0$ by linearity. Therefore $L_1=L_2$. $\square$

Differentiability also implies the directional derivative in every direction. For a fixed direction $V$, substitute $H=tV$ into the definition and divide the change in $F$ by $t$. The remainder satisfies $\|R(tV)\|/|t|\to 0$, so the directional derivative is $DF(X)[V]$. The converse does not follow merely from the existence of directional derivatives: differentiability requires one linear map to approximate the function uniformly over all sufficiently small perturbation directions.

Affine Maps and Scalar Quadratics

An affine map is a useful first case because it is already linear in its perturbation. If $F(x)=Ax+b$, where $A$ and $b$ are fixed and $x$ is a vector, then $$ F(x+h)-F(x)=Ah. $$ There is no remainder, so the differential is the linear map $h\mapsto Ah$. The next example makes the input, output, and perturbation explicit.

Worked Example: An Affine Vector Map

Let $$ F(x)=Ax+b,\qquad A=\begin{pmatrix}2&0&1\\-1&3&2\end{pmatrix}, \qquad b=\begin{pmatrix}1\\-2\end{pmatrix}, \qquad x=\begin{pmatrix}1\\-1\\2\end{pmatrix}. $$ Find the differential at $x$, and use it to calculate the change for $h=(0.2,-0.1,0.4)^{\mathsf T}$.

For any perturbation $h$, direct subtraction gives $$ F(x+h)-F(x)=A(x+h)+b-(Ax+b)=Ah. $$ Thus $DF(x)[h]=Ah$, independently of the value of $x$. For the stated perturbation, $$ Ah= \begin{pmatrix}2&0&1\\-1&3&2\end{pmatrix} \begin{pmatrix}0.2\\-0.1\\0.4\end{pmatrix} = \begin{pmatrix}0.4+0.4\\-0.2-0.3+0.8\end{pmatrix} = \begin{pmatrix}0.8\\0.3\end{pmatrix}. $$ This is the exact change in $F$, not just a first-order approximation, because the remainder is zero.

For a scalar function, the differential maps a vector perturbation to a scalar. Consider $f(x)=x^{\mathsf T}x$. Expanding at $x$ in the direction $h$ gives $$ f(x+h)-f(x) =x^{\mathsf T}h+h^{\mathsf T}x+h^{\mathsf T}h =2x^{\mathsf T}h+h^{\mathsf T}h. $$ The first two terms are equal because each is a scalar. The last term is the remainder: by the Cauchy–Schwarz inequality, $|h^{\mathsf T}h|=\|h\|^2$, so its ratio to $\|h\|$ tends to zero. Consequently $Df(x)[h]=2x^{\mathsf T}h$.

Worked Example: A Scalar Quadratic Differential

Let $f(x)=x^{\mathsf T}x$ for $x=(1,-2)^{\mathsf T}$, and consider $h=(0.1,0.2)^{\mathsf T}$. Find the differential’s predicted change and check the exact change.

The differential is $Df(x)[h]=2x^{\mathsf T}h$. Here, $$ x^{\mathsf T}h=(1)(0.1)+(-2)(0.2)=0.1-0.4=-0.3, \qquad Df(x)[h]=2(-0.3)=-0.6. $$ The base value is $f(x)=1^2+(-2)^2=5$. The perturbed vector is $x+h=(1.1,-1.8)^{\mathsf T}$, so $$ f(x+h)=1.1^2+(-1.8)^2=1.21+3.24=4.45. $$ The exact change is $4.45-5=-0.55$. The discrepancy from the differential is $0.05$, which is exactly $h^{\mathsf T}h=0.1^2+0.2^2=0.05$, the quadratic remainder.

A Matrix-Valued Example: Squaring a Matrix

For a matrix-valued function, the differential must return a matrix of the same size as the function’s output. Consider the square map $S(X)=X^2$, defined on all square matrices of a fixed size. Its differential can be found by expanding $S(X+H)$ directly. The expansion also shows why the order of the matrix factors must be preserved.

Theorem: For the function $S(X)=X^2$ on square matrices, the differential at $X$ is $$ DS(X)[H]=XH+HX. $$

Proof. Expand the perturbed square without changing the order of any factors: $$ S(X+H)=(X+H)(X+H)=X^2+XH+HX+H^2. $$ Therefore $$ S(X+H)-S(X)=XH+HX+H^2. $$ The map $H\mapsto XH+HX$ is linear because matrix multiplication by a fixed matrix is linear in the other factor. The remaining term is $H^2$. For the Frobenius norm, $\|AB\|_{\mathrm F}\leq \|A\|_{\mathrm F}\|B\|_{\mathrm F}$, so $$ \frac{\|H^2\|_{\mathrm F}}{\|H\|_{\mathrm F}} \leq \|H\|_{\mathrm F}\longrightarrow 0. $$ Thus $H^2$ is a valid remainder, and the stated linear map is the differential. Its uniqueness follows from the preceding theorem. $\square$

Worked Example: The Differential of a Matrix Square

Let $$ X=\begin{pmatrix}1&2\\0&-1\end{pmatrix}, \qquad H=\begin{pmatrix}0.1&-0.2\\0.3&0.1\end{pmatrix}. $$ Calculate the first-order change in $X^2$, then compare it with the exact change.

The differential gives $$ DS(X)[H]=XH+HX. $$ Multiplying in the displayed order, $$ XH=\begin{pmatrix}0.7&0\\-0.3&-0.1\end{pmatrix}, \qquad HX=\begin{pmatrix}0.1&0.4\\0.3&0.5\end{pmatrix}, $$ so the first-order change is $$ DS(X)[H]=\begin{pmatrix}0.8&0.4\\0&0.4\end{pmatrix}. $$ The exact change also includes $H^2$: $$ H^2= \begin{pmatrix}-0.05&-0.04\\0.06&-0.05\end{pmatrix}. $$ Hence $$ (X+H)^2-X^2 =DS(X)[H]+H^2 =\begin{pmatrix}0.75&0.36\\0.06&0.35\end{pmatrix}. $$ The entries differ from the differential’s prediction by precisely the entries of $H^2$, as the expansion requires.

What the Differential Does—and Does Not—Say

The differential separates a function’s local linear behavior from its higher-order change. For the affine map, the approximation is exact. For the scalar quadratic and matrix-square examples, the remainders are quadratic in the perturbation. As the perturbation shrinks, these remainders become negligible relative to its size, which is the criterion for differentiability.

A common mistake is to take a formula for a single directional derivative and call it the differential without checking that it is one linear map with a suitable remainder bound. Directional derivatives examine changes along individual lines. Differentiability demands a single linear approximation that works as the perturbation approaches zero from every direction. The distinction is important in optimization, where a first-order approximation is used to reason about changes in an objective near a candidate solution.

Another error is to reorder matrix factors as if they were scalars. In the square-map expansion, $XH$ and $HX$ are generally different matrices, so both terms must be retained. The differential is a map acting on a perturbation; it is not automatically a gradient matrix or a scalar derivative. Those representations require additional conventions.

Key takeaway: A differential is the unique linear map that leaves an error negligible relative to the perturbation. To compute one, expand the function around its input, identify the terms linear in the perturbation, and verify that the remaining terms satisfy the required limit.

Check Your Understanding

Use the definition and examples to check both the linear map and the remainder.

For an affine function $F(x)=Ax+b$, what is $DF(x)[h]$, and what is its remainder?
For $f(x)=x^{\mathsf T}x$, identify the linear term and remainder in $f(x+h)-f(x)$.
Why does $H^2$ qualify as a remainder for the matrix-square differential under the Frobenius norm?
In general, why can the directional derivatives in every direction not be used alone to conclude differentiability?
For the matrix-square function, why must both $XH$ and $HX$ appear in the differential?