Worked Trace-Differentiation Problems

Why Work Through the Scalar Differential?

A matrix expression can look compact while hiding several opportunities for error: a transpose may be missing, factors may not have compatible dimensions, or a gradient may be written in an orientation inconsistent with the chosen inner product. A reliable approach is to compute the first-order change in the scalar objective, rearrange it into a trace involving the perturbation, and then read off the gradient.

This tutorial uses that approach on trace products, quadratic forms, and objectives involving an inverse matrix. The goal is not to memorize a list of formulas, but to see why each formula has its particular order of factors and transpose.

Definition: For a differentiable scalar function $f$ of a real $m\times n$ matrix $X$, its Frobenius gradient $\nabla_X f$ is the $m\times n$ matrix satisfying $$ df=\operatorname{tr}\big((\nabla_X f)^T\,dX\big). $$ Here $df$ is the first-order differential, and $dX$ is an arbitrary $m\times n$ perturbation. The gradient is unique because the Frobenius inner product $\langle U,V\rangle_F=\operatorname{tr}(U^T V)$ is an inner product on the space of $m\times n$ matrices.

In particular, if the differential has been rearranged to $df=\operatorname{tr}(M\,dX)$, then $(\nabla_X f)^T=M$, so $\nabla_X f=M^T$. This last transpose is a frequent source of mistakes. The dimensions provide a useful check: the gradient must have the same dimensions as $X$.

Trace Products: Rearrange, Then Read Off

Suppose $X$ is $m\times n$, $B$ is $n\times p$, and $A$ is $m\times p$. Then $A^T X B$ is $p\times p$, so its trace is a scalar. The trace identities $\operatorname{tr}(UV)=\operatorname{tr}(VU)$ when the products are defined, and $\operatorname{tr}(M)=\operatorname{tr}(M^T)$, let us put the differential in the form required by the gradient definition.

Theorem: For fixed real matrices $A\in\mathbb{R}^{m\times p}$ and $B\in\mathbb{R}^{n\times p}$, define $f(X)=\operatorname{tr}(A^T X B)$, where $X\in\mathbb{R}^{m\times n}$. Then $$ \nabla_X f=AB^T. $$

Proof. Since $f$ is linear in $X$, its differential is $df=\operatorname{tr}(A^T\,dX\,B)$. By cyclically moving the final factor to the front, $df=\operatorname{tr}(BA^T\,dX)$. The defining gradient equation requires $(\nabla_X f)^T=BA^T$, hence $\nabla_X f=AB^T$. The product $AB^T$ is $m\times n$, as required. $\square$

Worked Example: A Rectangular Trace Product

Let $X$ be $2\times2$, and take $$ A=\begin{pmatrix}1&2\\-1&0\end{pmatrix}, \qquad B=\begin{pmatrix}2&1\\3&-1\end{pmatrix}. $$ For $f(X)=\operatorname{tr}(A^T X B)$, the theorem gives $$ \nabla_X f=AB^T =\begin{pmatrix}1&2\\-1&0\end{pmatrix} \begin{pmatrix}2&3\\1&-1\end{pmatrix} =\begin{pmatrix}4&1\\-2&-3\end{pmatrix}. $$

To check the result at $X_0=\begin{pmatrix}1&2\\0&-1\end{pmatrix}$, first compute $$ X_0B=\begin{pmatrix}8&-1\\-3&1\end{pmatrix}, \qquad A^T X_0B=\begin{pmatrix}11&-2\\16&-2\end{pmatrix}. $$ Thus $f(X_0)=11+(-2)=9$. The gradient expression also gives the linear representation $f(X)=\sum_{i,j}(AB^T)_{ij}X_{ij}$; at $X_0$ this is $4(1)+1(2)+(-2)(0)+(-3)(-1)=9$. The two calculations agree.

Quadratic Forms: Only the Symmetric Part Contributes

For a vector $x\in\mathbb{R}^q$ and a fixed $q\times q$ matrix $A$, consider the scalar $x^T A x$. A nonsymmetric $A$ is allowed. Since a scalar equals its transpose, $x^T A x=x^T A^T x$, and therefore $$ x^T A x=x^T\left(\frac{A+A^T}{2}\right)x. $$ The antisymmetric part has no effect on the value. Its lack of contribution also appears directly in the differential.

Result: For $f(x)=x^T A x$, with $A$ fixed and $x\in\mathbb{R}^q$, $$ \nabla_x f=(A+A^T)x. $$ If $A$ is symmetric, this reduces to $2Ax$.

Derivation. Perturb $x$ by $dx$. Applying the product rule gives $df=(dx)^T A x+x^T A\,dx$. The first scalar can be transposed without changing its value, so $(dx)^T A x=x^T A^T dx$. Hence $$ df=x^T(A^T+A)\,dx=((A+A^T)x)^T dx. $$ Comparing this with $df=(\nabla_x f)^Tdx$ proves the result. No symmetry assumption on $A$ was needed. $\square$

Worked Example: A Quadratic Form with a Nonsymmetric Matrix

Take $$ A=\begin{pmatrix}2&3\\-1&4\end{pmatrix}, \qquad x_0=\begin{pmatrix}1\\-2\end{pmatrix}. $$ First, $Ax_0=\begin{pmatrix}-4\\-9\end{pmatrix}$, so $f(x_0)=x_0^TAx_0=1(-4)+(-2)(-9)=14$.

The symmetric part, doubled, is $A+A^T=\begin{pmatrix}4&2\\2&8\end{pmatrix}$. Therefore $$ \nabla_x f\big|_{x_0} =(A+A^T)x_0 =\begin{pmatrix}4&2\\2&8\end{pmatrix} \begin{pmatrix}1\\-2\end{pmatrix} =\begin{pmatrix}0\\-14\end{pmatrix}. $$ For a direction $h=\begin{pmatrix}0.2\\0.1\end{pmatrix}$, the first-order change per unit step in that direction is $(\nabla_x f)^T h=0(0.2)+(-14)(0.1)=-1.4$.

As a direct check, $Ah=\begin{pmatrix}0.7\\0.2\end{pmatrix}$, $Ax_0=\begin{pmatrix}-4\\-9\end{pmatrix}$, and the product-rule directional derivative is $h^TAx_0+x_0^TAh$. These terms are $0.2(-4)+0.1(-9)=-1.7$ and $1(0.7)+(-2)(0.2)=0.3$, respectively. Their sum is $-1.4$, agreeing with the gradient calculation.

Inverse-Matrix Objectives: Differentiate Before Rearranging

An inverse $X^{-1}$ is defined only when $X$ is square and invertible. On the open set of invertible matrices, differentiating the identity $XX^{-1}=I$ gives the inverse differential. The resulting formula is then inserted into the scalar objective; trace rearrangement comes afterward.

Theorem: Let $X\in\mathbb{R}^{q\times q}$ be invertible, and let $B,C\in\mathbb{R}^{q\times p}$ be fixed. For $f(X)=\operatorname{tr}(C^T X^{-1}B)$, $$ \nabla_X f=-X^{-T}CB^T X^{-T}, $$ where $X^{-T}=(X^{-1})^T$.

Proof. Differentiating $XX^{-1}=I$ by the product rule gives $dX\,X^{-1}+X\,d(X^{-1})=0$. Multiplication on the left by $X^{-1}$ yields $d(X^{-1})=-X^{-1}(dX)X^{-1}$. Consequently, $$ df=\operatorname{tr}\big(C^T d(X^{-1})B\big) =-\operatorname{tr}\big(C^T X^{-1}(dX)X^{-1}B\big). $$ Cyclically rearranging the trace puts the perturbation at the end: $$ df=-\operatorname{tr}\big(X^{-1}BC^TX^{-1}\,dX\big). $$ By the gradient definition, $(\nabla_X f)^T=-X^{-1}BC^TX^{-1}$. Transposing both sides gives $\nabla_X f=-X^{-T}CB^TX^{-T}$, as claimed. The dimensions are $q\times q$, matching $X$. $\square$

Worked Example: A Bilinear Form Through an Inverse

Let $X_0=\begin{pmatrix}2&0\\0&1\end{pmatrix}$, $b=\begin{pmatrix}1\\2\end{pmatrix}$, and $c=\begin{pmatrix}3\\-1\end{pmatrix}$. Set $f(X)=c^T X^{-1}b$, a special case of the theorem with $B=b$ and $C=c$. Since $X_0^{-1}=X_0$, $X_0^{-1}b=\begin{pmatrix}0.5\\2\end{pmatrix}$, and $f(X_0)=3(0.5)-2=-0.5$.

The two vectors needed for the gradient are $X_0^{-T}c=\begin{pmatrix}1.5\\-1\end{pmatrix}$ and $X_0^{-1}b=\begin{pmatrix}0.5\\2\end{pmatrix}$. Thus $$ \nabla_X f\big|_{X_0} =-(X_0^{-T}c)(X_0^{-1}b)^T =-\begin{pmatrix}1.5\\-1\end{pmatrix} \begin{pmatrix}0.5&2\end{pmatrix} =\begin{pmatrix}-0.75&-3\\0.5&2\end{pmatrix}. $$

For the perturbation $H=\begin{pmatrix}1&0.3\\0.3&-0.2\end{pmatrix}$, the Frobenius inner product gives $$ df[H]=\operatorname{tr}\big((\nabla_X f)^T H\big) =-0.75-3(0.3)+0.5(0.3)+2(-0.2)=-1.9. $$ The inverse differential provides an independent check. With $u=X_0^{-T}c=\begin{pmatrix}1.5\\-1\end{pmatrix}$ and $v=X_0^{-1}b=\begin{pmatrix}0.5\\2\end{pmatrix}$, it gives $df[H]=-u^THv$. Since $Hv=\begin{pmatrix}1.1\\-0.25\end{pmatrix}$, $u^THv=1.5(1.1)+(-1)(-0.25)=1.9$. The directional derivative is again $-1.9$.

A Practical Checklist and a Common Pitfall

These calculations become more reliable when each one follows the same sequence. First check that every product is dimensionally defined and that the objective is scalar. Next take the differential using the product rule or the inverse differential. Then use trace identities to place $dX$ at the end, and transpose the coefficient to match the Frobenius-gradient definition. Finally, check the gradient’s dimensions and, when practical, compare its inner product with a test perturbation against the differential derived directly.

Differentiate the expression.
Keep the order of matrix factors unchanged while applying the product rule or the inverse rule.

Collect the perturbation.
Use trace identities to write the scalar differential as $\operatorname{tr}(M\,dX)$, or as $v^Tdx$ for a vector argument.

Identify and check the gradient.
Set $(\nabla_X f)^T=M$, then check its dimensions and, if useful, its action on a chosen direction.

Common pitfall: treating a trace coefficient as the gradient without checking orientation. From $df=\operatorname{tr}(M\,dX)$, it is tempting to write $\nabla_X f=M$. That is generally wrong: the definition uses $\operatorname{tr}((\nabla_X f)^T dX)$, so the correct gradient is $M^T$. For example, in the trace-product theorem the coefficient after rearranging is $BA^T$; the gradient is its transpose, $AB^T$, not $BA^T$. A second related error is replacing $A$ by $2A$ in the quadratic-form result when $A$ is not symmetric. The correct formula is $(A+A^T)x$; $2Ax$ follows only when $A=A^T$.

The scalar-differential method is useful because it separates two tasks: finding the first-order change and expressing that change as an inner product. It also makes assumptions visible. The inverse example, for instance, requires a square invertible $X$; the trace-product and quadratic-form calculations do not. Keeping the domain, dimensions, and gradient convention explicit is part of the calculation, not an optional final check.

Check Your Understanding

Use a scalar differential in each problem, and check the shape of the result.

For $X\in\mathbb{R}^{m\times n}$, $A\in\mathbb{R}^{m\times p}$, and $B\in\mathbb{R}^{n\times p}$, find the Frobenius gradient of $\operatorname{tr}(A^T X B)$.
Why does the gradient of $x^T A x$ involve $A+A^T$, even when $A$ is not symmetric?
State the domain condition needed to differentiate $X^{-1}$, and give its differential.
Suppose $df=\operatorname{tr}(M\,dX)$. Which matrix is the Frobenius gradient, and why?
For $f(X)=c^T X^{-1}b$, identify the gradient from the trace-inverse result using $B=b$ and $C=c$.