Product, Transpose, and Trace Differential Rules

Why Matrix Differential Rules Need Care

When a matrix depends on a parameter, a small change in that parameter produces a first-order change in the matrix. For a product, that change can come from either factor; for a transpose, the dimensions switch; and for a trace, the matrix must be square. These rules look familiar from scalar calculus, but matrix multiplication is order-sensitive. A valid formula must preserve both the factor order and the dimensions at every step.

We use $dA$ to denote the first-order change in a matrix $A$, as in the differential viewpoint established in the earlier material on matrix differentials. At a parameter value $x$, if $A$ depends on $x$, then $dA$ represents the linear part of its change under a small increment in $x$. The rules below apply to differentiable matrix-valued functions; they do not require the matrices to be invertible or symmetric.

The Differential Rules and Their Dimensions

Definition: If $A$ is $m\times k$ and $B$ is $k\times n$, then $AB$ is $m\times n$, and its differential is $$ d(AB)=(dA)B+A(dB). $$ Here $dA$ is $m\times k$, $dB$ is $k\times n$, and each term on the right is $m\times n$. The factor order in each term is part of the rule.

Definition: If $A$ is $m\times n$, then $A^T$ and $dA^T=(dA)^T$ are $n\times m$. Thus $$ d(A^T)=(dA)^T. $$ If $C$ is square, its trace $\operatorname{tr}(C)$ is the sum of its diagonal entries, and $$ d\operatorname{tr}(C)=\operatorname{tr}(dC). $$

The product rule extends to any finite chain of compatible matrix factors. For example, if $A$, $B$, and $C$ have dimensions $m\times k$, $k\times r$, and $r\times n$, respectively, then

$$ d(ABC)=(dA)BC+A(dB)C+AB(dC). $$

Every term is $m\times n$. The rule is obtained by applying the two-factor product rule twice; it does not permit the factors to be reordered.

Why the Rules Hold

The product rule follows from first-order linear approximations, with the term involving two simultaneous changes contributing only a higher-order remainder. To make this precise, let $A(x)$ and $B(x)$ be differentiable matrix-valued functions at $x$, with compatible dimensions. For an increment $h$, write

A(x+h)=A(x)+DA(x)[h]+R_A(h), \qquad B(x+h)=B(x)+DB(x)[h]+R_B(h),

where $\|R_A(h)\|_F/\|h\|\to 0$ and $\|R_B(h)\|_F/\|h\|\to 0$ as $h\to 0$. Here $DA(x)[h]$ and $DB(x)[h]$ are the linear first-order changes, and $\|\cdot\|_F$ is the Frobenius norm.

Theorem: Under these differentiability and dimension conditions, the differential of a product is $d(AB)=(dA)B+A(dB)$.

Proof. Let $A=A(x)$, $B=B(x)$, $a=DA(x)[h]$, and $b=DB(x)[h]$. Expanding the product at $x+h$ gives

\begin{aligned} A(x+h)B(x+h) &=(A+a+R_A(h))(B+b+R_B(h))\\ &=AB+aB+Ab+R_A(h)B+AR_B(h)+ab\\ &\quad +aR_B(h)+R_A(h)b+R_A(h)R_B(h). \end{aligned}

The linear part in $h$ is $aB+Ab$. Each remaining term, divided in norm by $\|h\|$, tends to zero: the remainders are $o(\|h\|)$, while $a=O(\|h\|)$, $b=O(\|h\|)$, and $ab=O(\|h\|^2)$. These bounds follow from differentiability and the inequality $\|UV\|_F\leq \|U\|_F\|V\|_F$. Therefore $D(AB)(x)[h]=DA(x)[h]B+A\,DB(x)[h]$, which is the stated differential rule. $\square$

Transpose is a linear operation: it distributes over sums and scalar multiples. In the Frobenius norm, $\|M^T\|_F=\|M\|_F$, so transposing a first-order expansion preserves the size of its remainder.

Theorem: If $A(x)$ is differentiable, then $d(A^T)=(dA)^T$.

Proof. Write $A(x+h)=A(x)+DA(x)[h]+R_A(h)$. Transposing both sides gives $A(x+h)^T=A(x)^T+DA(x)[h]^T+R_A(h)^T$. Since $\|R_A(h)^T\|_F/\|h\|=\|R_A(h)\|_F/\|h\|\to 0$, the linear part of the transpose is $DA(x)[h]^T$. This proves the rule. $\square$

Finally, trace is a linear map on square matrices: it sums selected entries, so it takes a first-order expansion to the corresponding expansion of those sums. If $C(x)$ is differentiable and square, then

d\operatorname{tr}(C)=\operatorname{tr}(dC).

This rule requires $C$ to be square, but does not require its differential to be symmetric. Applying it after a product rule is often the safest way to differentiate a trace expression.

Worked Applications

Worked Example: Differentiating a Product of Rectangular Matrices

Let $A(t)$ be $2\times 3$, $B(t)$ be $3\times 2$, and suppose at $t=0$

A(0)=\begin{pmatrix}1&2&0\\-1&1&3\end{pmatrix},\quad A'(0)=\begin{pmatrix}0&1&2\\2&0&-1\end{pmatrix},

B(0)=\begin{pmatrix}1&0\\2&-1\\1&3\end{pmatrix},\quad B'(0)=\begin{pmatrix}1&2\\0&1\\-1&0\end{pmatrix}.

Find the derivative of $C(t)=A(t)B(t)$ at zero. Since $C(t)$ is $2\times2$, the product rule gives

$$ C'(0)=A'(0)B(0)+A(0)B'(0). $$

The first product is

A'(0)B(0)= \begin{pmatrix} 0(1)+1(2)+2(1)&0(0)+1(-1)+2(3)\\ 2(1)+0(2)+(-1)(1)&2(0)+0(-1)+(-1)(3) \end{pmatrix} = \begin{pmatrix}4&5\\1&-3\end{pmatrix}.

The second product is

A(0)B'(0)= \begin{pmatrix} 1(1)+2(0)+0(-1)&1(2)+2(1)+0(0)\\ (-1)(1)+1(0)+3(-1)&(-1)(2)+1(1)+3(0) \end{pmatrix} = \begin{pmatrix}1&4\\-4&-1\end{pmatrix}.

Therefore

C'(0)=\begin{pmatrix}5&9\\-3&-4\end{pmatrix}.

Both terms in the product rule are $2\times2$, as required. The result also illustrates why the two contributions must be computed separately before adding them.

Worked Example: Differentiating a Transpose

Suppose $X(t)$ is $2\times3$, with

X(0)=\begin{pmatrix}1&-2&0\\3&1&4\end{pmatrix}, \qquad X'(0)=\begin{pmatrix}2&0&-1\\-1&3&2\end{pmatrix}.

The transpose $X(t)^T$ is $3\times2$. The transpose rule says to transpose the derivative, so

\left.\frac{d}{dt}X(t)^T\right|_{t=0} =X'(0)^T =\begin{pmatrix}2&-1\\0&3\\-1&2\end{pmatrix}.

The dimensions have changed from $2\times3$ to $3\times2$, and the entries have moved across the diagonal. No additional product rule is needed because transposition itself is a linear operation.

Worked Example: Differentiating the Trace of a Product

Use the matrices and derivatives in the first example to find $g'(0)$ for $g(t)=\operatorname{tr}(A(t)B(t))$. The product is square, so the trace is defined. First apply the product rule, then the trace rule:

g'(0)=\operatorname{tr}\big(A'(0)B(0)+A(0)B'(0)\big).

From the two calculated products, their sum is $\begin{pmatrix}5&9\\-3&-4\end{pmatrix}$. Its trace is the sum of its diagonal entries:

$$ g'(0)=5+(-4)=1. $$

A direct check is available by expanding the linear paths $A(t)=A(0)+tA'(0)$ and $B(t)=B(0)+tB'(0)$. The trace at $t=0$ is $13$, since $A(0)B(0)=\begin{pmatrix}5&-2\\4&8\end{pmatrix}$. The coefficient of $t$ is the trace of $A'(0)B(0)+A(0)B'(0)$, which is $1$. Thus the derivative at zero is indeed $1$.

Common Mistakes and a Reliable Check

A frequent error is to write $d(AB)=A(dB)+(dA)B$ and then reorder terms as if matrix multiplication were commutative. In this particular expression the two summands may be written in either order because matrix addition is commutative, but the factors within each summand cannot be rearranged: $(dA)B$ is not generally $B(dA)$, and $A(dB)$ is not generally $(dB)A$. Some reordered expressions may not even have compatible dimensions.

Another mistake is to take the trace of a nonsquare matrix. The trace rule applies only after the traced matrix has been shown to be square. For example, if $A$ is $m\times k$ and $B$ is $k\times n$, then $\operatorname{tr}(AB)$ is defined only when $m=n$. The product rule still holds without that additional condition; it is the trace operation that imposes it.

Key takeaway: Before simplifying a matrix differential, check the dimensions of every factor, differentiate each changing factor once, preserve the order of all factors, and take a trace only of a square matrix. These checks catch many errors before any entrywise arithmetic is needed.

These rules are useful whenever an objective or constraint is built from matrix-valued quantities: differentiate the matrix expression first, then apply the trace rule if the result is a scalar trace. This separation keeps the calculation transparent and makes dimension errors visible. In particular, the differential rules do not authorize cyclic rearrangements inside a trace; such rearrangements require separate trace identities.

Check Your Understanding

For each question, write the dimensions as well as the differential formula.

If $A$ is $4\times2$ and $B$ is $2\times5$, what are the dimensions of $dA$, $dB$, and each term in $d(AB)$?
For compatible $A,B,C$, write the three terms in $d(ABC)$ in their correct factor order.
If $X$ is $3\times4$, what are the dimensions of $d(X^T)$, and how is it related to $dX$?
What condition on $C$ is needed for $d\operatorname{tr}(C)=\operatorname{tr}(dC)$?
Explain why $\operatorname{tr}(AB)$ is not defined for every pair of matrices for which $AB$ exists.