Trace Identities for Scalar Matrix Functions

Why Rewrite Scalar Matrix Expressions?

A matrix expression can represent a single number in several different ways. Those forms are equal in value, but they may not be equally convenient when a later calculation involves a matrix perturbation. Trace identities provide a disciplined way to change the arrangement of factors without changing the scalar being represented.

The central rule is that factors inside a trace can be moved cyclically, provided their dimensions are compatible. Transposition supplies a second tool: it preserves a scalar trace while reversing the order of factors and transposing each one. These rules are useful only when dimensions are checked; matrix multiplication itself is not commutative.

Definition: For a square matrix \(A\in\mathbb{R}^{m\times m}\), its trace is the sum of its diagonal entries, \(\operatorname{tr}(A)=\sum_{i=1}^{m}A_{ii}\). A scalar \(c\) can be viewed as the trace of the \(1\times1\) matrix \([c]\), so \(c=\operatorname{tr}(c)\).

Cyclic Permutation Under the Trace

For two matrices, the trace of a product is unchanged if the factors are swapped, even when the two products have different sizes. This is an important distinction: the matrices \(AB\) and \(BA\) need not be equal, but their traces are equal when both products are defined.

Theorem: If \(A\in\mathbb{R}^{m\times n}\) and \(B\in\mathbb{R}^{n\times m}\), then \(\operatorname{tr}(AB)=\operatorname{tr}(BA)\). In particular, if \(A_1,\ldots,A_k\) have compatible dimensions and their product is square, the trace is unchanged by a cyclic rotation of the factors.

Proof. The \(i\)th diagonal entry of \(AB\) is \((AB)_{ii}=\sum_{j=1}^{n}A_{ij}B_{ji}\). Therefore

\operatorname{tr}(AB) =\sum_{i=1}^{m}\sum_{j=1}^{n}A_{ij}B_{ji} =\sum_{j=1}^{n}\sum_{i=1}^{m}B_{ji}A_{ij} =\operatorname{tr}(BA).

For a longer product, group all but one factor into a single matrix and apply the two-factor identity. For example, if \(A\in\mathbb{R}^{m\times n}\), \(B\in\mathbb{R}^{n\times p}\), and \(C\in\mathbb{R}^{p\times m}\), then \(\operatorname{tr}(ABC)=\operatorname{tr}((AB)C)=\operatorname{tr}(C(AB))=\operatorname{tr}(CAB)\). Repeating this argument gives any cyclic rotation that preserves the order of the factors. It does not justify an arbitrary reordering. \(\square\)

A practical use is moving a factor away from a location where it makes a later expression awkward. For instance, when \(X\in\mathbb{R}^{m\times n}\), \(A\in\mathbb{R}^{n\times m}\), and \(dX\) is an \(m\times n\) perturbation, both \(\operatorname{tr}(A\,dX)\) and \(\operatorname{tr}(dX\,A)\) are defined and equal. The trace changes the arrangement, not the order of factors relative to one another.

Worked Example: Converting a Bilinear Form to a Trace

Let \(u=(2,-1)^T\), \(v=(1,2,-1)^T\), and \(A=\begin{pmatrix}1&2&0\\-1&3&4\end{pmatrix}\). Rewrite \(u^TAv\) as a trace and verify its value.

First compute \(Av=(5,1)^T\), so \(u^TAv=2(5)-1(1)=9\). Since this is a scalar, it equals its own trace. Cyclic permutation gives a form involving only a product of two matrices:

u^TAv =\operatorname{tr}(u^TAv) =\operatorname{tr}(Avu^T) =\operatorname{tr}(A vu^T).

Here \(vu^T\) is \(3\times2\), so \(A vu^T\) is \(2\times2\), as required for its trace. In detail, \(vu^T=\begin{pmatrix}2&-1\\4&-2\\-2&1\end{pmatrix}\), and \(A vu^T=\begin{pmatrix}10&-5\\2&-1\end{pmatrix}\). Its trace is \(10+(-1)=9\), agreeing with the direct calculation.

Transpose Identities

Transposition preserves the diagonal entries of a square matrix, so it preserves the trace. For products, the transpose reverses factor order. Together, these facts yield a second family of equivalent trace expressions.

Theorem: For every square matrix \(M\), \(\operatorname{tr}(M^T)=\operatorname{tr}(M)\). For any conformable product \(A_1A_2\cdots A_k\) that is square, \(\operatorname{tr}(A_1A_2\cdots A_k)=\operatorname{tr}(A_k^T\cdots A_2^TA_1^T)\).

Proof. The diagonal entries of \(M^T\) satisfy \((M^T)_{ii}=M_{ii}\), hence \(\operatorname{tr}(M^T)=\sum_i(M^T)_{ii}=\sum_iM_{ii}=\operatorname{tr}(M)\). The transpose rule for a product follows by repeated use of \((AB)^T=B^TA^T\). Consequently, \((A_1\cdots A_k)^T=A_k^T\cdots A_1^T\). Applying trace invariance under transpose gives the stated identity. \(\square\)

In particular, a real scalar expression equals its transpose. If \(x\) and \(y\) are vectors and \(A\) is a matrix of compatible dimensions, this gives \(x^TAy=(x^TAy)^T=y^TA^Tx\). This identity does not claim \(x^TAy=y^TAx\) for an arbitrary \(A\); the transposed expression contains \(A^T\).

Worked Example: Reversing a Scalar Product

Let \(x=(1,2)^T\), \(y=(3,-1)^T\), and \(A=\begin{pmatrix}2&1\\-3&4\end{pmatrix}\). Find \(x^TAy\), then express the same scalar using a transposed product.

Compute \(Ay=(5,-13)^T\), giving \(x^TAy=1(5)+2(-13)=-21\). By transposing the scalar,

$$ x^TAy=(x^TAy)^T=y^TA^Tx. $$

For verification, \(A^Tx=(-4,9)^T\), so \(y^TA^Tx=3(-4)+(-1)(9)=-21\). The two expressions agree; changing \(A^T\) to \(A\) in the reversed expression would not follow from the transpose identity.

Quadratic Forms and Frobenius-Style Trace Forms

A quadratic form is a scalar of the form \(x^TAx\), where \(A\) is square and \(x\) is a vector of matching length. By treating the scalar as a trace and rotating the factors, one obtains a trace involving the outer product \(xx^T\). This is often a useful representation when the matrix is the object that will vary.

Identity: For \(x\in\mathbb{R}^m\) and \(A\in\mathbb{R}^{m\times m}\), \(x^TAx=\operatorname{tr}(Axx^T)=\operatorname{tr}(xx^TA)\). More generally, if \(U,V\in\mathbb{R}^{m\times n}\), then \(\operatorname{tr}(U^TV)=\sum_{i=1}^{m}\sum_{j=1}^{n}U_{ij}V_{ij}\), the Frobenius inner product of \(U\) and \(V\).

For the quadratic identity, \(x^TAx\) is \(1\times1\), so it equals its trace. Cyclically rotating the factors gives \(\operatorname{tr}(Axx^T)\); another rotation gives \(\operatorname{tr}(xx^TA)\). For the inner-product formula, the \(j\)th diagonal entry of \(U^TV\) is \(\sum_{i=1}^{m}U_{ij}V_{ij}\). Summing over \(j\) yields the stated double sum.

Worked Example: Rewriting a Squared Residual

Let \(A=\begin{pmatrix}1&2\\0&1\end{pmatrix}\), \(x=(1,-1)^T\), and \(b=(0,2)^T\). Write the squared residual as traces in two different orders and check the result.

First, \(Ax=(-1,-1)^T\), so the residual \(r=Ax-b=(-1,-3)^T\). Its squared Euclidean length is \(r^Tr=(-1)^2+(-3)^2=10\). Since it is a scalar, it is also \(\operatorname{tr}(r^Tr)\). Cyclicity gives

(Ax-b)^T(Ax-b) =\operatorname{tr}\big((Ax-b)^T(Ax-b)\big) =\operatorname{tr}\big((Ax-b)(Ax-b)^T\big).

Indeed, \(r^Tr=(10)\), while \(rr^T=\begin{pmatrix}1&3\\3&9\end{pmatrix}\), whose trace is also \(1+9=10\). The two matrices differ, but their traces agree.

Dimension Checks and Common Mistakes

Trace manipulations are reliable only when every product is defined and the final matrix inside the trace is square. Check dimensions before applying an identity, especially when the factors are rectangular. The equality \(\operatorname{tr}(AB)=\operatorname{tr}(BA)\) specifically requires \(A\) to be \(m\times n\) and \(B\) to be \(n\times m\); it does not mean that arbitrary factors can be swapped within a longer product.

Wrong reasoning: “The trace lets me reorder the factors however I want.” Correction: It permits cyclic rotations, such as \(ABC\) to \(BCA\), not a swap that changes the order, such as \(ABC\) to \(ACB\).
Wrong reasoning: “Because \(x^TAy\) is a scalar, reversing the vectors gives \(y^TAx\).” Correction: Transposing gives \(y^TA^Tx\); the matrix must also be transposed.
Wrong reasoning: “If two products have the same factors, their traces are automatically comparable.” Correction: First verify that the products are defined and square. Some cyclic rotations change the matrix dimensions, even though their traces remain equal.

These identities matter because many scalar objectives can be written in several equivalent forms. A form with a matrix appearing in a particular position may make a later differential calculation more direct. The identities do not themselves differentiate the expression; they ensure that any rearrangement made before differentiation preserves the scalar function.

Key takeaway: Use cyclicity to rotate, not arbitrarily reorder, matrix factors under a trace. Use transpose invariance to reverse factor order while transposing every factor. In either case, check dimensions and verify that the trace is applied to a square matrix.

Check Your Understanding

For each question, justify the rearrangement using dimensions and the relevant identity.

If \(A\) is \(2\times3\) and \(B\) is \(3\times2\), which of \(AB\) and \(BA\) is square, and why do their traces agree?
Rewrite \(u^TCv\), for conformable vectors \(u,v\) and matrix \(C\), as a trace involving \(C\) and an outer product.
What is the correct transpose of the scalar \(x^TAy\)? Explain why \(y^TAx\) is not generally the same expression.
State a valid cyclic rotation of \(\operatorname{tr}(ABC)\), and give one factor reordering that cyclicity does not justify.
Why can \(\operatorname{tr}(rr^T)\) equal \(r^Tr\) even though \(rr^T\) and \(r^Tr\) are different matrices?