Matrix Calculus Dimension and Symmetry Pitfalls

Why Matrix Derivatives Need an Audit

A matrix derivative can look plausible and still be wrong. A missing transpose may give a gradient with the wrong shape; swapping two factors may silently change the function; and differentiating with respect to a symmetric matrix as if all its entries varied independently can give an incorrect answer. These are not merely notational slips: each changes what the derivative means.

A reliable calculation has three checks. First, record the dimensions of every matrix. Second, use the differential to track factor order and transposes. Third, identify the space in which the parameter varies. The last check matters especially when the parameter is constrained to be symmetric.

Definition: For a scalar-valued function $f$ of a matrix $X\in\mathbb{R}^{m\times n}$, its Frobenius gradient $\nabla_X f\in\mathbb{R}^{m\times n}$ is defined by the first-order differential $$ df=\operatorname{tr}\big((\nabla_X f)^T\,dX\big). $$ Here $dX$ is an arbitrary perturbation of the same shape as $X$. The gradient is the matrix representing the differential under the Frobenius inner product $\langle U,V\rangle_F=\operatorname{tr}(U^T V)$.

The gradient has the same dimensions as the variable. This is a useful first check, but it is not enough: two different matrices can have the same shape, and only the differential identity determines the correct one. Throughout, differentials are first-order linear maps, and the gradient convention is the Frobenius-inner-product convention.

Check Dimensions Before Manipulating Factors

Matrix multiplication is defined only when the inner dimensions match. For example, if $X\in\mathbb{R}^{m\times n}$, $A\in\mathbb{R}^{r\times m}$, and $B\in\mathbb{R}^{n\times r}$, then $AXB\in\mathbb{R}^{r\times r}$, so $\operatorname{tr}(AXB)$ is a scalar. Its differential with respect to $X$ must therefore be represented by an $m\times n$ gradient.

Worked Example: Auditing a Trace Expression by Shape

Let $X$ be $2\times3$, $A$ be $4\times2$, and $B$ be $3\times4$. Determine whether $f(X)=\operatorname{tr}(AXB)$ is defined, and find the dimensions required for its gradient.

Check the products. The product $AX$ has shape $(4\times2)(2\times3)=4\times3$. Then $AXB$ has shape $(4\times3)(3\times4)=4\times4$. Its trace is defined, so $f$ is scalar-valued.

Check the gradient shape. Since the variable $X$ is $2\times3$, its Frobenius gradient must also be $2\times3$. An answer of shape $3\times2$, even if obtained from a familiar-looking formula, cannot be the gradient under this convention.

Differentiate to verify the shape. The differential is $df=\operatorname{tr}(A\,dX\,B)$. The product $A\,dX\,B$ is $4\times4$, so the trace is valid. Cyclically moving factors inside the trace gives $df=\operatorname{tr}(BA\,dX)$, where $BA$ is $3\times2$. Thus the matrix paired on the left of $dX$ is $3\times2$, its transpose is $2\times3$, and the gradient has the required shape.

A common dimension error is to transpose the final answer without first checking which matrix is paired with $dX$. The defining equation uses $\operatorname{tr}(G^T dX)$, not $\operatorname{tr}(G\,dX)$, where $G=\nabla_X f$. That transpose is part of the convention, not an optional adjustment.

Factor Order and Transposes Are Not Interchangeable

Matrix multiplication is generally noncommutative: even when both products exist, $AB$ need not equal $BA$. Trace identities permit cyclic rearrangement of a product inside a trace when the products are dimensionally valid; they do not permit arbitrary reordering. A dependable procedure is to write the differential first, then rearrange only as needed to match the gradient definition.

Result: If $X\in\mathbb{R}^{m\times n}$, $a\in\mathbb{R}^{m}$, and $b\in\mathbb{R}^{n}$, then for $f(X)=a^T Xb$, $$ \nabla_X f=ab^T. $$

Derivation. For an arbitrary perturbation $dX$, $df=a^T(dX)b$. The scalar can be written as a trace, and then in the gradient convention:

df=a^T(dX)b =\operatorname{tr}(b\,a^T dX) =\operatorname{tr}\big((ab^T)^T dX\big).

Since the differential is represented by the Frobenius inner product with $ab^T$, the gradient is $ab^T$. Its shape is $m\times n$, as required. $\square$

Worked Example: Finding the Gradient Without Guessing a Transpose

Take $a=(1,2)^T$, $b=(3,-1,4)^T$, and $X\in\mathbb{R}^{2\times3}$. Find the gradient of $f(X)=a^T Xb$.

The dimensions are compatible: $a^T$ is $1\times2$, $X$ is $2\times3$, and $b$ is $3\times1$, so $f$ is scalar. The result above gives

\nabla_X f=ab^T = \begin{pmatrix}1\\2\end{pmatrix} \begin{pmatrix}3&-1&4\end{pmatrix} = \begin{pmatrix} 3&-1&4\\ 6&-2&8 \end{pmatrix}.

This is $2\times3$, matching $X$. To check the transpose and order, let $dX=(h_{ij})$. Directly, $df=a^T(dX)b=3h_{11}-h_{12}+4h_{13}+6h_{21}-2h_{22}+8h_{23}$. The Frobenius product of the displayed gradient with $dX$ gives exactly the same sum of entries. A transposed $3\times2$ answer would fail the shape check and could not be paired with $dX$ in the defining equation.

For the trace expression from the dimension audit, the same logic gives a practical formula. The differential is $df=\operatorname{tr}(A\,dX\,B)$. Cycling factors within the trace yields $\operatorname{tr}(BA\,dX)$; matching this with $\operatorname{tr}(G^T dX)$ identifies $G^T=BA$, hence $G=A^T B^T$. The order $A^T B^T$ follows from transposing the whole product $BA$; it is not interchangeable with $B^T A^T$.

Symmetric Parameters Have Fewer Directions

If the variable is constrained to be symmetric, its perturbations are symmetric too. Consequently, the differential does not distinguish every possible ambient gradient: only the part that pairs with symmetric perturbations is identifiable. The gradient on the symmetric-matrix space is conventionally represented by a symmetric matrix.

Result: Suppose $S=S^T$ varies over the vector space of symmetric matrices and a differential has the form $df=\operatorname{tr}(G^T dS)$, where $G$ is any square matrix. The unique symmetric matrix representing this same differential for all symmetric $dS$ is $$ \operatorname{sym}(G)=\frac{G+G^T}{2}. $$

Proof. Write $G=\operatorname{sym}(G)+K$, where $K=(G-G^T)/2$ is skew-symmetric. For symmetric $dS$, the skew-symmetric part contributes zero. Indeed, the scalar $q=\operatorname{tr}(K^T dS)$ satisfies $q=\operatorname{tr}((K^T dS)^T)=\operatorname{tr}(dS^T K)=\operatorname{tr}(dS K)$. But $K^T=-K$, and the Frobenius pairing of a skew-symmetric matrix with a symmetric one is zero: equivalently, $\operatorname{tr}(K^T dS)=-\operatorname{tr}(K dS)$, while cyclicity gives $\operatorname{tr}(K dS)=\operatorname{tr}(dS K)$, so $q=-q$ and $q=0$. Therefore

\operatorname{tr}(G^T dS) =\operatorname{tr}\big(\operatorname{sym}(G)^T dS\big).

For uniqueness among symmetric representatives, suppose symmetric $R$ also gives the same differential. Then $\operatorname{tr}((R-\operatorname{sym}(G))^T dS)=0$ for every symmetric $dS$. Choose $dS=R-\operatorname{sym}(G)$, which is symmetric. The resulting squared Frobenius norm is zero, so $R=\operatorname{sym}(G)$. $\square$

Worked Example: Differentiating with Respect to a Symmetric Matrix

Let $S=\begin{pmatrix}s_{11}&s_{12}\\s_{12}&s_{22}\end{pmatrix}$ be symmetric, and let $C=\begin{pmatrix}1&2\\4&3\end{pmatrix}$. Find the gradient of $f(S)=\operatorname{tr}(C^T S)$ when $S$ is restricted to symmetric matrices.

Expanding the trace using the entries gives $f(S)=s_{11}+2s_{12}+4s_{12}+3s_{22}=s_{11}+6s_{12}+3s_{22}$. Thus the derivatives with respect to the three independent coordinates $(s_{11},s_{12},s_{22})$ are $(1,6,3)$.

If $S$ were instead treated as an unrestricted $2\times2$ matrix, the ambient gradient would be $C$, since $df=\operatorname{tr}(C^T dS)$. But for symmetric perturbations the gradient must be symmetric. Its correct representative is

\operatorname{sym}(C) =\frac12 \begin{pmatrix}1&2\\4&3\end{pmatrix} +\frac12 \begin{pmatrix}1&4\\2&3\end{pmatrix} = \begin{pmatrix}1&3\\3&3\end{pmatrix}.

To check, write $dS=\begin{pmatrix}h_{11}&h_{12}\\h_{12}&h_{22}\end{pmatrix}$. Then $\operatorname{tr}(\operatorname{sym}(C)^T dS)=h_{11}+3h_{12}+3h_{12}+3h_{22}=h_{11}+6h_{12}+3h_{22}$, matching the direct coordinate derivatives. The off-diagonal derivative is split equally across the two matching matrix entries in the symmetric Frobenius gradient.

Common Mistakes and Corrections

Many errors arise from applying a familiar rule without checking the object it acts on. The following comparisons are useful audits, not substitutes for deriving the differential.

Incorrect reasoning	Correction
“The factors inside a trace can be reordered however I like.”	Only cyclic rearrangements preserve a trace, and only when dimensions make the products valid. Reversing factor order is generally invalid.
“The gradient is the matrix appearing beside $dX$.”	Match the differential to $\operatorname{tr}(G^T dX)$. The matrix beside $dX$ in a trace may be $G^T$, not $G$.
“A gradient for symmetric $S$ is just the unrestricted gradient.”	Project the ambient gradient onto symmetric matrices: use $(G+G^T)/2$ for the Frobenius-gradient representative.
“A familiar formula must be correct if it is algebraically plausible.”	Check every product dimension and verify the proposed gradient has the same shape as the variable.

The symmetric-coordinate example also exposes a frequent factor-of-two confusion. The derivative with respect to the single coordinate $s_{12}$ is $6$, because changing $s_{12}$ changes both entries $S_{12}$ and $S_{21}$. The symmetric Frobenius gradient has $3$ in each off-diagonal position; its inner product counts both entries and therefore contributes $3h_{12}+3h_{12}=6h_{12}$. These are two different coordinate conventions, not conflicting answers.

Key takeaway: Start with the dimensions and the differential. Preserve factor order, match the Frobenius-gradient definition including its transpose, and restrict the gradient to the correct parameter space. When the parameter is symmetric, use the symmetric representative of the differential.

Check Your Understanding

For each question, justify your answer with a shape check or a differential identity.

If $X$ is $3\times2$, what shape must its Frobenius gradient have?
For $f(X)=a^T Xb$, where $a$ is $m\times1$ and $b$ is $n\times1$, which matrix is the gradient, and why is its shape $m\times n$?
Why can a trace be cyclically rearranged but not arbitrarily reordered?
If an ambient gradient is $G$ and the parameter varies only over symmetric matrices, what symmetric matrix represents the restricted differential?
In the symmetric $2\times2$ example, why is the derivative with respect to $s_{12}$ twice the off-diagonal entry of the Frobenius gradient?