Differentiating Matrix Inverses

Why Differentiating an Inverse Requires Care

Inverse matrices appear whenever a model uses a linear system to express a solution, a weighted quadratic form, or a transformation back from coordinates. When the matrix changes, its inverse changes too—but not by applying the scalar rule $d(1/x)=-x^{-2}dx$ entry by entry. Matrix multiplication is ordered, and the correct differential must preserve that order.

The central rule is compact: for an invertible square matrix $X$, the change in $X^{-1}$ is a linear function of the change in $X$, with a copy of $X^{-1}$ on each side of that change. Once this rule is established, scalar objectives containing $X^{-1}$ can be differentiated using the product and trace rules.

Definition: Let $X$ be an invertible $n\times n$ matrix. Its inverse differential is the linear map taking a perturbation $H\in\mathbb{R}^{n\times n}$ in $X$ to the first-order change in $X^{-1}$. In differential notation, this map is $$ d(X^{-1})=-X^{-1}(dX)X^{-1}. $$

Deriving the Inverse Differential

The formula follows from the identity $X X^{-1}=I$, where $I$ is the $n\times n$ identity matrix. The differential of a product gives $dX\,X^{-1}+X\,d(X^{-1})=0$. Multiplying on the left by $X^{-1}$ isolates the inverse differential. The argument below also verifies that the result is a genuine first-order approximation, not just a formal manipulation.

Theorem: The set of invertible $n\times n$ matrices is an open domain. At any $X$ in this domain, the map $X\mapsto X^{-1}$ is differentiable, and its derivative in direction $H$ is $$ D(X^{-1})[H]=-X^{-1}HX^{-1}. $$ Equivalently, $d(X^{-1})=-X^{-1}(dX)X^{-1}$.

Ensure nearby matrices remain invertible.
If $\lVert X^{-1}H\rVert<1$, then $X+H=X(I+X^{-1}H)$ is invertible: $I+X^{-1}H$ has an inverse given by its convergent geometric series. Thus sufficiently small perturbations stay in the domain.

Use the defining identity.
Direct multiplication verifies $$ (X+H)^{-1}-X^{-1}=-(X+H)^{-1}HX^{-1}. $$

Separate the linear part and bound the remainder.
The proposed linear part is $-X^{-1}HX^{-1}$. Subtracting it from the change in the inverse gives $$ (X+H)^{-1}-X^{-1}+X^{-1}HX^{-1} =(X+H)^{-1}HX^{-1}HX^{-1}. $$ For $H$ sufficiently small, $\lVert(X+H)^{-1}\rVert$ is bounded. By submultiplicativity of the matrix norm, the right-hand side is therefore bounded in norm by a constant times $\lVert H\rVert^2$. Dividing the remainder by $\lVert H\rVert$ makes it tend to zero as $H\to0$. This proves differentiability with derivative $-X^{-1}HX^{-1}$. $\square$

A useful check is the order of the factors. The perturbation $H$ lies between two copies of $X^{-1}$; in general, those factors cannot be moved across $H$. The formula also has the right dimensions: each factor is $n\times n$, so the result is an $n\times n$ matrix.

From the Differential to a Scalar Gradient

For a scalar function of a matrix, the Frobenius gradient $G$ is defined by $df=\operatorname{tr}(G^T dX)$. This convention turns a differential into a gradient by collecting the coefficient of $dX$ inside a trace. The inverse rule supplies the differential; trace identities then put it into the required form.

Result: For fixed $n\times n$ matrices $A$ and $B$, let $f(X)=\operatorname{tr}(A X^{-1}B)$ on invertible $X$. Then $$ df=-\operatorname{tr}(A X^{-1}(dX)X^{-1}B), \qquad \nabla_X f=-X^{-T}A^TB^TX^{-T}. $$

To obtain the gradient, apply the inverse differential inside the trace and cycle factors so that $dX$ is last:

\begin{aligned} df &=\operatorname{tr}\!\left(A\,d(X^{-1})\,B\right)\\ &=-\operatorname{tr}\!\left(A X^{-1}(dX)X^{-1}B\right)\\ &=-\operatorname{tr}\!\left(X^{-1}BA X^{-1}(dX)\right). \end{aligned}

For any matrix $M$, $\operatorname{tr}(M\,dX)=\operatorname{tr}((M^T)^T dX)$, so its Frobenius-gradient coefficient is $M^T$. Here $M=-X^{-1}BA X^{-1}$, and transposing gives the stated gradient. This derivation depends on cyclically rotating factors inside a trace, not on commuting them.

Worked Example: A Bilinear Objective

Let $X$ be invertible and define $f(X)=a^T X^{-1}b$, where $a$ and $b$ are fixed $n$-vectors. Find the differential and evaluate the gradient at $X=\begin{pmatrix}2&0\\0&3\end{pmatrix}$, $a=\begin{pmatrix}1\\2\end{pmatrix}$, and $b=\begin{pmatrix}3\\-1\end{pmatrix}$.

First rewrite the scalar as $f(X)=\operatorname{tr}(a^T X^{-1}b)=\operatorname{tr}(b a^T X^{-1})$. Equivalently, differentiate the scalar directly:

df=-a^T X^{-1}(dX)X^{-1}b.

The scalar can be written as a trace with $dX$ at the end:

df=-\operatorname{tr}\!\left(X^{-1}b a^T X^{-1}(dX)\right), \qquad \nabla_X f=-X^{-T}a b^T X^{-T}.

At the specified diagonal matrix, $X^{-1}=X^{-T}=\begin{pmatrix}1/2&0\\0&1/3\end{pmatrix}$. Thus $X^{-1}a=\begin{pmatrix}1/2\\2/3\end{pmatrix}$ and $X^{-1}b=\begin{pmatrix}3/2\\-1/3\end{pmatrix}$. The gradient is their negative outer product:

\nabla_X f =-\begin{pmatrix}1/2\\2/3\end{pmatrix} \begin{pmatrix}3/2&-1/3\end{pmatrix} =\begin{pmatrix}-3/4&1/6\\-1&2/9\end{pmatrix}.

As a check on the scalar value, $X^{-1}b=(3/2,-1/3)^T$, so $f=a^T X^{-1}b=3/2-2/3=5/6$. The gradient gives the directional change for any perturbation $H$ through $df=\operatorname{tr}((\nabla_X f)^T H)$.

Worked Applications

The same rule applies whether an inverse appears between vectors or inside a trace. The following examples make explicit how to handle the factor order and, in the quadratic example, how the domain of the matrix affects the interpretation of the gradient.

Worked Example: A Trace Objective

Let $f(X)=\operatorname{tr}(C X^{-1})$, with $C=\begin{pmatrix}1&2\\0&3\end{pmatrix}$, and evaluate its value and gradient at $X=\begin{pmatrix}2&0\\0&1\end{pmatrix}$.

The inverse is $X^{-1}=\operatorname{diag}(1/2,1)$. The product $CX^{-1}$ is $\begin{pmatrix}1/2&2\\0&3\end{pmatrix}$, so $f(X)=1/2+3=7/2$. For the differential,

df=-\operatorname{tr}(C X^{-1}(dX)X^{-1}) =-\operatorname{tr}(X^{-1}C X^{-1}(dX)).

Therefore $\nabla_X f=-(X^{-1}C X^{-1})^T=-X^{-T}C^T X^{-T}$. In this example,

X^{-1}C X^{-1} =\begin{pmatrix}1/2&0\\0&1\end{pmatrix} \begin{pmatrix}1&2\\0&3\end{pmatrix} \begin{pmatrix}1/2&0\\0&1\end{pmatrix} =\begin{pmatrix}1/4&1\\0&3\end{pmatrix}, $$ $$ \nabla_X f=-\begin{pmatrix}1/4&0\\1&3\end{pmatrix}.

For a direct check, take a perturbation with only its $(1,2)$ entry nonzero. The differential is 0, matching the gradient's $(1,2)$ entry. A perturbation with only its $(2,1)$ entry nonzero gives coefficient $-1$, matching the gradient's $(2,1)$ entry.

Worked Example: A Quadratic Form on Symmetric Matrices

Consider $q(X)=x^T X^{-1}x$, where $X$ is symmetric positive definite and $x=\begin{pmatrix}1\\2\end{pmatrix}$. Evaluate the value and gradient at $X=\begin{pmatrix}2&0\\0&1\end{pmatrix}$, and find the first-order change in the direction $H=\begin{pmatrix}1&1/5\\1/5&-1/2\end{pmatrix}$.

The value is $q(X)=x^T\begin{pmatrix}1/2&0\\0&1\end{pmatrix}x=1/2+4=9/2$. Since $X$ and its allowed perturbations are symmetric, $X^{-1}$ is symmetric. Differentiating gives

dq=-x^T X^{-1}(dX)X^{-1}x =-\operatorname{tr}\!\left(X^{-1}xx^T X^{-1}(dX)\right).

Thus the gradient is $-X^{-1}xx^T X^{-1}$. With $z=X^{-1}x=(1/2,2)^T$,

\nabla_X q=-zz^T =-\begin{pmatrix}1/4&1\\1&4\end{pmatrix}.

The directional differential is

dq[H]=\operatorname{tr}((\nabla_X q)^T H) =-\left(\frac14+\frac15+\frac15-2\right) =\frac{27}{20}.

Equivalently, $-z^T H z=27/20$. The two off-diagonal entries of $H$ both contribute to the Frobenius inner product; omitting either would give the wrong directional change.

Common Pitfalls and Why the Rule Matters

Reversing or rearranging factors. In general $X^{-1}H X^{-1}$ cannot be changed to $H X^{-2}$. Such a rearrangement requires a commutation property that need not hold.
Differentiating entries as if the inverse acted entry by entry. The inverse is a matrix function, and a perturbation in one entry can affect several entries of the inverse. Use the full matrix differential.
Forgetting the minus sign. A small scalar increase in a positive number decreases its reciprocal. The minus sign in the matrix rule is consistent with that one-dimensional case.
Ignoring the domain. The differential is defined at an invertible matrix. If the model restricts $X$ to symmetric positive-definite matrices, its allowed perturbations must also be symmetric. A gradient on the unrestricted matrix space and a gradient interpreted only on a restricted domain need not be interchangeable without considering those directions.
Confusing a differential with a gradient. The differential is a scalar linear function of $dX$; the Frobenius gradient is the matrix $G$ representing it as $\operatorname{tr}(G^T dX)$. The transpose arises from this representation.

The inverse differential is useful because it separates a common nonlinear operation from the particular objective being studied. First replace $d(X^{-1})$ by $-X^{-1}(dX)X^{-1}$; then use product and trace rules to collect the terms multiplying $dX$. Keeping the matrix order visible throughout is the most reliable way to obtain a correct scalar derivative.

Key takeaway: For an invertible matrix, $d(X^{-1})=-X^{-1}(dX)X^{-1}$. To find a scalar objective's Frobenius gradient, insert this differential, move factors only by valid trace identities, and match the result to $\operatorname{tr}(G^T dX)$.

Check Your Understanding

For each question, keep the order of the matrix factors explicit.

Starting from $X X^{-1}=I$, derive the differential of $X^{-1}$.
For fixed vectors $a$ and $b$, write the differential of $a^T X^{-1}b$, and state its Frobenius gradient.
For $f(X)=\operatorname{tr}(C X^{-1})$, which trace rearrangement places $dX$ last?
Why is the gradient of $x^T X^{-1}x$ symmetric when $X$ is symmetric positive definite?
What is the error in replacing $X^{-1}H X^{-1}$ by $H X^{-2}$ without further assumptions?