Why Quadratic Gradients Use the Symmetric Part

Why the Symmetric Part Appears

Quadratic expressions appear throughout multivariate statistics and optimization. Their matrix notation is compact, but it can hide an important detail: the matrix in \(x^T A x\) need not be symmetric. The value of the expression, however, is determined entirely by a symmetric matrix. That fact explains why the gradient contains \(A+A^T\), rather than just \(Ax\), and why symmetry assumptions should be stated rather than silently imposed.

Throughout, \(x\) is a vector in \(\mathbb{R}^n\), \(A\) is a fixed real \(n\times n\) matrix, and \(q:\mathbb{R}^n\to\mathbb{R}\) is defined by \(q(x)=x^T A x\). We use the column-gradient convention: for a differentiable scalar function \(f\), its differential is \(df=(\nabla f)^T dx\). For a vector-valued gradient, the Hessian \(H(x)\) is the matrix satisfying \(d(\nabla f)=H(x)\,dx\), when this derivative exists.

Definition: The symmetric part of a square matrix \(A\) is \(S=(A+A^T)/2\). Its skew-symmetric part is \(K=(A-A^T)/2\). Thus \(A=S+K\), \(S^T=S\), and \(K^T=-K\).

A Quadratic Form Only Sees the Symmetric Part

The decomposition \(A=S+K\) separates the part that affects a quadratic form from the part that does not. The key fact is that a scalar equals its transpose. Applying this to \(x^T Kx\) gives a direct proof.

Theorem: For every real square matrix \(A\) and every \(x\in\mathbb{R}^n\), \(x^T A x=x^T\bigl((A+A^T)/2\bigr)x\). Equivalently, \(x^T Kx=0\) for every skew-symmetric matrix \(K\).

Proof. Since \(x^T Kx\) is a scalar, it is equal to its transpose. Therefore

$$ x^T Kx=(x^T Kx)^T=x^T K^T x=-x^T Kx. $$

It follows that \(2x^T Kx=0\), and hence \(x^T Kx=0\). Now write \(A=S+K\). Then

x^T A x=x^T Sx+x^T Kx=x^T Sx =x^T\left(\frac{A+A^T}{2}\right)x.

This proves both claims. \(\square\)

The conclusion is about the value of the scalar function for every \(x\); it does not say that \(A\) itself is symmetric. Different matrices can therefore represent exactly the same quadratic form if they have the same symmetric part.

Worked Example: Rewriting a Nonsymmetric Quadratic Form

Let

A=\begin{pmatrix}2&3\\-1&4\end{pmatrix}, \qquad x=\begin{pmatrix}x_1\\x_2\end{pmatrix}.

The symmetric and skew-symmetric parts are

S=\frac{A+A^T}{2} =\begin{pmatrix}2&1\\1&4\end{pmatrix}, \qquad K=\frac{A-A^T}{2} =\begin{pmatrix}0&2\\-2&0\end{pmatrix}.

Direct multiplication gives

x^T A x =\begin{pmatrix}x_1&x_2\end{pmatrix} \begin{pmatrix}2x_1+3x_2\\-x_1+4x_2\end{pmatrix} =2x_1^2+2x_1x_2+4x_2^2.

Using the symmetric part gives the same result:

x^T Sx =\begin{pmatrix}x_1&x_2\end{pmatrix} \begin{pmatrix}2x_1+x_2\\x_1+4x_2\end{pmatrix} =2x_1^2+2x_1x_2+4x_2^2.

The off-diagonal entries of \(A\) are not equal, but their sum determines the cross-term coefficient. The skew-symmetric part contributes nothing to the quadratic form.

Deriving the Gradient and Hessian

The symmetric-part identity gives one route to the derivatives: differentiate \(x^T Sx\), where \(S=(A+A^T)/2\). A differential calculation also shows exactly where both \(A\) and \(A^T\) enter.

Theorem: Let \(A\) be a fixed real \(n\times n\) matrix and let \(q(x)=x^T A x\). Then \(q\) is twice differentiable on \(\mathbb{R}^n\), with

\nabla q(x)=(A+A^T)x, \qquad H_q(x)=A+A^T.

In particular, if \(A\) is symmetric, then \(\nabla q(x)=2Ax\) and \(H_q(x)=2A\).

Derivation. For an increment \(dx\), apply the product rule to \(q=x^T A x\). Since \(A\) is fixed,

dq=(dx)^T A x+x^T A\,dx.

The first term can be written in the column-gradient convention as \((dx)^T Ax=(Ax)^T dx\). The second is \(x^T A\,dx=(A^T x)^T dx\). Consequently,

dq=\bigl((A+A^T)x\bigr)^T dx.

By the definition of the gradient, \(\nabla q(x)=(A+A^T)x\). The matrix \(A+A^T\) is constant in \(x\), so differentiating the gradient gives

d(\nabla q)=(A+A^T)\,dx.

By the definition of the Hessian, \(H_q(x)=A+A^T\). If \(A=A^T\), these formulas reduce to \(2Ax\) and \(2A\), respectively. \(\square\)

The Hessian is constant because the gradient is a linear function of \(x\). The factor of two in the symmetric case comes from the two product-rule terms: when \(A=A^T\), each contributes the same linear expression.

Worked Example: Evaluating the Derivatives at a Point

Use the matrix from the first example and evaluate the function and its derivatives at \(x=(1,-1)^T\). Its symmetric part gives the quadratic form \(q(x)=2x_1^2+2x_1x_2+4x_2^2\). Substitution yields

$$ q(1,-1)=2(1)^2+2(1)(-1)+4(-1)^2=2-2+4=4. $$

The gradient matrix is

A+A^T= \begin{pmatrix}4&2\\2&8\end{pmatrix}, \qquad \nabla q(1,-1)= \begin{pmatrix}4&2\\2&8\end{pmatrix} \begin{pmatrix}1\\-1\end{pmatrix} = \begin{pmatrix}2\\-6\end{pmatrix}.

For a direct check, differentiating the expanded polynomial with respect to \(x_1\) gives \(4x_1+2x_2\), which is \(2\) at the chosen point. Differentiating with respect to \(x_2\) gives \(2x_1+8x_2\), which is \(-6\). Differentiating these two gradient components gives

H_q= \begin{pmatrix}4&2\\2&8\end{pmatrix}=A+A^T.

The Hessian is the same at every point, not only at \((1,-1)^T\).

Worked Example: The Symmetric-Matrix Special Case

Let

A=\begin{pmatrix}3&-2\\-2&5\end{pmatrix}, \qquad x=\begin{pmatrix}1\\2\end{pmatrix}.

Here \(A^T=A\). First compute \(Ax=(-1,8)^T\), so

q(x)=x^T Ax =\begin{pmatrix}1&2\end{pmatrix} \begin{pmatrix}-1\\8\end{pmatrix} =-1+16=15.

The symmetric-case formulas give

\nabla q(x)=2Ax =2\begin{pmatrix}-1\\8\end{pmatrix} =\begin{pmatrix}-2\\16\end{pmatrix}, \qquad H_q=2A= \begin{pmatrix}6&-4\\-4&10\end{pmatrix}.

The factor of two can also be checked by expanding: \(q(x)=3x_1^2-4x_1x_2+5x_2^2\). Its first partial derivatives are \(6x_1-4x_2\) and \(-4x_1+10x_2\), which at \((1,2)^T\) equal \(-2\) and \(16\). Their derivatives form the displayed Hessian.

Equivalent Matrices, Identical Derivatives

The symmetric-part identity has a derivative consequence: if two matrices have the same symmetric part, they generate not only the same quadratic-form values but also the same gradient and Hessian. To see this, if \(A=S+K\) with \(K^T=-K\), then \(A+A^T=2S\). Thus both the function and its derivatives depend only on \(S\).

Worked Example: Adding a Skew-Symmetric Matrix Changes Nothing

Take

A=\begin{pmatrix}1&2\\0&3\end{pmatrix}, \qquad K=\begin{pmatrix}0&4\\-4&0\end{pmatrix}, \qquad B=A+K=\begin{pmatrix}1&6\\-4&3\end{pmatrix}.

The symmetric part of each matrix is

\frac{A+A^T}{2} =\frac{B+B^T}{2} =\begin{pmatrix}1&1\\1&3\end{pmatrix}.

At \(x=(2,-1)^T\), multiplying by \(A\) gives \(Ax=(0,-3)^T\), and therefore \(x^T Ax=2(0)+(-1)(-3)=3\). Multiplying by \(B\) gives \(Bx=(-4,-11)^T\), so \(x^T Bx=2(-4)+(-1)(-11)=3\) as well.

For either matrix, the gradient is twice the common symmetric part times \(x\):

\nabla q(x) =2\begin{pmatrix}1&1\\1&3\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} =2\begin{pmatrix}1\\-1\end{pmatrix} =\begin{pmatrix}2\\-2\end{pmatrix}.

The Hessian for both representations is \(2S=\begin{pmatrix}2&2\\2&6\end{pmatrix}\). Adding the skew-symmetric matrix changed neither the function nor its derivatives.

Common Mistake and Why the Distinction Matters

A common mistake is to differentiate \(x^T A x\) as though it were \(Ax\), or to write the gradient as \(2Ax\) without checking symmetry. The latter expression is valid when \(A=A^T\); for a general matrix, the correct gradient is \((A+A^T)x\). In the first example, at \(x=(1,-1)^T\), \(Ax=( -1,-5)^T\), so \(2Ax=(-2,-10)^T\), which does not match the correct gradient \((2,-6)^T\).

The correction is to apply the product rule to both appearances of \(x\), then express each term as a coefficient vector transposed times \(dx\). This yields one contribution from \(A\) and one from \(A^T\). Equivalently, replace \(A\) by its symmetric part \(S\) first, then use \(\nabla q(x)=2Sx\).

This distinction matters whenever a quadratic expression is written using a matrix that was not explicitly assumed symmetric. The scalar function remains well defined, but its derivatives must reflect the symmetric part actually governing that function. Checking the dimensions is also useful: the gradient is an \(n\)-vector, while the Hessian is an \(n\times n\) matrix.

Key takeaway: For \(q(x)=x^T A x\), only \(S=(A+A^T)/2\) affects the function. Therefore \(\nabla q(x)=2Sx=(A+A^T)x\) and \(H_q=2S=A+A^T\). If \(A\) is symmetric, these simplify to \(2Ax\) and \(2A\).

Check Your Understanding

Use the definitions and derivations above to answer the following questions.

Why must \(x^T Kx=0\) when \(K\) is skew-symmetric?
For a general fixed matrix \(A\), state the gradient and Hessian of \(x^T A x\).
What additional condition allows the gradient to be written as \(2Ax\)?
If \(B=A+K\) and \(K\) is skew-symmetric, what can be said about the quadratic forms and derivatives associated with \(A\) and \(B\)?
For \(A=\begin{pmatrix}1&3\\-1&2\end{pmatrix}\), calculate the symmetric part and state the Hessian of \(x^T A x\).