Hessians of Quadratic Objectives

Why Quadratic Hessians Are Especially Simple

Quadratic objectives occur whenever a model combines squared deviations, penalties, or other second-degree terms in a vector \(x\). Their Hessians are particularly useful because they do not depend on \(x\): once the coefficient matrix is identified, the second-order behavior is determined everywhere. The main care needed is keeping track of whether the objective has a factor of \(1/2\), and whether its displayed coefficient matrix is symmetric.

Throughout, \(x\in\mathbb{R}^p\) is a column vector, \(A\in\mathbb{R}^{p\times p}\), \(b\in\mathbb{R}^p\), and \(c\in\mathbb{R}\). The gradient is a column vector, and the Hessian is the matrix of second partial derivatives.

Definition: A scalar quadratic function of \(x\) has the form \(f(x)=c+b^Tx+x^TAx\). Its associated quadratic form is the scalar \(x^TAx\). The Hessian \(\nabla^2 f(x)\) is the \(p\times p\) matrix whose \((i,j)\) entry is \(\partial^2 f(x)/(\partial x_i\partial x_j)\).

The Hessian of a General Quadratic

A quadratic term is sometimes written with a symmetric matrix, but the matrix supplied in a formula need not be symmetric. The function itself only depends on its symmetric part. Define \(A_s=(A+A^T)/2\). The following derivation gives both the gradient and Hessian and makes the role of \(A_s\) explicit.

Result: For \(f(x)=c+b^Tx+x^TAx\), with \(A\) any real \(p\times p\) matrix, \(\nabla f(x)=b+(A+A^T)x\) and \(\nabla^2 f(x)=A+A^T=2A_s\). In particular, the Hessian is constant and symmetric.

To derive the result, take an arbitrary increment \(h\in\mathbb{R}^p\) and expand:

\begin{aligned} f(x+h)-f(x) &=b^Th+(x+h)^TA(x+h)-x^TAx\\ &=b^Th+x^TAh+h^TAx+h^TAh\\ &=\bigl(b+(A+A^T)x\bigr)^Th+h^TAh. \end{aligned}

The first-order term is the gradient transposed times \(h\); the remaining term is quadratic in the increment. Because \(h^TAh=h^TA_sh\), its second differential is \(2h^TA_sh\). Equivalently, differentiating the gradient \(b+(A+A^T)x\) gives the constant Hessian \(A+A^T\). Thus

d f=\bigl(b+(A+A^T)x\bigr)^Tdx,\qquad d^2 f=dx^T(A+A^T)dx.

The matrix in the second differential is the Hessian; the second differential is a scalar, not a matrix. The factor of two in the Hessian follows from the convention \(x^TAx\) with no \(1/2\) in front. If the quadratic term is written \(\frac12 x^TQx\) with \(Q\) symmetric, its Hessian is \(Q\), not \(2Q\).

Worked Example: A Symmetric Quadratic With a Linear Term

Let \(f(x)=3x_1^2+2x_1x_2+4x_2^2-5x_1+2x_2+7\), where \(x=(x_1,x_2)^T\). Find its coefficient matrix, gradient, and Hessian, then calculate the curvature in direction \(d=(1,-1)^T\).

For a symmetric matrix \(Q=\begin{pmatrix}q_{11}&q_{12}\\q_{12}&q_{22}\end{pmatrix}\), the expansion of \(x^TQx\) has cross term \(2q_{12}x_1x_2\). Therefore the quadratic coefficient matrix here is \(Q=\begin{pmatrix}3&1\\1&4\end{pmatrix}\), since \(2q_{12}=2\). The linear coefficient is \(b=(-5,2)^T\), and \(c=7\).

The formula for the gradient and Hessian gives \(\nabla f(x)=b+2Qx\) and \(\nabla^2f=2Q\). Explicitly,

\nabla f(x)= \begin{pmatrix}-5\\2\end{pmatrix} + \begin{pmatrix}6&2\\2&8\end{pmatrix} \begin{pmatrix}x_1\\x_2\end{pmatrix}, \qquad \nabla^2 f= \begin{pmatrix}6&2\\2&8\end{pmatrix}.

The second derivative along \(d\) is \(d^T(\nabla^2 f)d\). Multiplying gives \((\nabla^2 f)d=(4,-6)^T\), so \(d^T(\nabla^2 f)d=(1,-1)(4,-6)^T=10\). The objective therefore curves upward along this direction. The linear and constant terms do not enter this second derivative.

Why Only the Symmetric Part Matters

The coefficient matrix of a quadratic expression may look asymmetric, even though the Hessian cannot be asymmetric. The key identity is that for every real vector \(x\), the scalar \(x^TAx\) equals its transpose. Hence \(x^TAx=(x^TAx)^T=x^TA^Tx\), and averaging these equal scalars yields \(x^TAx=x^T(A+A^T)x/2\).

Result: If \(K^T=-K\), then \(x^TKx=0\) for every real vector \(x\). Consequently, adding a skew-symmetric matrix to \(A\) changes neither \(x^TAx\) nor the Hessian of \(x^TAx\).

Here is a direct proof. Since \(x^TKx\) is a scalar, it equals its transpose. But \((x^TKx)^T=x^TK^Tx=-x^TKx\). Thus \(x^TKx=-x^TKx\), which implies \(x^TKx=0\). Every square matrix decomposes as \(A=A_s+A_k\), where \(A_s=(A+A^T)/2\) is symmetric and \(A_k=(A-A^T)/2\) is skew-symmetric. The identity just proved gives \(x^TAx=x^TA_sx\). Differentiating this expression gives Hessian \(2A_s=A+A^T\). \(\square\)

Worked Example: A Nonsymmetric Coefficient Matrix

Consider \(g(x)=x^TAx+b^Tx\), where \(A=\begin{pmatrix}2&3\\-1&4\end{pmatrix}\) and \(b=(1,-2)^T\). Find the scalar quadratic expression and its Hessian.

Writing out the matrix product gives \(x^TAx=2x_1^2+3x_1x_2-x_1x_2+4x_2^2 =2x_1^2+2x_1x_2+4x_2^2\). The two off-diagonal contributions combine; they do not remain as separate coefficients of the cross term. The symmetric part is \(A_s=\begin{pmatrix}2&1\\1&4\end{pmatrix}\).

The gradient is \(b+(A+A^T)x\), and \(A+A^T=\begin{pmatrix}4&2\\2&8\end{pmatrix}\). Therefore

\nabla g(x)= \begin{pmatrix}1\\-2\end{pmatrix} + \begin{pmatrix}4&2\\2&8\end{pmatrix} \begin{pmatrix}x_1\\x_2\end{pmatrix}, \qquad \nabla^2g= \begin{pmatrix}4&2\\2&8\end{pmatrix}.

For a direction \(d=(u,v)^T\), its second derivative is \(d^T(\nabla^2 g)d=4u^2+4uv+8v^2\). In particular, for \(d=(1,0)^T\) this is \(4\), and for \(d=(0,1)^T\) it is \(8\). The nonsymmetric part of \(A\) has no effect: \(A-A_s=\begin{pmatrix}0&2\\-2&0\end{pmatrix}\) is skew-symmetric and contributes zero to every \(x^TAx\).

Directional Curvature and the Quadratic Form

For a twice differentiable scalar function, the second derivative along the line \(x+td\) at \(t=0\) is \(d^T\nabla^2 f(x)d\). For a quadratic, this value is the same at every base point \(x\), because its Hessian is constant. It is exactly the associated quadratic form of the Hessian. The sign has a direct interpretation: positive values indicate upward curvature along that direction, negative values downward curvature, and zero values no second-order curvature along that direction.

Directional-curvature formula: For \(f(x)=c+b^Tx+x^TAx\) and any direction \(d\in\mathbb{R}^p\), define \(\phi(t)=f(x+td)\). Then \(\phi''(t)=d^T(A+A^T)d\) for every \(t\). The value is independent of both \(x\) and \(t\).

To verify this formula, substitute \(x+td\) into the objective:

\begin{aligned} \phi(t) &=c+b^T(x+td)+(x+td)^TA(x+td)\\ &=c+b^Tx+x^TAx +t\bigl(b^Td+x^TAd+d^TAx\bigr)+t^2d^TAd. \end{aligned}

Differentiating this scalar polynomial twice gives \(\phi''(t)=2d^TAd\). Since \(d^TAd=d^TA_sd\) and \(2A_s=A+A^T\), this equals \(d^T(A+A^T)d\), as claimed. This also confirms that linear terms affect the slope but not curvature. \(\square\)

Worked Example: Curvature in Every Direction for a Squared Residual

Let \(B=\begin{pmatrix}1&2\\2&-1\\0&1\end{pmatrix}\), \(y=(1,0,2)^T\), and \(r(x)=\frac12\|Bx-y\|^2\). Find the Hessian and the second derivative along \(d=(1,1)^T\).

The squared Euclidean norm gives \(r(x)=\frac12(Bx-y)^T(Bx-y) =\frac12x^TB^TBx-x^TB^Ty+\frac12y^Ty\). The columns of \(B\) are \(b_1=(1,2,0)^T\) and \(b_2=(2,-1,1)^T\). Their inner products are \(b_1^Tb_1=5\), \(b_1^Tb_2=2-2+0=0\), and \(b_2^Tb_2=4+1+1=6\). Thus

B^TB=\begin{pmatrix}5&0\\0&6\end{pmatrix}.

Since the quadratic term has the form \(\frac12x^TQx\) with symmetric \(Q=B^TB\), its Hessian is \(Q\). Equivalently, applying the general formula to \(A=\frac12B^TB\) gives \(A+A^T=B^TB\). The linear term \(-x^TB^Ty\) has zero Hessian. Therefore \(\nabla^2r=\begin{pmatrix}5&0\\0&6\end{pmatrix}\). For \(d=(1,1)^T\), the directional second derivative is \(d^T\nabla^2r\,d=(1,1)(5,6)^T=11\), which is positive.

Reading the Coefficient Matrix Without Overstating It

For a symmetric matrix \(Q\), the signs of \(d^TQd\) describe curvature direction by direction. If this expression is positive for every nonzero \(d\), curvature is upward in every nonzero direction. If it is negative for every nonzero \(d\), curvature is downward in every nonzero direction. If it is zero in some nonzero direction, the quadratic has no second-order curvature in that direction; if it takes both positive and negative values, the surface curves upward in some directions and downward in others.

These statements concern the quadratic form itself. They are not a substitute for checking the coefficient matrix actually used in the objective: for \(x^TAx\), use \(A+A^T\) as the Hessian, or equivalently use the quadratic form of \(A_s\) and remember the factor of two. A coefficient matrix that is not symmetric cannot be interpreted by reading its entries as though they were independent curvature coefficients.

Common Mistakes

Dropping the factor of two. For \(x^TAx\), the Hessian is \(A+A^T\); when \(A\) is symmetric, this is \(2A\). For \(\frac12x^TQx\) with symmetric \(Q\), it is \(Q\).
Using a nonsymmetric coefficient matrix as the Hessian. The Hessian is symmetric. Replace \(A\) by \(A_s=(A+A^T)/2\) in the quadratic form, or calculate \(A+A^T\) directly.
Reading the cross-term coefficient as a matrix entry. In a symmetric quadratic form, \(x^TQx\) contains \(2Q_{ij}x_ix_j\) for \(i\ne j\). A displayed cross-term coefficient \(m\) corresponds to \(Q_{ij}=m/2\).
Including linear terms in the curvature calculation. The vector \(b\) changes the gradient but contributes zero to the Hessian and to every directional second derivative.
Confusing the second differential with the Hessian. \(d^2f=dx^T(\nabla^2f)dx\) is a scalar expression; \(\nabla^2f\) is the matrix between the increments.

Key takeaway: Write the quadratic term as \(x^TAx\), symmetrize its coefficient matrix, and then use \(\nabla^2 f=A+A^T\). The quadratic form \(d^T\nabla^2 f\,d\) gives the second derivative—and hence the curvature—in any chosen direction.

Check Your Understanding

Use the quadratic-form and directional-curvature formulas to answer the following.

For \(f(x)=\frac12x^TQx+b^Tx\) with symmetric \(Q\), what is the Hessian, and why does \(b\) not affect it?
For \(x^TAx\), explain why replacing \(A\) with \((A+A^T)/2\) leaves the function unchanged.
If a two-variable objective contains the term \(10x_1x_2\), what are the two off-diagonal entries of its symmetric quadratic coefficient matrix?
For a quadratic with Hessian \(H\), write the second derivative along a direction \(d\). What does a negative value mean for curvature along that direction?
Suppose a quadratic Hessian has \(d^THd=0\) for a nonzero vector \(d\). What can be concluded about second-order curvature along \(d\)?