Quadratic Forms and Congruence

Quadratic Forms and Their Matrix Representations

Quadratic expressions summarize how a vector interacts with a matrix. In multivariate statistics, they occur in expressions for distances, variances, and the curvature of objective functions. Understanding their matrix representation makes it possible to analyze their sign and to see which features survive a change of coordinates.

For example, if \(x=(x_1,x_2)^T\), then \(x^TAx\) is a scalar. Its terms involve products \(x_ix_j\), including squared terms \(x_i^2\) and cross-products such as \(x_1x_2\). The coefficient of a cross-product is split across two entries of \(A\) when \(A\) is symmetric.

Why the Symmetric Matrix Is the Essential Representation

A quadratic form can be represented using a matrix that is not symmetric. However, only its symmetric part affects the value of the form. For any real square matrix \(A\), write its symmetric and skew-symmetric parts as \((A+A^T)/2\) and \((A-A^T)/2\), respectively. The latter contributes zero to \(x^TAx\).

To prove the identity, the scalar \(x^TAx\) equals its transpose, \(x^TA^Tx\). Adding these two equal quantities and dividing by two gives the stated formula. Thus replacing \(A\) by its symmetric part leaves the form unchanged.

For uniqueness, suppose symmetric matrices \(S\) and \(T\) satisfy \(x^TSx=x^TTx\) for every \(x\). Set \(D=S-T\), which is symmetric. Then \(x^TDx=0\) for every \(x\). Taking \(x=e_i\), the \(i\)th coordinate vector, gives \(D_{ii}=0\). For \(i\ne j\), take \(x=e_i+e_j\). Since the diagonal entries are zero and \(D\) is symmetric, \(0=x^TDx=2D_{ij}\), so \(D_{ij}=0\). Every entry of \(D\) is zero, hence \(S=T\). \(\square\)

Classifying the Sign of a Quadratic Form

The sign of a quadratic form is assessed over all vectors, not just over the coordinate axes. A form is positive definite if \(q(x)>0\) for every nonzero \(x\), and negative definite if \(q(x)<0\) for every nonzero \(x\). It is positive semidefinite if \(q(x)\ge 0\) for all \(x\), and negative semidefinite if \(q(x)\le 0\) for all \(x\). It is indefinite if it takes both positive and negative values. The zero form is both positive and negative semidefinite, but is neither definite nor indefinite.

For a real symmetric matrix, the spectral theorem gives an orthogonal matrix \(U\) and a real diagonal matrix \(\Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_p)\) such that \(A=U\Lambda U^T\). Since \(U^TU=I\), putting \(z=U^Tx\) gives \(q(x)=\sum_i\lambda_i z_i^2\). This expresses the sign classification in terms of the eigenvalues: all positive eigenvalues give a positive definite form; all nonnegative eigenvalues give a positive semidefinite form; all negative eigenvalues give a negative definite form; and eigenvalues of both signs give an indefinite form. Zero eigenvalues prevent definiteness but do not, by themselves, make a form indefinite.

Congruence: Changing Coordinates in a Quadratic Form

Let \(x=Cy\), where \(C\) is a real \(p\times p\) matrix. Substitution into the quadratic form gives

Congruence is not the same as similarity: similarity has the form \(C^{-1}AC\), while congruence has the form \(C^TAC\). An invertible congruence can change the eigenvalues, so eigenvalues are not generally preserved. What it does preserve for a symmetric matrix is the number of positive, zero, and negative eigenvalues, counted with multiplicity. This triple is called the inertia of the matrix.

To prove the theorem, first note that \(B=C^TAC\) is symmetric. For a symmetric matrix, the number of positive eigenvalues equals the maximum dimension of a subspace on which its quadratic form is strictly positive for every nonzero vector. Indeed, the span of the positive-eigenvalue eigenvectors is such a subspace. If the positive eigenvalue count is \(r\), any subspace of dimension greater than \(r\) contains a nonzero vector whose projection onto that positive-eigenvalue span is zero. In an orthonormal eigenbasis, that vector has only zero or negative-eigenvalue coordinates, so its quadratic-form value is nonpositive. Thus no larger strictly positive subspace exists. The corresponding argument applied to \(-A\) gives the negative eigenvalue count.

Now \(q_B(y)=q_A(Cy)\). Since \(C\) is invertible, it maps subspaces bijectively and preserves their dimensions. A subspace is strictly positive for \(q_B\) exactly when its image under \(C\) is strictly positive for \(q_A\). Therefore the maximum dimensions of positive subspaces agree; the same reasoning applies to negative subspaces. Finally, \(By=0\) holds exactly when \(ACy=0\), since \(C^T\) is invertible. The map \(y\mapsto Cy\) therefore gives a bijection between the null spaces of \(B\) and \(A\), so their nullities agree as well. These three equal counts establish equality of inertia. \(\square\)

Why Congruence Matters—and a Common Pitfall

In applications, a change of coordinates can make a quadratic expression easier to interpret or calculate. Congruence explains which features are coordinate-dependent and which are not: the entries and eigenvalues may change, but an invertible change preserves inertia. This distinction is useful when analyzing quadratic criteria or expressions involving several variables.

For example, take \(A=\operatorname{diag}(1,4)\) and \(C=\operatorname{diag}(2,1)\). Then \(C^TAC=\operatorname{diag}(4,4)\). The eigenvalues changed from \(1,4\) to \(4,4\), while both matrices have two positive eigenvalues. It would be a mistake to claim that congruent matrices have the same eigenvalues; the invariant here is inertia, not the eigenvalue values.

• Cross-term error: In a symmetric matrix, the entries \(A_{ij}\) and \(A_{ji}\) both contribute to \(x_ix_j\). Thus an off-diagonal entry \(c\) gives a cross-term \(2c x_ix_j\), not \(c x_ix_j\).
• Similarity error: Replacing \(C^TAC\) by \(C^{-1}AC\) uses the wrong transformation for a quadratic form. Substitution \(x=Cy\) directly gives \(C^TAC\).
• Sign error: Checking only a few vectors cannot establish that a form is nonnegative everywhere. A single positive and a single negative value do establish that it is indefinite.
• Invertibility error: Inertia is preserved by invertible congruence. If \(C\) is singular, the substitution covers only a lower-dimensional set, and the theorem does not apply.