Correlation Matrices for Gaussian Models

Why Correlation Matrices Need Joint Checks

A covariance matrix records both the scales of the variables and their dependence. A correlation matrix removes the scales, leaving a dimensionless description of their pairwise linear associations. This can make a Gaussian model easier to interpret, but it does not make the parameters independent: the correlations must still fit together as one joint distribution.

In particular, requiring every correlation to lie between \(-1\) and \(1\) is necessary but, with three or more variables, not sufficient. The matrix as a whole must be positive semidefinite. This tutorial relates the covariance and correlation parameterizations and develops an explicit admissibility test for three variables.

From Covariance to Correlation

Let \(X=(X_1,\ldots,X_p)^\mathsf{T}\) have finite second moments, mean vector \(\mu\), and covariance matrix \(\Sigma\). Write \(\sigma_i^2=\operatorname{Var}(X_i)\). When every \(\sigma_i^2>0\), define the diagonal scaling matrix \(D=\operatorname{diag}(\sigma_1,\ldots,\sigma_p)\). The standardized vector is \(Z=D^{-1}(X-\mu)\).

Because \(D\) is invertible, standardization preserves positive semidefiniteness in both directions. Indeed, for any vector \(u\), \(u^\mathsf{T}Ru=(D^{-1}u)^\mathsf{T}\Sigma(D^{-1}u)\); conversely, \(v^\mathsf{T}\Sigma v=(Dv)^\mathsf{T}R(Dv)\). The same argument shows that \(R\) is positive definite exactly when \(\Sigma\) is positive definite. Thus a covariance matrix and its corresponding correlation matrix have the same admissibility status, although their entries have different scales.

A covariance matrix is positive semidefinite because, for every vector \(u\), \(u^\mathsf{T}\Sigma u=\operatorname{Var}(u^\mathsf{T}X)\geq 0\). Consequently, a correlation matrix must also be positive semidefinite. For a Gaussian vector, positive semidefiniteness permits a possibly singular Gaussian law; positive definiteness is required for a nonsingular law with an ordinary density on the full space. If a marginal variance is zero, its correlation with another coordinate is undefined, so the correlation parameterization above does not apply to that coordinate.

Basic Constraints and the Three-Variable Test

For two variables, the correlation matrix has the form \(\begin{pmatrix}1&\rho\\\rho&1\end{pmatrix}\). Its eigenvalues are \(1+\rho\) and \(1-\rho\), so it is positive semidefinite exactly when \(|\rho|\leq 1\), and positive definite exactly when \(|\rho|<1\). With three variables, each of the three pairwise correlations must satisfy the same bound, but a further joint constraint is needed.

The determinant appearing here is obtained by direct expansion: \[ \det(R)=1(1-c^2)-a(a-bc)+b(ac-b)=1+2abc-a^2-b^2-c^2. \] The other required minors are the three \(1\)-by-\(1\) diagonal entries, all equal to one, and the three \(2\)-by-\(2\) principal minors \(1-a^2\), \(1-b^2\), and \(1-c^2\).

Proof. If \(R\) is positive semidefinite, each principal submatrix is positive semidefinite. The \(2\)-by-\(2\) submatrices therefore have nonnegative determinants, giving \(|a|,|b|,|c|\leq 1\). Also, a positive-semidefinite matrix has nonnegative determinant, so the displayed determinant condition holds.

For sufficiency, assume all four stated inequalities. For any \(\varepsilon>0\), the leading principal minors of \(R+\varepsilon I\) are positive. The first is \(1+\varepsilon\). The second is \((1+\varepsilon)^2-a^2>0\), since \(|a|\leq1\). The third is the determinant \[ \det(R+\varepsilon I)=\det(R)+\varepsilon\{(1-a^2)+(1-b^2)+(1-c^2)\}+3\varepsilon^2+\varepsilon^3>0. \] Here \(\det(R)\geq0\), each \(2\)-by-\(2\) principal minor is nonnegative, and the final terms are strictly positive. Sylvester's criterion states that a real symmetric matrix is positive definite if and only if all its leading principal minors are positive. Hence \(R+\varepsilon I\) is positive definite for every \(\varepsilon>0\). For any \(x\), \(x^\mathsf{T}Rx=\lim_{\varepsilon\downarrow0}x^\mathsf{T}(R+\varepsilon I)x\geq0\), proving that \(R\) is positive semidefinite.

For positive definiteness, Sylvester's criterion applied directly to \(R\) requires \(1>0\), \(1-a^2>0\), and \(\det(R)>0\). These imply all three pairwise inequalities are strict, since \(1-b^2\) and \(1-c^2\) are positive leading or principal minors of a positive-definite matrix. Conversely, the stated strict conditions make the leading principal minors positive, so Sylvester's criterion gives positive definiteness. \(\square\)

Boundary Cases and Singular Gaussian Models

A zero determinant is not automatically a defect. It means the matrix is singular; for a Gaussian vector, singular covariance is allowed when a degenerate law is intended. The three-variable criterion also clarifies the special case in which one pairwise correlation is exactly \(1\) or \(-1\).

More generally, if \(a^2=1\), positive semidefiniteness forces \(b=ac\). To see why, the vector \(v=(-a,1,0)^\mathsf{T}\) satisfies \(v^\mathsf{T}Rv=1-a^2=0\). For a positive-semidefinite matrix, a vector with zero quadratic form lies in its null space; hence \(Rv=0\). The third coordinate of \(Rv\) is \(-ab+c\), so \(c=ab\), which, because \(a^2=1\), is equivalent to \(b=ac\). Conversely, if \(a^2=1\) and \(b=ac\), then \[ x^\mathsf{T}Rx=(x+ay+acz)^2+(1-c^2)z^2\geq0, \] using \(|c|\leq1\). Expanding the right-hand side gives \(x^2+y^2+z^2+2axy+2acxz+2cyz\), exactly the quadratic form because \(b=ac\). The matching of the cross terms is essential: a boundary factorization must reproduce every entry of the matrix.

Why the Joint Constraint Matters

The correlation parameterization is useful because it separates marginal scale from dependence. Researchers can specify interpretable standard deviations and correlations, then reconstruct the covariance matrix as \(DRD\). But a list of pairwise associations is not necessarily a coherent multivariate model: all entries must satisfy the matrix-level positive-semidefinite constraint.

For three variables, the determinant test gives a compact complete check. In higher dimensions, positive semidefiniteness still governs admissibility, but the three-variable determinant formula is no longer enough; the full matrix must be checked. A further common pitfall is to equate “singular” with “invalid.” A singular correlation matrix is valid for a degenerate Gaussian law, though it does not define a nonsingular Gaussian density. Conversely, merely having every pairwise correlation in the allowed interval does not guarantee validity.