Diagnosing Gaussian Parameter Specifications

What Makes a Gaussian Specification Feasible?

A proposed Gaussian model may look plausible entry by entry and still fail as a probability distribution. A covariance matrix must encode variances and covariances that can occur together; a correlation matrix must satisfy an additional standardization requirement. The mean vector, by contrast, has no positivity restriction. The diagnostic task is to separate these conditions and check each one explicitly.

Let \(X\) be a random vector in \(\mathbb{R}^p\), with mean vector \(\mu=E[X]\) and covariance matrix \(\Sigma=\operatorname{Cov}(X)\). A Gaussian specification gives a proposed pair \((\mu,\Sigma)\). It may describe a nonsingular Gaussian distribution or, if \(\Sigma\) is singular, a degenerate Gaussian law concentrated on a lower-dimensional affine subspace.

Definition: A real \(p\times p\) matrix \(\Sigma\) is positive semidefinite if it is symmetric and \(a^\mathsf{T}\Sigma a\geq 0\) for every \(a\in\mathbb{R}^p\). It is positive definite if it is symmetric and \(a^\mathsf{T}\Sigma a>0\) for every nonzero \(a\in\mathbb{R}^p\). A covariance matrix must be positive semidefinite; it need not be positive definite.

The quadratic form has a direct probabilistic meaning. For any coefficient vector \(a\), \(a^\mathsf{T}X\) is a scalar linear combination, and its variance is \(a^\mathsf{T}\Sigma a\). A negative value would therefore propose a negative variance, which is impossible. A zero value is allowed: it means that particular linear combination has no random variation.

The proposed mean \(\mu\) can be any vector of finite real numbers. Neither its entries nor their signs need to satisfy a matrix-like constraint. The restrictions concern the covariance structure.

A Feasibility Theorem for Gaussian Parameters

Theorem: For any \(\mu\in\mathbb{R}^p\), a Gaussian random vector with mean \(\mu\) and covariance matrix \(\Sigma\) exists if and only if \(\Sigma\) is symmetric and positive semidefinite. If \(\Sigma\) has rank \(r\), the Gaussian law has \(r\) nonzero directions of variation. It is nonsingular exactly when \(r=p\), equivalently when \(\Sigma\) is positive definite.

Proof. First suppose such a random vector \(X\) exists and has finite second moments. Covariance symmetry follows entrywise: \(\operatorname{Cov}(X_i,X_j)=\operatorname{Cov}(X_j,X_i)\). For every \(a\in\mathbb{R}^p\),

a^\mathsf{T}\Sigma a =\operatorname{Var}(a^\mathsf{T}X) =E\!\left[\{a^\mathsf{T}(X-\mu)\}^2\right]\geq 0.

Thus \(\Sigma\) must be positive semidefinite.

Conversely, suppose \(\Sigma\) is symmetric and positive semidefinite. By the spectral theorem, it has an orthogonal diagonalization \(\Sigma=Q\Lambda Q^\mathsf{T}\), where the diagonal entries of \(\Lambda\) are nonnegative. Let its positive eigenvalues be \(\lambda_1,\ldots,\lambda_r\), with corresponding orthonormal eigenvectors \(q_1,\ldots,q_r\). Take \(Z\sim N(0,I_r)\) and define

X=\mu+\sum_{j=1}^{r}\sqrt{\lambda_j}\,q_j Z_j.

This is an affine image of a standard Gaussian vector, so it is Gaussian. Since \(E[Z]=0\) and \(\operatorname{Cov}(Z)=I_r\), its mean is \(\mu\), and its covariance is

\operatorname{Cov}(X) =\sum_{j=1}^{r}\lambda_j q_jq_j^\mathsf{T} =\Sigma.

There are exactly \(r\) positive eigenvalues, so the construction has \(r\) independent standard-normal coordinates contributing variation. When \(r=p\), every eigenvalue is positive and the covariance is positive definite; when \(r<p\), at least one direction has zero variance and the law is singular. \(\square\)

Worked Covariance Diagnostics

Worked Example: A Valid Positive-Definite Covariance

Consider the proposed mean and covariance

\mu= \begin{pmatrix}2\\-1\end{pmatrix}, \qquad \Sigma= \begin{pmatrix}4&3\\3&9\end{pmatrix}.

The mean is a real vector, so it presents no feasibility problem. The covariance is symmetric. To check positive definiteness, calculate its leading principal minors: the first is \(4>0\), and the determinant is

\det(\Sigma)=4(9)-3(3)=36-9=27>0.

For a symmetric \(2\times2\) matrix, these two strict inequalities imply positive definiteness. More directly, its quadratic form is

4a^2+6ab+9b^2 =4\left(a+\frac{3}{4}b\right)^2+\frac{27}{4}b^2,

which is positive for every \((a,b)\ne(0,0)\). Therefore the specification is valid and nonsingular. The corresponding correlation is \(3/(\sqrt{4}\sqrt{9})=3/6=1/2\), which lies in the permitted interval \([-1,1]\).

Worked Example: Positive Diagonal Entries Do Not Rescue an Invalid Matrix

Consider

\Sigma= \begin{pmatrix} 1&2&0\\ 2&1&0\\ 0&0&4 \end{pmatrix}.

The matrix is symmetric, and its diagonal entries are all positive. Those checks are necessary but not sufficient. Choose \(a=(1,-1,0)^\mathsf{T}\). Then

a^\mathsf{T}\Sigma a = \begin{pmatrix}1&-1&0\end{pmatrix} \begin{pmatrix}-1\\1\\0\end{pmatrix} =-2<0.

This proposed matrix is not positive semidefinite, so it cannot be a covariance matrix. Equivalently, the proposed variance of \(X_1-X_2\) would be \(\Sigma_{11}+\Sigma_{22}-2\Sigma_{12}=1+1-4=-2\), which is impossible.

Worked Example: A Valid Singular Covariance

Now take

\Sigma= \begin{pmatrix} 1&1&0\\ 1&1&0\\ 0&0&0 \end{pmatrix}.

This matrix is symmetric. For any \(a=(a_1,a_2,a_3)^\mathsf{T}\),

a^\mathsf{T}\Sigma a=(a_1+a_2)^2\geq 0.

It is therefore positive semidefinite. Its rank is one: its first two rows are equal and nonzero, while the third row is zero. A Gaussian vector with this covariance is possible, for example \(X=\mu+(Z,Z,0)^\mathsf{T}\), where \(Z\sim N(0,1)\). The first and second centered coordinates are identical, and the third coordinate is constant. This is a valid Gaussian specification, but not a nonsingular one.

Diagnosing Correlation Matrices

A correlation matrix is not an arbitrary covariance matrix with plausible-looking entries. Each diagonal entry must be one, and the full matrix must be positive semidefinite. In particular, checking that every individual correlation lies between \(-1\) and \(1\) is not enough to guarantee that all the correlations can coexist.

Theorem: A real matrix \(R\) is a feasible correlation matrix if and only if it is symmetric, positive semidefinite, and has every diagonal entry equal to one. If all variances are positive and \(D=\operatorname{diag}(\sigma_1,\ldots,\sigma_p)\), where \(\sigma_i=\sqrt{\Sigma_{ii}}>0\), then covariance and correlation specifications agree through \(\Sigma=DRD\) and \(R=D^{-1}\Sigma D^{-1}\).

Proof. If \(R\) is the correlation matrix of a random vector with positive marginal variances, its diagonal entries are \(\operatorname{Corr}(X_i,X_i)=1\), it is symmetric, and for any \(a\),

a^\mathsf{T}Ra =\operatorname{Var}\!\left(\sum_{i=1}^p a_i\frac{X_i-E[X_i]}{\sigma_i}\right)\geq 0.

So it is positive semidefinite. Conversely, if \(R\) is symmetric, positive semidefinite, and has unit diagonal, the feasibility theorem constructs a Gaussian vector with mean zero and covariance \(R\). Its component variances are all one, so its covariance matrix is also its correlation matrix. For positive marginal standard deviations, scaling each component by \(\sigma_i\) gives \(\Sigma=DRD\); reversing this scaling gives \(R=D^{-1}\Sigma D^{-1}\). \(\square\)

Every \(2\times2\) principal submatrix of a feasible correlation matrix must itself be positive semidefinite. Its form is \(\begin{pmatrix}1&r\\r&1\end{pmatrix}\), whose determinant is \(1-r^2\). Thus each pairwise correlation must satisfy \(|r|\leq1\). This pairwise bound is necessary, but a matrix with all pairwise correlations in range can still fail the full positive-semidefiniteness check.

Worked Example: Pairwise-Plausible Correlations That Cannot Coexist

Suppose a proposed three-variable correlation matrix is

R= \begin{pmatrix} 1&0.8&0.8\\ 0.8&1&0.2\\ 0.8&0.2&1 \end{pmatrix}.

It is symmetric, has unit diagonal, and all three off-diagonal entries are between \(-1\) and \(1\). But its determinant is

\det(R) =1+2(0.8)(0.8)(0.2)-(0.8^2+0.8^2+0.2^2) =1+0.256-1.32=-0.064.

A symmetric positive semidefinite matrix cannot have a negative determinant: its eigenvalues are all nonnegative, so their product is nonnegative. Hence \(R\) is not positive semidefinite and cannot be a correlation matrix. The defect is a joint inconsistency among the three proposed correlations, not a violation of any one pairwise bound.

Worked Example: A Singular but Valid Correlation Matrix

Consider three variables with equal pairwise correlations of \(-1/2\):

R= \begin{pmatrix} 1&-\frac12&-\frac12\\ -\frac12&1&-\frac12\\ -\frac12&-\frac12&1 \end{pmatrix}.

This matrix can be written \(R=\frac32 I-\frac12 J\), where \(J\) is the \(3\times3\) all-ones matrix. The vector \((1,1,1)^\mathsf{T}\) is an eigenvector of \(J\) with eigenvalue \(3\), so it is an eigenvector of \(R\) with eigenvalue \(0\). Every vector orthogonal to \((1,1,1)^\mathsf{T}\) is an eigenvector of \(J\) with eigenvalue \(0\), and hence an eigenvector of \(R\) with eigenvalue \(3/2\). Thus the eigenvalues of \(R\) are \(0,3/2,3/2\): it is positive semidefinite with rank two. It is a valid singular correlation matrix. In a Gaussian model, the zero eigenvalue means a centered linear combination of the three standardized variables has zero variance; it does not make the proposed correlations invalid.

A Reliable Diagnostic Sequence

Check the dimensions and entries.
Confirm that the mean has \(p\) finite real entries and the proposed covariance or correlation matrix is \(p\times p\).

Check symmetry.
Verify that the \((i,j)\) and \((j,i)\) entries agree. A covariance matrix is symmetric by definition.

Check positive semidefiniteness.
Use eigenvalues or test the quadratic form. Any negative eigenvalue, or any vector producing a negative quadratic form, rules out a covariance specification.

Interpret the rank.
Full rank means a nonsingular Gaussian law; lower rank means a valid but degenerate law, provided the matrix is still positive semidefinite.

For correlations, check standardization too.
Require unit diagonal and check the matrix as a whole, not only whether each off-diagonal entry lies in \([-1,1]\).

Common Mistakes and Why These Checks Matter

Rejecting a singular covariance automatically. Singularity means some linear combination has zero variance. It is allowed for a Gaussian law; only a negative eigenvalue makes the covariance infeasible.
Accepting a matrix because its diagonal is positive. Diagonal entries check individual variances, but they do not ensure that combinations of variables have nonnegative variance. The invalid example above had positive diagonal entries and a negative quadratic form.
Checking correlations one at a time and stopping. The pairwise bounds \(|r_{ij}|\leq1\) are necessary, not sufficient. Positive semidefiniteness is a joint condition on the entire matrix.
Treating every covariance matrix as a correlation matrix. Covariance diagonal entries are variances and need not equal one. Standardization uses \(D^{-1}\Sigma D^{-1}\), and this requires positive marginal variances.

If a proposed covariance has a zero diagonal entry, positive semidefiniteness forces every entry in that row and column to be zero. Indeed, if \(\Sigma_{ii}=0\), the variance of \(X_i\) is zero, so \(\operatorname{Cov}(X_i,X_j)=0\) for every \(j\). In that case a correlation involving \(X_i\) is undefined, because its standard deviation is zero; it is not a correlation of zero. This is why covariance and correlation specifications cannot always be converted into one another.

Key takeaway: A Gaussian mean may be any finite real vector. A covariance matrix must be symmetric and positive semidefinite; rank determines whether the law is nonsingular or degenerate. A correlation matrix must also have unit diagonal, and its entries must be jointly compatible.

Check Your Understanding

For each proposed specification, identify the relevant checks before deciding whether it is feasible.

Can a Gaussian vector have mean \((3,-8,0)^\mathsf{T}\)? What, if anything, must be checked about this mean?
A symmetric covariance matrix has eigenvalues \(5,2,0\). Is it feasible, and is the associated Gaussian law nonsingular?
Why are positive diagonal entries and pairwise correlations in \([-1,1]\) not enough to certify a proposed covariance or correlation matrix?
For a covariance matrix with positive diagonal entries, how do you convert it to a correlation matrix?
If one component has variance zero, can its correlation with another component be reported as zero? Explain.