Positive-Definite Repairs After Covariance Thresholding

Why a Thresholded Covariance Matrix May Need Repair

A covariance matrix must be positive semidefinite: every linear combination of the variables has nonnegative variance. Entrywise thresholding can reduce noisy off-diagonal entries, but it makes decisions one entry at a time and does not enforce this joint constraint. The resulting estimate can be symmetric, sparse, and still fail to be a valid covariance matrix. That matters when a downstream calculation requires a matrix inverse, a multivariate normal model, or nonnegative estimated variances for all linear combinations.

The repair method affects more than numerical stability. Projection onto the positive semidefinite cone has a precise squared-Frobenius-loss guarantee. Projection onto correlation matrices has a related guarantee on the correlation scale. Neither statement says that every repair improves every loss, especially after imposing a strict eigenvalue floor or changing marginal variances.

Definition: A real symmetric matrix \(A\) is positive semidefinite (PSD) if \(x^{\mathsf T}Ax\geq 0\) for every vector \(x\), and positive definite (PD) if \(x^{\mathsf T}Ax>0\) for every nonzero \(x\). A covariance matrix is PSD; a nonsingular covariance matrix is PD. The squared Frobenius loss for estimating \(\Sigma\) by \(A\) is \(\|A-\Sigma\|_F^2=\sum_{i,j}(A_{ij}-\Sigma_{ij})^2\).

Eigenvalue Clipping

Let \(T\) be a symmetric thresholded estimate, with spectral decomposition \(T=Q\operatorname{diag}(\lambda_1,\ldots,\lambda_p)Q^{\mathsf T}\). Replacing each negative eigenvalue by zero gives

T_+=Q\operatorname{diag}\bigl(\max(\lambda_1,0),\ldots,\max(\lambda_p,0)\bigr)Q^{\mathsf T}.

This is PSD, but it may be singular. If an application requires an inverse, a positive floor \(\varepsilon>0\) gives \(T_\varepsilon=Q\operatorname{diag}(\max(\lambda_1,\varepsilon),\ldots,\max(\lambda_p,\varepsilon))Q^{\mathsf T}\), which is PD. The floor is a further modification: unlike clipping at zero, it does not in general have the loss guarantee for projection onto the PSD cone.

Worked Example: A Thresholded Matrix with a Negative Eigenvalue

Consider the symmetric matrix

T=\begin{pmatrix}1&0.9&0.9\\0.9&1&-0.9\\0.9&-0.9&1\end{pmatrix}.

Its diagonal entries are positive, but the unit vector \(u=(1,-1,-1)^{\mathsf T}/\sqrt{3}\) is an eigenvector with eigenvalue \(-0.8\): multiplying \(T\) by \((1,-1,-1)^{\mathsf T}\) gives \((-0.8,0.8,0.8)^{\mathsf T}\). The two orthogonal eigenvectors have eigenvalue \(1.9\), so the full spectrum is \(1.9,1.9,-0.8\). Thus \(T\) is indefinite.

Clipping the negative eigenvalue to zero adds \(0.8uu^{\mathsf T}\) to \(T\). Since \(uu^{\mathsf T}\) has diagonal entries \(1/3\), its repaired diagonal entries are \(1+0.8/3=19/15\). Its off-diagonal entries are \(0.9-0.8/3=19/30\) in positions \((1,2)\) and \((1,3)\), and \(-0.9+0.8/3=-19/30\) in position \((2,3)\). The repaired matrix is PSD and singular: its eigenvalues are \(1.9,1.9,0\). A floor of \(0.1\) instead adds \(0.9uu^{\mathsf T}\), producing a PD matrix with eigenvalues \(1.9,1.9,0.1\).

This repair changes the diagonal as well as the off-diagonal entries. If retaining the original variances is important, clipping alone does not accomplish that; a correlation-scale repair is one alternative.

What the PSD Projection Guarantees

For symmetric matrices, the PSD matrices form a closed convex set. The nearest point in that set under Frobenius distance is exactly \(T_+\). The next result establishes both the projection formula and its risk consequence, without assuming that \(T\) itself is PSD.

Proof. Write \(T=Q\operatorname{diag}(\lambda_i)Q^{\mathsf T}\), and for any PSD \(B\) set \(C=Q^{\mathsf T}BQ\). Then \(C\) is PSD, so \(C_{ii}\geq0\). Orthogonal invariance of the Frobenius norm gives

\|T-B\|_F^2=\sum_i(\lambda_i-C_{ii})^2+\sum_{i\ne j}C_{ij}^2 \geq \sum_i\bigl(\lambda_i-\max(\lambda_i,0)\bigr)^2.

Equality is attained by choosing \(C\) diagonal with entries \(\max(\lambda_i,0)\). Hence \(T_+\) is a nearest PSD matrix. To obtain the loss claim, projection onto a closed convex set satisfies \(\langle T-T_+,\,\Sigma-T_+\rangle\leq0\) for every \(\Sigma\) in the set. Expanding the squared distance,

\|T-\Sigma\|_F^2 =\|T-T_+\|_F^2+\|T_+-\Sigma\|_F^2 +2\langle T-T_+,T_+-\Sigma\rangle \geq \|T_+-\Sigma\|_F^2.

This applies pointwise whenever the target covariance is PSD, so the repair cannot increase squared Frobenius error in that comparison. It does not establish the same guarantee for a positive eigenvalue floor, for operator-norm loss, or for a procedure that subsequently rescales the matrix. \(\square\)

Worked Example: Checking the Frobenius-Loss Guarantee

Take \(T=\begin{pmatrix}1&2\\2&1\end{pmatrix}\), whose eigenvalues are \(3\) and \(-1\). Its PSD projection is \(T_+=\begin{pmatrix}1.5&1.5\\1.5&1.5\end{pmatrix}\). Let the true covariance be \(\Sigma=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\), which has eigenvalues \(1.5\) and \(0.5\) and is therefore PSD.

Before repair, the error is \(\begin{pmatrix}0&1.5\\1.5&0\end{pmatrix}\), so \(\|T-\Sigma\|_F^2=1.5^2+1.5^2=4.5\). After repair, the error is \(\begin{pmatrix}0.5&1\\1&0.5\end{pmatrix}\), so \(\|T_+-\Sigma\|_F^2=0.5^2+1^2+1^2+0.5^2=2.5\). The decrease is consistent with the theorem. A different target or a different loss would require a separate comparison.

Nearest-Correlation Projection

When the diagonal entries of a covariance estimate are positive and should be retained, first express the estimate as a covariance-scale matrix \(T=D^{1/2}AD^{1/2}\), where \(D\) is its diagonal matrix of variances and \(A\) has diagonal entries one. The correlation matrix \(A\) may be indefinite even though its diagonal is one. Its nearest-correlation repair solves a constrained problem rather than clipping covariance eigenvalues.

Definition: A correlation matrix is a symmetric PSD matrix with every diagonal entry equal to one. The nearest-correlation projection of a symmetric matrix \(A\) is the matrix \(C_*\) minimizing \(\|C-A\|_F\) over all correlation matrices \(C\). This minimizer exists and is unique: the feasible set is nonempty, closed, bounded, and convex, and squared Frobenius distance is strictly convex. The covariance-scale repaired estimate is \(D^{1/2}C_*D^{1/2}\).

A nearest-correlation projection is PSD, not necessarily PD. If strict positive definiteness is needed, one can instead minimize distance over correlation matrices satisfying \(C\succeq\varepsilon I\), with \(0<\varepsilon<1\); this set is nonempty because \(I\succeq\varepsilon I\). This constrained projection remains a well-defined convex projection, but it is not the unconstrained nearest-correlation matrix.

Theorem: If the target \(R\) is a correlation matrix, then its nearest-correlation projection satisfies \(\|C_*-R\|_F\leq\|A-R\|_F\).

Proof. The projection property for a closed convex set gives \(\langle A-C_*,R-C_*\rangle\leq0\). Since \(A-R=(A-C_*)+(C_*-R)\), expansion yields

\|A-R\|_F^2 =\|A-C_*\|_F^2+\|C_*-R\|_F^2 +2\langle A-C_*,C_*-R\rangle \geq\|C_*-R\|_F^2.

The final inequality follows because \(\langle A-C_*,C_*-R\rangle\geq0\). The result is a Frobenius guarantee on the correlation scale; covariance-scale consequences depend on the marginal variances used to rescale. \(\square\)

Worked Example: Projecting an Indefinite Matrix onto Correlations

Return to the matrix \(T\) in the first example, whose diagonal is \(I\), so it is already on the correlation scale. Write its off-diagonal signs as \(+,+,-\), and let \(B\) have those signs off the diagonal and zeros on the diagonal. Then \(T=I+0.9B\). The eigenvalues of \(B\) are \(1,1,-2\), so the eigenvalues of \(I+xB\) are \(1+x,1+x,1-2x\). It is a correlation matrix exactly when \(-1\leq x\leq0.5\).

The signed pattern is unchanged by relabeling the three variables together with suitable sign changes, and these transformations preserve both the feasible set and Frobenius distance. Uniqueness of the projection therefore forces its off-diagonal entries to retain this pattern with a common magnitude \(x\). Its squared distance from \(T\) is \(6(x-0.9)^2\), because there are six off-diagonal entries. Over \(-1\leq x\leq0.5\), this is minimized at \(x=0.5\). Thus the nearest correlation matrix has off-diagonal entries \(0.5,0.5,-0.5\) and eigenvalues \(1.5,1.5,0\): it is PSD but singular.

For a covariance-scale illustration, let \(D=\operatorname{diag}(1,4,9)\) and suppose the true covariance is \(\Sigma=D\), whose correlation matrix is \(I\). The uncorrected covariance estimate is \(D^{1/2}TD^{1/2}\). Its squared Frobenius error is \(2(0.9)^2(1\cdot4+1\cdot9+4\cdot9)=79.38\): each off-diagonal error occurs twice. After correlation projection, the corresponding error is \(2(0.5)^2(4+9+36)=24.5\). Here the repaired covariance has lower loss, but the correlation-scale theorem alone does not ensure this outcome for every choice of scale matrix and target.

Loss, Scaling, and Practical Choices

For fixed positive diagonal \(D\) and a true covariance \(\Sigma=D^{1/2}RD^{1/2}\), rescaling converts a correlation error \(E=C-R\) into covariance error \(D^{1/2}ED^{1/2}\). If \(d_{\min}\) and \(d_{\max}\) are the smallest and largest diagonal entries of \(D\), then

d_{\min}\|E\|_F \leq \|D^{1/2}ED^{1/2}\|_F \leq d_{\max}\|E\|_F.

To see this, the squared middle quantity is \(\sum_{i,j}d_i d_j E_{ij}^2\), and each weight \(d_i d_j\) lies between \(d_{\min}^2\) and \(d_{\max}^2\). The nearest-correlation guarantee thus implies a covariance-error bound by comparing the two scales, but it does not generally imply a smaller covariance Frobenius error than the original estimate. If \(D\) is itself estimated, its error also contributes and must be accounted for separately.

The methods therefore answer different practical needs. Eigenvalue clipping is the direct projection onto the PSD cone and has a clean Frobenius guarantee, but may change the diagonal. Nearest-correlation projection enforces unit diagonal on the standardized scale and then restores the chosen variances, but its covariance-scale loss depends on those variances. A strict eigenvalue floor can support inversion, at the cost of another biasing modification. The choice should match the required matrix constraints and the loss being assessed, rather than treating positive definiteness as a purely cosmetic adjustment.

Common Pitfall

Wrong reasoning: “The thresholded matrix has positive diagonal entries, so it must be a valid covariance matrix.” Positive diagonal entries check only the variances of individual coordinates. A negative eigenvalue supplies a linear combination with negative estimated variance.
Correction: Check positive semidefiniteness, for example by inspecting eigenvalues. Then choose a repair according to whether preserving the diagonal, ensuring invertibility, or controlling a particular estimation loss is most important.
Wrong reasoning: “Any positive-definite repair must improve covariance estimation.” The Frobenius guarantee applies to projection onto the PSD cone, not automatically to eigenvalue flooring, rescaling, or other losses.

Check Your Understanding

Use the definitions and results above to answer the following questions.

Why does entrywise thresholding fail to guarantee that an estimate is a valid covariance matrix?
What is the difference between clipping negative eigenvalues to zero and replacing them by a positive floor?
State the condition on the target matrix that is needed for the PSD projection Frobenius-loss guarantee.
Why can a nearest-correlation projection be PSD but not PD?
What additional issue arises when translating a correlation-scale loss guarantee into covariance-scale loss?