CCA with Singular Covariance Matrices

Why Singular Covariance Matrices Matter

Canonical correlation analysis (CCA) seeks linear combinations of two data blocks whose scores have high correlation. Its classical calculations rely on the within-block covariance matrices being invertible. In multi-omics data, however, a block may contain more features than samples, exact duplicate measurements, or features determined by other features. The resulting covariance matrix is singular: at least one nonzero direction in feature space has zero variance. In that case, the inverse-based equations cannot be used as written, and the canonical weights may not be uniquely determined.

Singularity is not just a numerical inconvenience. It can signal redundant measurements, an insufficient sample size for the number of features, or a structural constraint in the data. A numerical routine may still return weights or correlations, but those outputs require interpretation: the correlations can be determined on the observed subspaces while the weights in the original feature coordinates are not uniquely identified.

Rank, Null Spaces, and the Classical CCA Problem

Let \(X\) and \(Y\) be centered data matrices for two blocks measured on the same \(n\) samples, with \(p\) and \(q\) features. Write their sample covariance matrices and cross-covariance matrix as

For coefficient vectors \(a\) and \(b\), the corresponding scores are \(Xa\) and \(Yb\). Their sample variances are \(a^\top S_{xx}a\) and \(b^\top S_{yy}b\), and their sample covariance is \(a^\top S_{xy}b\). Classical CCA maximizes their correlation, equivalently maximizing covariance subject to both score variances being one. This formulation requires \(a^\top S_{xx}a>0\) and \(b^\top S_{yy}b>0\); a zero-variance score has no defined correlation.

A covariance matrix is singular when its rank is smaller than its dimension. Equivalently, it has at least one zero eigenvalue, and its null space contains a nonzero vector. For example, if \(S_{xx}a_0=0\), then \(Xa_0=0\): the data produce a score identically equal to zero in that direction. For centered data, \(\operatorname{rank}(X)\leq \min(n-1,p)\), so \(S_{xx}\) must be singular if \(p>n-1\). Exact linear dependencies among columns can make it singular even when \(p\leq n-1\).

When \(S_{xx}\) is singular, \(S_{xx}^{-1}\) does not exist, so the inverse-based generalized eigenvalue equations for classical CCA are undefined. The constrained optimization can still be considered over nonzero-variance scores, but its coefficients may not be identifiable. If \(a\) is a solution and \(a_0\) lies in the null space of \(S_{xx}\), then \(X(a+a_0)=Xa\). Thus these two coefficient vectors give exactly the same score. The reported weights are not unique even though the score can be.

What the Pseudoinverse Does

A standard way to describe CCA on the effective subspaces uses the Moore–Penrose pseudoinverse. For a symmetric positive semidefinite matrix \(S\), let \(S=U\Lambda U^\top\) be an eigenvalue decomposition. The matrix \(S^{+1/2}\) replaces each positive eigenvalue \(\lambda\) by \(\lambda^{-1/2}\), and leaves each zero eigenvalue at zero. It whitens the directions with positive variance while excluding the null space. The canonical correlations on the effective subspaces are the singular values of

This expression presumes that the covariance blocks are compatible parts of a joint covariance matrix, as they are when calculated from the same paired observations. In particular, a zero-variance direction in one block cannot have nonzero covariance with a direction in the other block. For example, if \(Xa_0=0\), then \(a_0^\top S_{xy}=0\) follows directly from \(S_{xy}=X^\top Y/(n-1)\). Therefore the cross-block covariance acts only between directions with nonzero variance. Whitening those directions gives a matrix whose singular values are correlations and hence cannot exceed one.

The pseudoinverse gives a well-defined calculation on the nonzero-variance subspaces; it does not make every original-coordinate weight uniquely meaningful. Any component added in a within-block null space leaves the score unchanged. A Moore–Penrose calculation selects a minimum-norm representative among equivalent choices, but that convention is not evidence that the selected weights are uniquely identified by the data.

Diagnosing Rank Deficiency

Diagnosis should use both the data and the covariance spectrum. Start by checking the number of samples and features, then compute numerical ranks or inspect eigenvalues or singular values. Exact rank is a mathematical property; numerical rank depends on a tolerance because floating-point calculations rarely produce exact zeros. A very small eigenvalue relative to the largest can indicate near rank deficiency: inversion is technically possible, but small perturbations can cause large changes in the estimated weights.

• Compare the number of features with the maximum possible centered-data rank, \(n-1\).
• Inspect eigenvalues or singular values for zeros and for a large gap between retained and discarded values.
• Check for constant features, duplicated measurements, and exact linear combinations of features.
• Assess weight and score stability under reasonable changes in preprocessing or resampling. Large weight changes with similar score correlations indicate weak identification or near-collinearity.

A condition number—the ratio of the largest to the smallest positive eigenvalue—can summarize the spread of scales on the effective subspace. It does not by itself identify a universal cutoff: the choice of numerical tolerance should reflect scaling, precision, and the analysis. Standardization changes feature scales and can help make spectral comparisons interpretable, but it cannot restore information to an exact linear dependency.

Two Remedies: Pseudoinverse and Dimension Reduction

A pseudoinverse retains all positive-variance directions and sets null directions aside. It is appropriate when the aim is to compute CCA on the observed subspaces without declaring any nonzero-variance direction irrelevant. Exact dimension reduction does the same thing in another coordinate system: represent each block in a basis for its full effective subspace, then run ordinary CCA on those reduced coordinates. If no positive-variance directions are discarded, these approaches preserve the same attainable canonical correlations.

Truncated dimension reduction is different. For example, retaining only the first few principal components discards positive-variance directions as well as null directions. That may stabilize or simplify an analysis, but it changes the space over which CCA is optimized. The resulting correlations describe the retained subspaces, and an association confined to a discarded direction can disappear. The choice of retained dimension should therefore be justified and assessed for stability rather than treated as a purely computational fix.

Assumptions, Limitations, and Common Pitfalls

The matrix arguments above require paired observations and covariance matrices that represent the same centered sample or a compatible joint covariance model. A pseudoinverse calculation still requires positive-variance directions in both blocks; if an entire block has zero variance, no canonical correlation is defined. For inference, the assumptions and procedures of classical tests cannot automatically be carried over after rank selection or truncation. Selection affects the analysis, and a computed sample correlation alone does not establish population association.

A common mistake is to interpret a minimum-norm pseudoinverse weight vector as the uniquely correct biological signature. In the duplicate-feature example, many weight vectors generated the same score; the minimum-norm choice was a computational convention. Report which subspace was used, how numerical rank was determined, and whether dimension reduction discarded positive-variance directions. If original features are highly redundant, score patterns may be more stable and interpretable than individual weights.

Another pitfall is to assume that removing exact zeros resolves severe near-singularity. A tiny but positive eigenvalue is inverted to a large value by ordinary whitening, potentially magnifying sampling noise. Rank thresholds and retained dimensions should be examined for sensitivity. When a regularized CCA method is considered as an alternative, its estimates depend on additional choices; it is not the same operation as taking a pseudoinverse or selecting an exact-rank basis.