Diagnosing Unstable Canonical Directions

Why Canonical Directions Need a Stability Check

Canonical correlation analysis (CCA) finds pairs of linear combinations of two variable sets that have large correlations. A reported canonical correlation can look well determined while the weights defining its two canonical variates are sensitive to small changes in the estimated covariance matrices. This matters when the scientific interpretation depends on which variables contribute to a direction, rather than only on the strength of the association.

Instability is not diagnosed by one number. Small eigenvalues in within-set covariance matrices can amplify estimation error during whitening; close canonical roots can make their directions difficult to distinguish; and large weights can signal either an ill-conditioned calculation or merely a choice of measurement units. Comparing results after realistic perturbations helps reveal which features persist.

Definition: Let $S_{XX}$ and $S_{YY}$ be positive definite sample covariance matrices for two variable sets, and let $S_{XY}$ be their sample cross-covariance matrix. Define the whitened cross-covariance matrix $$ K=S_{XX}^{-1/2}S_{XY}S_{YY}^{-1/2}. $$ Its singular values are the sample canonical correlations. The corresponding left and right singular vectors describe canonical directions in whitened coordinates; the original-variable coefficient vectors also depend on $S_{XX}^{-1/2}$ and $S_{YY}^{-1/2}$.

This representation makes the diagnostics concrete. The eigenvalues of $S_{XX}$ and $S_{YY}$ control the inverse square roots used to form $K$ and the coefficients. The gaps between singular values of $K$, or equivalently between canonical roots, help control how much the associated singular vectors can rotate. In these terms, a canonical root is the squared canonical correlation, while the singular value itself is the canonical correlation.

Covariance Eigenvalues and Whitening Sensitivity

Whitening rescales each covariance direction to unit variance. If $S_{XX}$ has a small eigenvalue, then the inverse square root has a large eigenvalue in that direction. A small change in a covariance block can therefore have a larger effect after whitening than it had on the original scale. Near-singularity also makes the original-variable coefficients sensitive to changes in the estimated covariance eigenvectors and eigenvalues.

A useful first check is to inspect the smallest and largest eigenvalues of each within-set covariance matrix, along with their ratio (the spectral condition number when the matrix is positive definite). A small minimum eigenvalue or a very large ratio is a warning, not a verdict: the relevant question is whether the estimated directions of interest depend on those weakly measured combinations of variables.

Worked Example: A Small Covariance Eigenvalue Amplifies a Change

Suppose $$ S_{XX}=\begin{pmatrix}1&0\\0&0.01\end{pmatrix},\qquad S_{YY}=I_2,\qquad S_{XY}=\begin{pmatrix}0.8&0\\0&0.02\end{pmatrix}. $$ The eigenvalues of $S_{XX}$ are $1$ and $0.01$, so its inverse square root is $\operatorname{diag}(1,10)$. Thus $$ K=S_{XX}^{-1/2}S_{XY}S_{YY}^{-1/2} =\begin{pmatrix}0.8&0\\0&0.2\end{pmatrix}. $$ The canonical correlations are $0.8$ and $0.2$.

Now increase the second diagonal entry of $S_{XY}$ by $0.001$, leaving the within-set covariance matrices unchanged. The corresponding entry of $K$ increases by $10(0.001)=0.01$, from $0.2$ to $0.21$. The change in that cross-covariance entry was only $0.001$, but whitening multiplied it by ten. This does not by itself show the resulting direction is unusable; it shows why small covariance eigenvalues deserve attention when interpreting sensitivity.

The matrices are compatible with a positive definite joint covariance: the singular values of $K$ are below one, both before and after the change. With identity $S_{YY}$, positive definiteness follows because the Schur complement $S_{XX}-S_{XY}S_{YY}^{-1}S_{YX}$ is positive definite exactly when $I-KK^{T}$ is positive definite. Its eigenvalues are $1-0.8^2$ and $1-0.21^2$ after the change, both positive.

Root Gaps and Directional Stability

A canonical correlation may be estimated accurately even when its associated direction is not. If two leading canonical correlations are close, a small perturbation can rotate the corresponding directions substantially while changing the two correlations only slightly. When roots are equal, the individual directions within their shared subspace are not uniquely identified: different orthonormal bases span the same canonical subspace.

Theorem: Let $A$ be a real symmetric matrix with a simple eigenvalue $\lambda$, unit eigenvector $u$, and separation $$ \delta=\min_{\mu\in\operatorname{spec}(A),\,\mu\ne\lambda}|\lambda-\mu|>0. $$ Let $\widetilde A=A+E$, where $E$ is symmetric, and let $\widetilde\lambda$ and unit eigenvector $v$ correspond to the perturbed eigenvalue matched to $\lambda$. If $\|E\|_2<\delta$, then $$ \sin\angle(u,v)\le \frac{\|E\|_2}{\delta-\|E\|_2}. $$

Here $\|E\|_2$ is the spectral norm, and the angle between unit vectors is taken without regard to their arbitrary signs. The bound shows that a small perturbation is most consequential when the eigenvalue separation is small.

Proof. By the eigenvalue perturbation bound for symmetric matrices, the matched eigenvalue satisfies $|\widetilde\lambda-\lambda|\le\|E\|_2$. Let $P$ be orthogonal projection onto the subspace perpendicular to $u$. Since $Au=\lambda u$, the restriction of $A-\widetilde\lambda I$ to that perpendicular subspace has eigenvalues $\mu-\widetilde\lambda$, where $\mu\ne\lambda$ ranges over the other eigenvalues of $A$. Each has absolute value at least $$ |\mu-\lambda|-|\widetilde\lambda-\lambda| \ge \delta-\|E\|_2>0. $$ The equation $(A+E)v=\widetilde\lambda v$, projected onto that subspace, gives $$ P(A-\widetilde\lambda I)Pv=-PEv, $$ because $PAv=P(A-\widetilde\lambda I)Pv+\widetilde\lambda Pv$ and projection removes the component along $u$. The inverse of the restricted operator has norm at most $1/(\delta-\|E\|_2)$. Therefore $$ \|Pv\|_2\le \frac{\|E\|_2\|v\|_2}{\delta-\|E\|_2} =\frac{\|E\|_2}{\delta-\|E\|_2}. $$ Since $\|Pv\|_2=\sin\angle(u,v)$, the claimed bound follows. $\square$

For CCA, singular vectors can be studied through the symmetric matrix $$ H=\begin{pmatrix}0&K\\K^T&0\end{pmatrix}. $$ Its nonzero eigenvalues are the positive and negative singular values of $K$, and its eigenvectors combine the corresponding left and right singular vectors. Thus a small separation between neighboring canonical correlations signals potential directional instability. The theorem is a diagnostic principle, not a complete finite-sample guarantee: perturbing the original covariance blocks also changes the whitening matrices that define $K$.

Worked Example: A Small Perturbation Rotates Close Directions

Consider a whitened cross-covariance matrix $$ K=\begin{pmatrix}0.9&0\\0&0.894427\end{pmatrix}. $$ Its singular values are $0.9$ and $0.894427$, separated by only $0.005573$. Perturb it symmetrically to $$ \widetilde K=\begin{pmatrix}0.9&0.002\\0.002&0.894427\end{pmatrix}. $$ The perturbation has spectral norm $0.002$. For a symmetric $2$ by $2$ matrix with diagonal difference $a-d$ and off-diagonal entry $b$, the rotation angle $\theta$ of the leading eigenvector satisfies $\tan(2\theta)=2b/(a-d)$. Here, $$ \tan(2\theta)=\frac{0.004}{0.9-0.894427} =\frac{0.004}{0.005573}\approx0.717746. $$ Consequently $2\theta\approx35.67^\circ$, so $\theta\approx17.83^\circ$. A perturbation of norm $0.002$ has rotated the leading direction by nearly eighteen degrees because the original roots were close.

The perturbed singular values are the eigenvalues of this positive definite symmetric matrix: $$ \frac{0.9+0.894427}{2} \ \pm\ \sqrt{\left(\frac{0.9-0.894427}{2}\right)^2+0.002^2}. $$ They are approximately $0.900643$ and $0.893784$, both below one. Hence this perturbed cross-covariance remains compatible with a positive definite joint covariance having identity diagonal blocks. The example isolates directional sensitivity rather than relying on an invalid covariance construction.

Coefficient Magnitudes: A Warning, Not a Verdict

Large canonical weights are often reported as evidence of instability, but their magnitudes depend on variable scaling. A change of units can make a coefficient numerically small or large while leaving the canonical variate and canonical correlation unchanged. Compare weights only after specifying the variable scales and normalization. Structure loadings and cross-loadings, introduced in the earlier material on interpreting canonical results, can provide complementary descriptions, but they do not eliminate the need to assess stability.

Worked Example: A Large Weight Can Be a Scale Effect

Use the first example's second canonical direction for the $X$ variables. Its unit-variance coefficient vector is $a=(0,10)^T$, since $a^TS_{XX}a=10^2(0.01)=1$. The canonical variate is $10X_2$.

Now record the second variable in units 100 times larger: let $X_2^*=100X_2$. Its variance is $100^2(0.01)=100$, and the same canonical variate is $0.1X_2^*$. The coefficient is now $0.1$, not $10$, even though the score is exactly the same. Therefore the raw weight magnitude alone cannot establish instability.

A more informative warning occurs when large weights accompany a small eigenvalue of $S_{XX}$, change sharply under small covariance perturbations, or depend on near-canceling combinations of variables. Report the scaling and normalization, and check whether the substantive pattern is stable rather than treating a numerical threshold for weights as universal.

Practical Perturbation Checks

A perturbation check asks whether the scientific interpretation survives plausible changes in the estimated covariance blocks. One can recompute CCA after resampling observations, deleting a small set of influential cases, or applying small changes calibrated to the sampling variability of the covariance estimates. An arbitrary perturbation is useful for illustrating sensitivity but should not be presented as a probability statement about the estimator.

Record the baseline.
Save the eigenvalues of $S_{XX}$ and $S_{YY}$, canonical correlations, root gaps, normalized coefficient vectors, and the directions' substantive interpretation.

Choose plausible changes.
Use resampling or covariance perturbations reflecting the data and its scale. Ensure each perturbed joint covariance remains admissible; do not interpret an invalid covariance matrix as a meaningful CCA result.

Recompute the full analysis.
Recalculate the within-set inverse square roots, whitened cross-covariance, roots, and directions. Perturbing $K$ alone does not capture changes in the within-set covariance blocks.

Compare matched directions.
Align signs before comparing vectors, since changing both signs of a canonical pair leaves its correlation unchanged. For close roots, compare the span of the directions as well as individual vectors.

Summarize what persists.
Separate stable features, such as a similar canonical correlation or subspace, from unstable details, such as which of two near-tied directions receives a particular label.

Common Pitfall and Interpretation

Wrong reasoning: “The first canonical correlation is large and statistically distinguishable from zero, so its coefficients must be reliable.” A test about the strength of association does not by itself establish that the estimated direction is stable. A large root can be close to the next root, and its coefficients can be affected by weak covariance directions.

Correction: examine root separation and covariance conditioning alongside the root itself, then assess sensitivity under plausible perturbations. It is possible for the leading canonical correlation to remain nearly unchanged while the associated weights rotate or exchange roles with a neighboring direction. In that case, report the robust association or joint subspace, and avoid overinterpreting the individual coefficient pattern.

This diagnostic work is especially useful when the number of variables is substantial relative to the sample size, when predictors are strongly correlated, or when the interpretation turns on a small subset of weights. It does not replace inferential analysis. It clarifies which estimated directions are adequately determined by the observed covariance structure and which conclusions require cautious wording.

Key takeaway: Diagnose canonical directions using several signals together: small within-set covariance eigenvalues, narrow gaps between canonical roots, scale-aware coefficient patterns, and changes under realistic perturbations. A stable root does not guarantee a stable direction.

Check Your Understanding

Use the diagnostics in this tutorial to answer the following questions.

Why can a small eigenvalue of $S_{XX}$ make the whitened cross-covariance sensitive to a small change in $S_{XY}$?
Two leading canonical correlations are nearly equal. What can change substantially even if both correlations change very little?
Why is comparing raw coefficient magnitudes across different variable units potentially misleading?
In a perturbation check, why should the inverse square roots and whitened cross-covariance be recomputed after changing a covariance block?
If two directions rotate under perturbation but their combined span is similar, what is a more defensible interpretation than treating each direction as individually stable?