Asymptotic Uncertainty of Canonical Correlations

Why Asymptotic Uncertainty Has Two Parts

Canonical correlation analysis estimates both the strengths of association between two variable sets and the directions that define those associations. These are different inferential targets. A canonical correlation is a scalar, while its canonical directions are vectors. A scalar root can be estimated fairly precisely even when small changes in the data produce substantial rotations of its associated directions.

The distinction is governed in part by separation. If one population canonical root is distinct from its neighbors, small changes in the estimated covariance matrices generally produce smooth changes in that root and its direction. When two roots are close, their directions can rotate markedly in response to small perturbations. This tutorial develops the large-sample conditions for root inference and derives the eigenvalue-gap mechanism behind direction uncertainty.

Population Roots and Regularity Conditions

Let \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) have means \(\mu_X,\mu_Y\), and let their joint covariance matrix be partitioned as

Assume the marginal covariance matrices \(\Sigma_{XX}\) and \(\Sigma_{YY}\) are positive definite. The whitened cross-covariance matrix is

The population canonical correlations \(\rho_1\geq\cdots\geq\rho_m\geq0\), where \(m=\min(p,q)\), are the singular values of \(K\). Equivalently, their squared values are eigenvalues of \(KK^{T}\), with zero eigenvalues included as needed. Sample canonical correlations \(\hat\rho_j\) are obtained by replacing the population covariance matrices with sample covariance matrices.

For standard first-order asymptotic inference, use the following conditions: the dimensions \(p\) and \(q\) are fixed as the sample size \(n\) increases; the observations \((X_i,Y_i)\) are independent and identically distributed; their joint distribution has finite fourth moments; both marginal covariance matrices are positive definite; and the target root is simple and separated. The finite-fourth-moment condition supplies a central limit theorem for the sample covariance entries. Joint normality is not required for this general result.

The root separation condition is essential, not cosmetic. A repeated root does not identify a unique direction within its eigenspace, and the ordinary derivative used for a single root or vector need not exist there. When two roots are distinct but close, the derivative exists but can be large, which is the source of increased direction uncertainty.

Large-Sample Inference for a Separated Root

Write \(\theta=\operatorname{vech}(\Sigma)\), the vector containing the distinct entries of the joint covariance matrix, and let \(\hat\theta=\operatorname{vech}(\hat\Sigma)\). Define \(g_j(\theta)\) as the \(j\)th ordered canonical correlation calculated from the covariance matrix represented by \(\theta\). At a covariance matrix satisfying the conditions above, \(g_j\) is differentiable. Let \(\nabla g_j(\theta)\) denote its gradient with respect to the entries in \(\theta\), and define

This is the asymptotic covariance matrix for \(\sqrt n(\hat\theta-\theta)\), including when the means are estimated and the usual centered sample covariance is used.

Proof. The finite fourth moments imply a multivariate central limit theorem for the distinct entries of the sample covariance matrix:

Positive definiteness of the marginal covariance matrices makes the whitening operations well-defined in a neighborhood of the population covariance. A simple positive eigenvalue varies differentiably with the matrix entries; taking its positive square root is also differentiable because \(\rho_j>0\). Thus \(g_j\) is differentiable at \(\theta\). A first-order Taylor expansion gives

The covariance central limit theorem implies \(\hat\theta-\theta=O_p(n^{-1/2})\), so the Taylor remainder, after multiplication by \(\sqrt n\), converges in probability to zero. Applying the continuous mapping theorem to the linear term yields the stated normal limit, with variance \(\nabla g_j(\theta)^T V\nabla g_j(\theta)\). \(\square\)

In practice, the unknown covariance and gradient are replaced by consistent estimates, giving an estimated standard error of the form

A first-order approximate confidence interval is \(\hat\rho_j\pm z_{1-\alpha/2}\widehat{\operatorname{se}}(\hat\rho_j)\), where \(z_{1-\alpha/2}\) is the standard normal quantile. This approximation is justified for an interior, separated root as sample size grows; a finite-sample interval that extends beyond the allowable range from zero to one should not be interpreted as a literal possible value of a canonical correlation.

Why Close Roots Make Directions Uncertain

The connection between root separation and direction uncertainty can be seen through a symmetric eigenvalue problem. Let \(A\) be a symmetric matrix with distinct eigenvalues \(\lambda_1,\ldots,\lambda_r\) and corresponding orthonormal eigenvectors \(u_1,\ldots,u_r\). Suppose \(A\) changes to \(A+dA\), where \(dA\) is a small symmetric perturbation. For a simple eigenvalue \(\lambda_j\), the first-order change in its normalized eigenvector is

Derivation. Differentiate \(Au_j=\lambda_j u_j\) to obtain

Because \(u_j^Tu_j=1\), differentiating the normalization gives \(u_j^Tdu_j=0\). Expand \(du_j\) in the other eigenvectors, writing \(du_j=\sum_{k\ne j}c_k u_k\). Premultiplying the differentiated eigenvalue equation by \(u_k^T\) eliminates the term involving \(d\lambda_j\), since \(u_k^Tu_j=0\). Using \(u_k^TA=\lambda_k u_k^T\) gives

Substitution of these coefficients gives the expression for \(du_j\). Its denominators show why a small eigenvalue gap can cause a large first-order change. For CCA, the squared canonical correlations are eigenvalues of \(KK^T\); the corresponding left singular vectors describe directions in the whitened \(X\)-space. The right singular vectors have the same gap sensitivity through \(K^TK\). Canonical coefficient vectors in the original coordinates also include covariance-based transformations, but the separation of the roots remains a key source of sensitivity. \(\square\)

The formula is local: it describes the first-order response near a simple eigenvalue. It does not say that every close pair must produce a large observed change, because the numerator \(u_k^T(dA)u_j\) may be small. It does show that, for perturbations with a component in the relevant direction, smaller gaps permit larger rotations. Sign changes of an eigenvector are a separate representational ambiguity; directions should be compared after choosing consistent signs.

Interpreting the Uncertainty

Root uncertainty and vector uncertainty should be reported and interpreted separately. The asymptotic standard error for \(\hat\rho_j\) measures variability in the strength of the \(j\)th association. The direction formula instead describes how perturbations can rotate the associated axes. A useful qualitative warning is that the direction becomes less reliably determined as its squared-root gap to a neighboring root shrinks.

This matters when canonical directions are used to describe which combinations of variables define an association. If two population roots are nearly equal, the estimated two-dimensional subspace may be more stable than either individual direction: small rotations within that subspace can change the individual vectors without changing the subspace much. Thus, interpreting one vector in isolation can overstate what the data determine. This is not a replacement for a formal uncertainty analysis, but it is an important consequence of the eigenvalue structure.

Common Pitfall

Wrong reasoning: “The canonical correlation has a small standard error, so its canonical direction must also be precisely estimated.” This confuses a scalar estimate with an eigenvector estimate. The root can be stable while its direction is sensitive, particularly when a neighboring root is close.

Correction: Check the relevant root gaps when interpreting directions. The first-order eigenvector change contains denominators \(\lambda_j-\lambda_k\); a small gap can magnify perturbations even if the estimated root itself changes little. Also, do not use the ordinary separated-root normal approximation at a repeated root: the uniqueness and differentiability conditions required for that approximation fail.