Resampling Canonical Correlations and Singular Roots

Why Resample Canonical Roots?

Canonical correlation analysis summarizes the association between two multivariate measurements observed on the same units. Its roots describe the strengths of successive linear associations; its canonical directions describe which combinations of the measured variables realize those strengths. Resampling can quantify uncertainty in both, but the two kinds of uncertainty behave differently. A well-separated root can have a stable direction, while directions attached to nearly tied roots can vary substantially even when their joint subspace is stable.

The central practical distinction is between an association measured by a root and a coordinate vector used to express that association. Roots are unchanged by nonsingular linear changes of measurement coordinates. Weight vectors, by contrast, must change with the coordinates, and their numerical entries can be sensitive to scaling and near-ties.

Roots, Directions, and the Paired Bootstrap

Suppose \(X\in\mathbb{R}^p\) and \(Y\in\mathbb{R}^q\) are paired random vectors with finite second moments. Write their covariance matrices as \(\Sigma_{XX}\), \(\Sigma_{YY}\), and \(\Sigma_{XY}\). Assume initially that both within-set covariance matrices are positive definite. The whitened cross-covariance matrix is

The canonical correlations \(\rho_1\geq\cdots\geq\rho_m\geq0\), where \(m=\min(p,q)\), are the singular values of \(R\). The corresponding canonical roots or squared canonical correlations are \(\lambda_j=\rho_j^2\). Equivalently, the nonzero roots are eigenvalues of the symmetric matrix \(A=RR^\mathsf{T}\) (or of \(R^\mathsf{T}R\), which has the same nonzero eigenvalues). A unit eigenvector \(u_j\) of \(A\) gives the \(X\)-side direction \(a_j=\Sigma_{XX}^{-1/2}u_j\); the corresponding \(Y\)-side direction is obtained from the right singular vector of \(R\). These directions are conventionally scaled so their canonical variates have variance one.

For paired observations \((X_i,Y_i)\), \(i=1,\ldots,n\), form the sample covariance matrices \(S_{XX},S_{YY},S_{XY}\) using the same observations and centering convention. The sample roots are the eigenvalues of

For an ordinary nonparametric bootstrap under independent sampling, draw \(n\) paired rows with replacement. Recompute all covariance matrices, the roots, and any directions or subspaces in each replicate. Resampling the \(X\)-rows and \(Y\)-rows separately destroys their pairing and therefore does not estimate uncertainty for the original cross-association. Other designs require resampling their appropriate units, as in the earlier material on choosing resampling units.

Worked Example: Calculating Roots and Summarizing Replicates

What Root Separation Says About Direction Uncertainty

If a root is simple and separated from the others, its eigenvector and its one-dimensional subspace change smoothly under small perturbations of \(\widehat A\). If roots are close, the corresponding individual directions may rotate considerably under perturbations that barely change the roots. When roots coincide, no particular basis within their joint eigenspace is distinguished by the population matrix. In that situation, inference should focus on the whole subspace rather than assigning separate meanings to arbitrary basis vectors.

Here the operator norm is \(\|E\|_2=\sup_{\|x\|_2=1}\|Ex\|_2\), and \(\angle\) is the acute angle between the one-dimensional subspaces. To verify the eigenvalue bound, the Rayleigh quotient satisfies \(|x^\mathsf{T}Ex|\leq\epsilon\) for every unit \(x\); the variational characterization of ordered eigenvalues therefore gives \(|\tilde\lambda_j-\lambda_j|\leq\epsilon\). For the direction bound, expand \(\tilde u_j\) in an orthonormal eigenbasis \(u_k\) of \(A\). Since \((A+E)\tilde u_j=\tilde\lambda_j\tilde u_j\), projection onto the orthogonal complement of \(u_j\) gives, in the eigenbasis coordinates,

A sharper norm form follows by applying the inverse of \(A-\tilde\lambda_j I\) on that complement: its smallest absolute eigenvalue is at least \(g-\epsilon\), because \(|\tilde\lambda_j-\lambda_j|\leq\epsilon\). Thus \(\|(I-u_ju_j^\mathsf{T})\tilde u_j\|_2\leq\epsilon/(g-\epsilon)\). The left side is the sine of the angle, proving the result. \(\square\)

For a cluster of roots, the corresponding object is the projector onto the joint eigenspace, not a separately labeled vector for each root. The same perturbation principle applies: the cluster subspace is stable when the cluster is separated from the rest of the spectrum, while the basis chosen inside the cluster may rotate freely. In bootstrap output, this means that aligned projectors or principal angles are often more meaningful than coordinate-by-coordinate comparisons of canonical vectors.

Bootstrap Approximation for Separated Roots

A standard regular setting clarifies when the ordinary paired bootstrap is appropriate. Assume independent, identically distributed paired observations, fixed \(p\) and \(q\), finite fourth moments, positive definite population within-set covariance matrices, and simple positive population roots separated from their neighbors. Assume also that the relevant sample covariance matrices are nonsingular with probability tending to one. Under these conditions, the root is a smooth function of the vector of covariance entries in a neighborhood of the population covariance matrices.

Here consistency of the conditional distribution means convergence in probability of its distribution function to the target limiting distribution at continuity points. The justification is the bootstrap delta method. The covariance entries form a finite-dimensional vector of sample averages of functions of \((X,Y)\); finite fourth moments give finite variances for these products. The ordinary empirical bootstrap consistently estimates the centered sampling limit for this vector. Near positive definite covariance matrices, matrix inversion and the positive-definite inverse square root are differentiable functions of their entries. A simple eigenvalue is differentiable as well: if \(A\) has unit eigenvector \(u_j\), its first-order change under \(H\) is \(u_j^\mathsf{T}Hu_j\). Composing these differentiable maps gives a first-order expansion with a remainder negligible at the \(n^{-1/2}\) scale. Applying that expansion to both the data and the bootstrap covariance vectors yields the stated conditional approximation. A direction is locally differentiable after fixing its sign; its projector \(u_ju_j^\mathsf{T}\) needs no sign choice. \(\square\)

This result is not a blanket guarantee for every root configuration. A zero or repeated root is not a regular, individually labeled smooth parameter. A root near a boundary, such as one, can also have a nonstandard sampling distribution. In such cases, ordinary percentile intervals for separately ordered roots may have poor coverage, even when bootstrap computations are numerically successful. Inspect root gaps and the replicate distribution; where an isolated direction is not supported, report uncertainty for the corresponding subspace instead.

Worked Example: Roots Are Invariant, Weights Are Not

Reading Bootstrap Output Carefully

For each paired replicate, keep the roots in a consistent decreasing order, but do not interpret that ordering as proof that a particular direction is consistently identified. A useful report includes root estimates and uncertainty summaries, the relevant observed gaps, and—when directions matter—bootstrap variation in a sign-invariant projector or in principal angles between subspaces. If the leading roots repeatedly approach or exchange order, that is evidence against interpreting their individual weight vectors as stable scientific findings.

• Roots: summarize the bootstrap distribution of \(\widehat\lambda_j^*\), or transform replicate roots to correlations if those are the target.
• Directions: account for arbitrary sign changes before comparing isolated vectors; use a projector to avoid the sign ambiguity.
• Clusters: select a root cluster based on the scientific question and its separation from the remaining roots, then compare the cluster subspaces.
• Coordinates: report variable scaling and weight normalization. Coefficient magnitudes across differently scaled variables are not directly comparable.

A common mistake is to treat a dramatic bootstrap spread in individual weights as evidence that the canonical association itself is equally uncertain. Near-tied roots can make the basis rotate while leaving their joint subspace and root values relatively stable. The correction is to diagnose the root gaps and examine the appropriate subspace. The converse mistake is to read a narrow root interval as proof that the associated weights are stable: root precision and direction precision are distinct.