Why Compute CCA with QR and SVD?
Canonical correlation analysis (CCA) finds the largest correlations between linear combinations of two measured data blocks. A direct calculation can form within-block covariance matrices, invert or whiten them, and then solve an eigenvalue problem. That route is mathematically valid under suitable rank conditions, but forming cross-products can worsen numerical conditioning. A QR factorization followed by a singular value decomposition (SVD) gives the same canonical correlations while avoiding explicit covariance-matrix inverses.
The calculation assumes that the two data blocks contain measurements on the same \(n\) observations in the same row order. Each block is centered column by column before analysis. The basic QR-SVD procedure below assumes that both centered blocks have full column rank: if \(X\) has \(p\) columns and \(Y\) has \(q\) columns, then \(p\le n-1\), \(q\le n-1\), and neither block has exact linear dependence among its columns. The method is a computation, not an inferential test; independence or a probability model is not needed to calculate its singular values.
The QR-SVD Computation
A thin QR factorization represents each centered data block using an orthonormal basis for its column space and a smaller triangular factor. In exact arithmetic, \(Q_X^\mathsf{T}Q_X=I_p\) and \(Q_Y^\mathsf{T}Q_Y=I_q\). Form the \(p\times q\) matrix \(C=Q_X^\mathsf{T}Q_Y\), then compute its SVD:
The \(\rho_j\) are the canonical correlations. This result follows from the principal-angle interpretation of CCA: the columns of \(Q_X\) and \(Q_Y\) are orthonormal bases for the two centered column spaces, and the singular values of their cross-product are the cosines of the principal angles between those spaces. Since cosines of angles lie between zero and one, each canonical correlation lies in \([0,1]\) in exact arithmetic. A computed value a little above one can occur from rounding, but a substantial excess suggests a coding, scaling, or factorization problem.
The SVD also supplies directions \(u_j\) and \(v_j\) in the two orthonormal coordinate systems. When coefficients in the original measured variables are needed, solve \(R_Xa_j=u_j\) and \(R_Yb_j=v_j\) by triangular back-substitution. This is preferable to explicitly calculating \(R_X^{-1}\) or \(R_Y^{-1}\). The main subject here is the computation of the correlations; interpretation of the corresponding coefficients and scores requires additional care.
Why the QR-SVD Result Matches Covariance CCA
For centered data, the sample covariance matrices are \(S_{XX}=X^\mathsf{T}X/(n-1)\), \(S_{YY}=Y^\mathsf{T}Y/(n-1)\), and \(S_{XY}=X^\mathsf{T}Y/(n-1)\). The covariance-whitening calculation takes the singular values of \(S_{XX}^{-1/2}S_{XY}S_{YY}^{-1/2}\). Under full column rank, these are defined. Substituting \(X=Q_XR_X\) and \(Y=Q_YR_Y\) gives
The factors \(n-1\) cancel in the whitened matrix. To see why its singular values match those of \(C\), define \(L_X=R_X(R_X^\mathsf{T}R_X)^{-1/2}\) and \(L_Y=R_Y(R_Y^\mathsf{T}R_Y)^{-1/2}\). Both are orthogonal matrices: for example, \(L_X^\mathsf{T}L_X=I_p\). The whitened matrix is \(L_X^\mathsf{T}CL_Y\). Multiplication on either side by an orthogonal matrix preserves singular values, so its singular values equal those of \(C\). This establishes the equivalence in exact arithmetic.
The practical advantage is numerical. Computing \(X^\mathsf{T}X\) squares the two-norm condition number: \(\kappa_2(X^\mathsf{T}X)=\kappa_2(X)^2\) when \(X\) has full column rank. Thus, if a data block has directions at very different scales or nearly dependent columns, its covariance matrix can be substantially harder to work with than the original data matrix. QR factorization and SVD are standard stable factorizations that avoid explicitly squaring the conditioning in this way.
Worked Example: A One-Variable Calculation Two Ways
Suppose four hypothetical observations have centered measurements \(x=(1,-1,1,-1)^\mathsf{T}\) and \(y=(2,0,0,-2)^\mathsf{T}\). Each vector is centered because its entries sum to zero. With one variable in each block, both have full column rank.
QR-SVD. The thin QR factors have \(Q_X=x/\lVert x\rVert\) and \(Q_Y=y/\lVert y\rVert\). Here \(\lVert x\rVert=2\), \(\lVert y\rVert=\sqrt{8}\), and \(x^\mathsf{T}y=4\). The \(1\times1\) cross-product and its singular value are therefore
Direct covariance calculation. Using \(n-1=3\), \(S_{XX}=4/3\), \(S_{YY}=8/3\), and \(S_{XY}=4/3\). The covariance-whitened value is
Both calculations give a sample canonical correlation of approximately 0.7071. In this one-variable setting it is simply the absolute sample correlation; the positive sign of the ordinary correlation is not retained by the singular value.
Worked Example: Two Blocks with Two Canonical Correlations
Consider five hypothetical observations and four centered orthonormal vectors: \(e_1=(1,-1,0,0,0)^\mathsf{T}/\sqrt{2}\), \(e_2=(1,1,-2,0,0)^\mathsf{T}/\sqrt{6}\), \(e_3=(1,1,1,-3,0)^\mathsf{T}/\sqrt{12}\), and \(e_4=(1,1,1,1,-4)^\mathsf{T}/\sqrt{20}\). Each vector sums to zero, has length one, and is orthogonal to the other three. Define the hypothetical centered blocks by \(X=[e_1\ e_2]\) and \(Y=[0.8e_1+0.6e_3\quad 0.3e_2+\sqrt{0.91}e_4]\). Both \(Y\) columns have length one, are centered, and are orthogonal; both blocks have full column rank.
Here the QR factors are \(Q_X=X\) and \(Q_Y=Y\), since their columns are already orthonormal. Taking their cross-product gives
The SVD of this diagonal matrix has singular values 0.8 and 0.3. Thus the two sample canonical correlations are 0.8000 and 0.3000.
For comparison, \(X^\mathsf{T}X=Y^\mathsf{T}Y=I_2\), so \(S_{XX}=S_{YY}=I_2/4\). The cross-covariance is \(S_{XY}=\operatorname{diag}(0.8,0.3)/4\). Whitening multiplies this cross-covariance by \(2I_2\) on each side, yielding \(\operatorname{diag}(0.8,0.3)\) again. The covariance method therefore produces the same two singular values. The example shows both the arithmetic and the equivalence; its orthonormal columns make it unusually simple, not a requirement of the method.
Diagnostics and Failure Modes
A stable factorization does not repair a data-design problem. Before calculating, verify that the rows of the two blocks refer to the same observations in the same order, that missing-value handling has not created different row sets, and that centering was performed consistently. A mismatch in row alignment changes the cross-block associations while potentially leaving each block’s individual covariance structure plausible.
Check the numerical rank of each centered block. In floating-point computation, software often treats a diagonal element of \(R_X\) or \(R_Y\) as negligible relative to a tolerance and the scale of the data. A very small diagonal element signals a weakly represented direction; results involving that direction may be sensitive to rounding. The full-rank QR-SVD calculation presumes that all columns are retained. Do not silently treat an effectively rank-deficient block as full rank: the appropriate treatment of singular covariance matrices is a separate problem.
Other useful checks are that the singular values are nonnegative and nonincreasing, and that none is materially greater than one. Small differences between the QR-SVD and covariance calculations are expected in floating-point arithmetic, particularly for ill-conditioned blocks. Large differences warrant checking the centering, use of the same observations, matrix dimensions, and whether a covariance inverse or square root was computed accurately.
Worked Example: Near-Collinearity and Conditioning
Let \(u=(1,-1,0,0)^\mathsf{T}/\sqrt{2}\) and \(v=(1,1,-1,-1)^\mathsf{T}/2\). They are centered, orthogonal, and each has length one. Consider a hypothetical block with columns \(u\) and \(u+0.01v\). Both columns are centered, and they are not exactly dependent, so the block has full rank.
The matrix of column inner products is
Its eigenvalues are approximately \(2.00005\) and \(0.000049999\), so its two-norm condition number is approximately \(2.00005/0.000049999\approx 4.00\times10^4\). The condition number of \(X\) itself is about \(\sqrt{4.00\times10^4}=200\). The calculation illustrates why forming \(X^\mathsf{T}X\) can magnify conditioning problems. A QR factorization works with \(X\) directly, but its small diagonal factor still alerts the analyst that the second direction is weakly distinguished. The correct conclusion is not that QR removes near-collinearity; it is that QR-SVD avoids unnecessarily worsening the numerical problem by forming cross-products.
Choosing Between QR-SVD and Covariance Calculations
| Approach | What is decomposed | Practical consideration |
|---|---|---|
| QR followed by SVD | \(Q_X^\mathsf{T}Q_Y\) | Works directly with the centered data and uses stable factorizations; generally preferable when conditioning is a concern. |
| Covariance whitening | \(S_{XX}^{-1/2}S_{XY}S_{YY}^{-1/2}\) | Gives the same canonical correlations when the covariance matrices are nonsingular; forming cross-products can worsen conditioning. |
| Generalized eigenvalue calculation | A covariance-based eigenvalue problem | Can be convenient in some software, but still relies on covariance matrices and appropriate rank and numerical handling. |
The methods are equivalent in exact arithmetic under the full-rank conditions stated above. Covariance calculations can be useful for checking a small, well-conditioned example or when covariance matrices are already the quantities of interest. QR-SVD is a sound default when starting from the data matrices. Neither approach resolves exact rank deficiency, chooses a regularization strategy, or determines whether a sample canonical correlation is statistically significant.
Common Mistakes
- Using \(X^\mathsf{T}Y\) in place of \(Q_X^\mathsf{T}Q_Y\). The former depends on the measurement scales and is not the QR-SVD calculation. Factor each centered block first.
- Comparing covariance calculations that use different conventions. Use the same observations and centering for all three covariance matrices. The common divisor \(n-1\) cancels from whitening, but inconsistent preprocessing does not.
- Assuming a stable algorithm cures poor data geometry. Nearly dependent columns remain a substantive and numerical difficulty; inspect rank and conditioning.
- Interpreting every computed singular value as a reliable population relationship. This calculation describes sample canonical correlations. Inference and validation require additional methods and assumptions.
Check Your Understanding
Try each question, then compare your reasoning with the solutions.
- Why must the rows of \(X\) and \(Y\) describe the same observations in the same order?
- If \(X\) has \(p=5\) columns and \(Y\) has \(q=3\) columns, how many canonical correlations are returned by the thin cross-product SVD?
- For centered, full-rank data, explain why \(S_{XX}^{-1/2}S_{XY}S_{YY}^{-1/2}\) and \(Q_X^\mathsf{T}Q_Y\) have the same singular values.
- If a computed singular value is 1.0000002, what should you check before treating it as a problem?
Solutions. (1) Each row contributes a paired observation to the cross-block calculation; reordering one block changes which measurements are paired and therefore changes \(Q_X^\mathsf{T}Q_Y\). (2) The SVD returns \(\min(5,3)=3\) singular values. (3) With \(X=Q_XR_X\) and \(Y=Q_YR_Y\), the whitened matrix is \(L_X^\mathsf{T}(Q_X^\mathsf{T}Q_Y)L_Y\), where \(L_X\) and \(L_Y\) are orthogonal; orthogonal multiplication preserves singular values. (4) A tiny excess above one can be rounding error. Check row alignment, centering, factorization, and numerical precision; a substantial excess is not explained by ordinary rounding and indicates a likely computational problem.