Sampling Bias and Overfitting in Sample CCA

Why Sample CCA Can Look Too Strong

Canonical correlation analysis (CCA) chooses linear combinations of two variable sets to maximize their correlation. That optimization is useful for finding shared patterns, but it also creates a risk: the same observations are used both to choose the combinations and to assess their correlation. With many variables and few observations, CCA has considerable freedom to fit accidental sample patterns.

The resulting sample canonical correlation measures the strongest association available in the observed data. It need not be a reliable estimate of the strongest population association. In particular, if the two variable sets are unrelated in the population, their sample canonical correlation can still be substantial—and under a dimension condition it is exactly one.

Definition: Let $X$ and $Y$ denote two population random vectors with positive-definite covariance matrices. Their first population canonical correlation is the maximum correlation between $a^\mathsf{T}X$ and $b^\mathsf{T}Y$ over nonzero coefficient vectors $a$ and $b$. For $n$ paired observations, the first sample canonical correlation, $r_1$, is the maximum sample correlation between the corresponding linear combinations of the observed $X$-variables and $Y$-variables, when both sample covariance matrices are nonsingular.

There is a useful geometric description of this sample optimization. Center each data column by subtracting its sample mean. Regard the resulting $n$-row columns as vectors in the subspace of $\mathbb{R}^n$ whose coordinates sum to zero. The sample canonical correlation is the largest correlation between any vector in the span of the centered $X$-columns and any vector in the span of the centered $Y$-columns. Thus it depends not just on the underlying association, but also on how much freedom the two observed column spaces have to align.

If the population cross-covariance is zero, the population canonical correlations are zero, provided both within-set covariance matrices are positive definite. But this does not require the two sample column spaces to be perpendicular. In fact, maximizing over all linear combinations searches those spaces for their most aligned pair of directions.

A Dimension Result: When the Sample Fit Is Perfect

The centering constraint means the observations live in a space of dimension $d=n-1$, not $n$. Suppose there are $p$ variables in the first set and $q$ in the second. Under independent standard Gaussian sampling, the centered $X$-columns have rank $p$ and the centered $Y$-columns have rank $q$, with probability one, as long as $p,q\leq d$. Their spans are in general position in the $d$-dimensional centered space.

Theorem: Suppose the $n$ paired observations are independent draws from a population in which the two variable sets are independent multivariate standard normal vectors, with $p,q\leq n-1$. If $p+q>n-1$, then the first sample canonical correlation equals one with probability one.

The independence assumption makes every population canonical correlation zero. The theorem therefore identifies a setting in which a perfect sample canonical correlation is purely a consequence of the dimensions and the optimization, not evidence of population association.

Work in the centered observation space.
Choose an orthonormal basis for the $d=n-1$ dimensional space of vectors whose entries sum to zero. Express each centered data column in these coordinates.

Use the Gaussian ranks.
In these coordinates the combined $d$-by-$(p+q)$ data matrix has independent standard normal entries. Its rank is $\min(d,p+q)$ with probability one. Each separate block has rank $p$ or $q$ with probability one.

Find the dimension of the overlap.
For subspaces $U,V$, $\dim(U\cap V)=\dim(U)+\dim(V)-\dim(U+V)$. Here the sum of the two column spaces has dimension $d$ when $p+q>d$, so their intersection has dimension $p+q-d>0$.

Translate overlap into correlation.
A nonzero vector in the intersection can be expressed as a linear combination of the centered $X$-columns and also as a linear combination of the centered $Y$-columns. These two sample score vectors are identical, so their sample correlation is one. No correlation can exceed one; therefore $r_1=1$.

For completeness, the rank statement in Step 2 follows because a Gaussian matrix has a joint density and the event of rank less than $\min(d,p+q)$ is contained in the zero set of its maximal minors. At least one such minor is a nonzero polynomial, and the zero set of a nonzero polynomial has Lebesgue measure zero. The same argument applies to each separate block. The dimension formula in Step 3 then gives the claimed intersection dimension, completing the proof. $\square$

Worked Example: A Perfect Sample Root With Independent Variables

Take $n=12$, with $p=q=6$, and suppose each observation consists of independent standard normal vectors $X_i\in\mathbb{R}^6$ and $Y_i\in\mathbb{R}^6$. The two population vectors are independent, so every population canonical correlation is zero. After centering, the observation space has dimension

$$ d=n-1=11. $$

Each set of six centered columns has rank six with probability one, while the combined matrix has rank $\min(11,12)=11$ with probability one. Therefore

\dim\{\operatorname{span}(X)\cap\operatorname{span}(Y)\} =6+6-11=1.

The two column spaces share a nonzero score vector. CCA can select it from each set, giving $r_1=1$. This is not a rare large sample correlation: under this model and dimension choice, it is the exact result with probability one. The within-set sample covariance matrices are nonsingular because each block has rank six and $6\leq 11$.

Even One Variable Pair Has Sampling Variation

The exact-one result requires enough total variables relative to $n$. Optimism does not begin only at that threshold. Even with one variable in each set, the observed correlation fluctuates away from zero under a zero-correlation population model. A simple calculation quantifies this for independent standard normal variables.

Result: If $X_1,\ldots,X_n$ and $Y_1,\ldots,Y_n$ are two independent samples of independent standard normal observations, paired by observation index, and $r$ is their centered sample correlation, then $$ E[r^2]=\frac{1}{n-1}. $$

To prove the result, write the centered data vectors as $X_c$ and $Y_c$, and normalize them to unit length: $u=X_c/\lVert X_c\rVert$ and $v=Y_c/\lVert Y_c\rVert$. The sample correlation is $r=u^\mathsf{T}v$. In an orthonormal basis of the centered space, each vector has a rotationally invariant direction on the unit sphere in dimension $d=n-1$; independence of the samples makes these directions independent. Conditional on $u$, rotate coordinates so that $u$ is the first basis vector. Then $r^2=v_1^2$. Rotational symmetry gives the same expectation for every squared coordinate of $v$, and the coordinates satisfy $\sum_{j=1}^{d}v_j^2=1$. Consequently,

d\,E[v_1^2] =\sum_{j=1}^{d}E[v_j^2] =E\!\left[\sum_{j=1}^{d}v_j^2\right] =1, \qquad E[r^2]=E[v_1^2]=\frac{1}{d}=\frac{1}{n-1}.

This proves the result. $\square$ The statement concerns the expected squared correlation for one variable pair under this specific model. It is not the expected value of the largest root in a many-variable CCA, nor does it say that every observed squared correlation will equal $1/(n-1)$.

Worked Example: A Single Null Pair at Five Observations

Let $n=5$, so the centered space has dimension $d=4$. Two centered unit vectors in $\mathbb{R}^5$ are

a=\frac{(1,-1,0,0,0)^\mathsf{T}}{\sqrt{2}}, \qquad b=\frac{(1,1,-2,0,0)^\mathsf{T}}{\sqrt{6}}.

Both coordinate sums are zero, both squared lengths are one, and their inner product is zero. Set $x=a$ and $y=\frac12a+\frac{\sqrt{3}}{2}b$. Then $y$ is centered and has squared length $\frac14+\frac34=1$. Therefore the sample correlation is

r=\frac{x^\mathsf{T}y}{\lVert x\rVert\lVert y\rVert} =a^\mathsf{T}\!\left(\frac12a+\frac{\sqrt{3}}{2}b\right) =\frac12, \qquad r^2=\frac14.

Under independent standard normal sampling with five pairs, the result just proved gives $E[r^2]=1/(5-1)=1/4$. The calculation illustrates the amount of squared correlation that can arise from sampling variation; it does not claim that this particular correlation is typical or assign it a probability.

How the Number of Variables Changes the Search

CCA does not merely report a correlation chosen in advance. It chooses coefficients after seeing the data. Adding variables expands the set of possible score vectors, so the maximization has more opportunities to find an accidental alignment. The exact dimension theorem gives a sharp warning threshold for a perfect first root under its Gaussian null model: $p+q>n-1$. Below that threshold, a perfect root is not forced by dimension, but the sample root can still be positive and optimistic.

Worked Example: Comparing Two Dimension Choices

Suppose $n=12$, so the centered observation space has dimension $11$, and consider independent standard normal variable sets. First let $p=q=5$. The combined matrix has 10 columns, and with probability one its two five-dimensional spaces have intersection dimension

\max(0,p+q-11)=\max(0,10-11)=0.

There is no dimension-forced shared score vector, so the theorem does not force $r_1=1$. But CCA still selects the most aligned pair of directions among many candidates; a positive sample root does not establish population association.

Now increase each set to six variables while keeping $n=12$. The intersection dimension becomes $6+6-11=1$, and the first sample root is one with probability one. Each individual sample covariance remains invertible in both comparisons, since each block has at most 11 columns. Thus the change from ten to twelve total variables crosses a structural threshold, not merely a vague rule of thumb.

This is why sample canonical correlations should be interpreted as in-sample measures of association. When the scientific goal is to assess whether a pattern will persist beyond the observed cases, evaluate the chosen directions on separate data or use an appropriate resampling and validation design. Such evaluation addresses the selection effect: the validation observations did not help choose the directions in the first place. This distinction does not replace formal inference, which requires its own model and assumptions.

Common Mistake: Treating the Largest Root as an Unselected Correlation

Wrong reasoning: “The two sets are independent in the population, so their sample canonical correlation should be near zero; a large root must indicate a real relationship.”
Correction: The first sample root is the maximum over many data-dependent linear combinations. Independent sets can have aligned sample spaces, and if $p+q>n-1$ under the theorem's conditions, they share a nonzero score vector and the root is exactly one.
Wrong reasoning: “The expected squared correlation for a single pair tells me the expected largest CCA root.”
Correction: $E[r^2]=1/(n-1)$ is a result for one independent scalar pair. Maximizing over multiple variables changes the quantity; do not use the single-pair formula as an expected-root formula.

Key takeaway: Sample CCA optimizes association in the same data used to report it. As the total number of variables grows relative to the number of observations, accidental alignment becomes more consequential; under the stated Gaussian null model, $p+q>n-1$ guarantees a first sample canonical correlation of one.

Check Your Understanding

Answer each question using the definitions and dimension argument above.

Why does centering $n$ observations leave an observation space of dimension $n-1$?
For $n=20$, $p=8$, and $q=12$, what does the dimension theorem imply about the first sample canonical correlation under independent standard Gaussian variable sets?
For one independent standard normal variable pair with $n=9$, what is $E[r^2]$? What does this result not tell you about a many-variable CCA?
If $p+q\leq n-1$, does that guarantee a zero sample canonical correlation? Explain.
Why is a validation sample useful after CCA has selected its canonical directions?