Redundancy Indices for Canonical Variates

Association Is Not the Same as Variance Explained

A large canonical correlation says that a particular linear combination of one variable set is strongly associated with a particular linear combination of the other set. It does not, by itself, say how much of either set’s variation those combinations represent. A canonical variate could be strongly correlated with only a narrow component of its own set. Redundancy indices bring these two questions together: how much of a set is represented by a canonical variate, and how much of that represented variation is associated with the opposite set?

Consider two random vectors, $X=(X_1,\ldots,X_p)'$ and $Y=(Y_1,\ldots,Y_q)'$, with finite second moments and positive variances for their individual variables. Write $U_k=a_k'X$ and $V_k=b_k'Y$ for the $k$th canonical pair, with both canonical variates normalized to have variance one. Their correlation is the $k$th canonical correlation, $\rho_k=\operatorname{Corr}(U_k,V_k)$, conventionally ordered from largest to smallest. The discussion below uses correlations, so the original variables need not share measurement units.

Definition: The within-set variance extracted by $U_k$ from the $X$-set is the average squared correlation of $U_k$ with the $p$ variables in $X$: $$ VE_{X,k}=\frac{1}{p}\sum_{i=1}^{p}\operatorname{Corr}(X_i,U_k)^2. $$ Similarly, $$ VE_{Y,k}=\frac{1}{q}\sum_{j=1}^{q}\operatorname{Corr}(Y_j,V_k)^2. $$ The directional redundancy contributions are $$ R_{X\leftarrow Y,k}=\rho_k^2VE_{X,k}, \qquad R_{Y\leftarrow X,k}=\rho_k^2VE_{Y,k}. $$

The arrow indicates the set whose variables’ variation is being described and the set supplying the paired canonical variate. Thus $R_{X\leftarrow Y,k}$ measures the average proportion of variance in the $X$-variables associated with the $k$th canonical variate from $Y$. It is not generally equal to $R_{Y\leftarrow X,k}$, because the two within-set extracted variances use different variables and denominators.

Why Squared Canonical Correlation Enters

The structure loading of $X_i$ on $U_k$ is $\operatorname{Corr}(X_i,U_k)$. The corresponding cross-loading is $\operatorname{Corr}(X_i,V_k)$. Under the canonical normalization, the cross-loading is the structure loading multiplied by $\rho_k$. This relationship is what makes the redundancy product more than a convenient convention: it is the average squared cross-loading, hence the average variance in the standardized $X$-variables associated with $V_k$.

Result: For a canonical pair normalized to unit variances, $$ \operatorname{Corr}(X_i,V_k) = \rho_k\operatorname{Corr}(X_i,U_k), \qquad \operatorname{Corr}(Y_j,U_k) = \rho_k\operatorname{Corr}(Y_j,V_k). $$ Consequently, $$ R_{X\leftarrow Y,k} =\frac{1}{p}\sum_{i=1}^{p}\operatorname{Corr}(X_i,V_k)^2, \qquad R_{Y\leftarrow X,k} =\frac{1}{q}\sum_{j=1}^{q}\operatorname{Corr}(Y_j,U_k)^2. $$

To verify the first identity, let $\Sigma_{XX}=\operatorname{Cov}(X,X)$ and $\Sigma_{XY}=\operatorname{Cov}(X,Y)$. The canonical equations for a normalized pair give $\Sigma_{XY}b_k=\rho_k\Sigma_{XX}a_k$. Therefore,

\operatorname{Cov}(X,V_k) =\Sigma_{XY}b_k =\rho_k\Sigma_{XX}a_k =\rho_k\operatorname{Cov}(X,U_k).

Taking the $i$th coordinate gives $\operatorname{Cov}(X_i,V_k)=\rho_k\operatorname{Cov}(X_i,U_k)$. Both canonical variates have variance one, so dividing by the standard deviation of $X_i$ gives the stated correlation identity. The other identity follows from the transposed canonical equation $\Sigma_{YX}a_k=\rho_k\Sigma_{YY}b_k$, by taking its $j$th coordinate and dividing by the standard deviation of $Y_j$. Squaring the first identity, summing over $i$, and dividing by $p$ yields $$ \frac{1}{p}\sum_{i=1}^{p}\operatorname{Corr}(X_i,V_k)^2 =\frac{1}{p}\sum_{i=1}^{p}\rho_k^2\operatorname{Corr}(X_i,U_k)^2 =\rho_k^2VE_{X,k}. $$ The same calculation for $Y$ proves the second redundancy identity. $\square$

Worked Examples

Worked Example: A Gaussian Model With One Canonical Pair

Let $X_1,\ldots,X_4,\varepsilon$ be independent $N(0,1)$ variables. Define $$ U=0.8X_1+0.4X_2+0.4X_3+0.2X_4, \qquad Y=0.8U+0.6\varepsilon. $$ Find the within-set extraction from $X$, the canonical correlation, and the redundancy of $X$ through $Y$.

First, the variance of $U$ is $0.8^2+0.4^2+0.4^2+0.2^2=0.64+0.16+0.16+0.04=1$. Also, $\operatorname{Var}(Y)=0.8^2+0.6^2=1$, and $\operatorname{Cov}(U,Y)=0.8$. Thus $\operatorname{Corr}(U,Y)=0.8$. Because $Y$ is the sole variable in its set, its canonical variate is $Y$; by the Cauchy–Schwarz inequality, no unit-variance linear combination of the independent $X_i$'s can correlate with $Y$ by more than $0.8$, and $U$ attains that value. Hence this is the canonical pair.

Independence and the unit variance of $U$ give $\operatorname{Corr}(X_i,U)=(0.8,0.4,0.4,0.2)_i$. Therefore, $$ VE_X=\frac{0.8^2+0.4^2+0.4^2+0.2^2}{4} =\frac{1}{4}=0.25. $$ The redundancy of the $X$-set through $Y$ is $$ R_{X\leftarrow Y}=0.8^2(0.25)=0.64(0.25)=0.16. $$ So the canonical pair is strongly correlated, but its redundancy amounts to 16% of the average standardized variance in the four $X$-variables.

Worked Example: Why the Two Directions Can Differ

Let $X_1,X_2,\varepsilon$ be independent $N(0,1)$ variables. Set $U=(X_1+X_2)/\sqrt{2}$ and $Y=0.6U+0.8\varepsilon$. Here the $X$-set has two variables and the $Y$-set has one. Calculate both directional redundancies.

The variance of $U$ is $(1+1)/2=1$, and $\operatorname{Var}(Y)=0.6^2+0.8^2=1$. Their covariance, and therefore their correlation, is $0.6$. Since $Y$ is the only variable in its set, this gives the canonical correlation $\rho=0.6$. Each $X_i$ has correlation $1/\sqrt{2}$ with $U$, so $$ VE_X=\frac{(1/\sqrt{2})^2+(1/\sqrt{2})^2}{2} =\frac{1/2+1/2}{2}=0.5. $$ The within-set extraction for $Y$ is $VE_Y=1$, because $Y$ is itself the canonical variate and its correlation with itself is one. Thus $$ R_{X\leftarrow Y}=0.6^2(0.5)=0.18, \qquad R_{Y\leftarrow X}=0.6^2(1)=0.36. $$ The difference is not a contradiction: the first index averages across two $X$-variables, while the second averages across one $Y$-variable. Each index describes a different set.

Worked Example: Summing Contributions Across Two Roots

Let $X_1,X_2,X_3,\varepsilon_1,\varepsilon_2$ be independent $N(0,1)$ variables. Define $U_1=X_1$, $U_2=X_2$, $V_1=0.9X_1+\sqrt{0.19}\varepsilon_1$, and $V_2=0.4X_2+\sqrt{0.84}\varepsilon_2$. The $X$-set is $(X_1,X_2,X_3)'$, and the $Y$-set is $(V_1,V_2)'$. Find the two roots’ redundancy contributions and their totals.

Both $V$-scores have variance one: $0.9^2+0.19=1$ and $0.4^2+0.84=1$. They are uncorrelated because they depend on disjoint independent variables, as are $U_1$ and $U_2$. The canonical correlations are $0.9$ and $0.4$. For each $X$-side variate, only one of the three $X$-variables has a nonzero structure loading, equal to one. Thus $VE_{X,1}=VE_{X,2}=1/3$. Each $Y$-side variate loads one of the two $Y$-variables by one and the other by zero, so $VE_{Y,1}=VE_{Y,2}=1/2$.

The contributions and their sums are $$ R_{X\leftarrow Y,1}=0.9^2\left(\frac{1}{3}\right)=0.27, \qquad R_{X\leftarrow Y,2}=0.4^2\left(\frac{1}{3}\right)=0.053\overline{3}, $$ $$ R_{X\leftarrow Y}=0.27+0.053\overline{3}=0.323\overline{3}. $$ In the reverse direction, $$ R_{Y\leftarrow X,1}=0.9^2\left(\frac{1}{2}\right)=0.405, \qquad R_{Y\leftarrow X,2}=0.4^2\left(\frac{1}{2}\right)=0.08, $$ $$ R_{Y\leftarrow X}=0.405+0.08=0.485. $$ The second root contributes less than the first in both directions, but totals differ between sets because their dimensions and within-set averages differ.

Total Redundancy Across Canonical Roots

For $s=\min(p,q)$ canonical pairs, the total directional redundancy is conventionally obtained by summing the contributions: $$ R_{X\leftarrow Y}=\sum_{k=1}^{s}\rho_k^2VE_{X,k}, \qquad R_{Y\leftarrow X}=\sum_{k=1}^{s}\rho_k^2VE_{Y,k}. $$ A useful interpretation follows when canonical scores within each set are mutually uncorrelated and have unit variance, as under the usual canonical normalization. The sum in the first formula is the average of the individual $X$-variables’ squared multiple correlations with the collection $V_1,\ldots,V_s$. It therefore describes the average linear variance accounted for in the $X$-set by these $Y$-side scores.

To prove this, fix a standardized $X_i$. Since the $V_k$'s have variance one and are mutually uncorrelated, the linear least-squares projection of $X_i$ onto their span is $\sum_{k=1}^{s}\operatorname{Cov}(X_i,V_k)V_k$. Its variance is $$ \operatorname{Var}\left(\sum_{k=1}^{s}\operatorname{Cov}(X_i,V_k)V_k\right) =\sum_{k=1}^{s}\operatorname{Cov}(X_i,V_k)^2, $$ because all cross-covariances between distinct $V$-scores are zero. As $\operatorname{Var}(X_i)=1$, this is the proportion of variance explained in $X_i$ by that projection. By the cross-loading identity, $\operatorname{Cov}(X_i,V_k)=\operatorname{Corr}(X_i,V_k)=\rho_k\operatorname{Corr}(X_i,U_k)$. Averaging the resulting squared correlations over the $p$ variables gives $$ \frac{1}{p}\sum_{i=1}^{p}\sum_{k=1}^{s}\operatorname{Corr}(X_i,V_k)^2 =\sum_{k=1}^{s}\rho_k^2\left\{\frac{1}{p}\sum_{i=1}^{p}\operatorname{Corr}(X_i,U_k)^2\right\} =\sum_{k=1}^{s}\rho_k^2VE_{X,k}. $$ This proves the stated interpretation; the same argument with $X$ and $Y$ exchanged applies in the other direction. $\square$

The sum is an average across variables, not a sum of variances across the two sets. It is also not a causal measure: canonical association and redundancy quantify linear association in the observed variables, not the effect of intervening on one set.

Interpretation and Common Pitfalls

A high canonical correlation can coexist with modest redundancy when the canonical variate represents little of its own variable set. Conversely, a canonical variate that extracts a broad share of its own set contributes little redundancy if its paired canonical correlation is small. Both factors matter, and neither the root alone nor the extracted variance alone gives the directional index.

Using $\rho_k$ instead of $\rho_k^2$. Correlation is not a proportion of variance. The transmitted variance uses the squared canonical correlation.
Confusing extraction with redundancy. $VE_{X,k}$ is within-set extraction by $U_k$; $R_{X\leftarrow Y,k}$ additionally accounts for the association with $V_k$.
Assuming the directions are equal. The $X$- and $Y$-side indices average over different sets and generally have different extracted variances.
Calling a redundancy index a causal effect. It summarizes linear variance association, not how one set would change under an intervention on the other.

Key takeaway: For a given root, multiply the average squared structure loadings in the target set by the squared canonical correlation. Keep the direction explicit: $R_{X\leftarrow Y,k}=\rho_k^2VE_{X,k}$ and $R_{Y\leftarrow X,k}=\rho_k^2VE_{Y,k}$.

Check Your Understanding

Use the definitions and derivations above to answer these questions.

If the average squared structure loading in the $X$-set is $0.30$ and the canonical correlation is $0.70$, what is the $X$-set redundancy contribution for that root?
Explain why the within-set variance extracted by $U_k$ is not itself the variance in $X$ associated with $V_k$.
What assumptions about the canonical scores allow total redundancy to be interpreted as average variance explained by their span?
Why can the two directional redundancy indices for one canonical pair differ?
A root has a high canonical correlation but low within-set extraction. What does that imply about its redundancy contribution?