Why Maximize Covariance?
In a multi-omics study, two data blocks may describe the same paired samples through different measurements—for example, one block of molecular features and a second block of molecular features. A two-block method forms one score from each block and looks for a pair that varies together. Partial least squares (PLS) does this by maximizing the covariance between the two scores. The choice matters: unlike correlation, covariance retains the measurement scales of the variables.
The earlier material on canonical correlation analysis (CCA) formulates its objective as maximizing the correlation between linear scores. Here, the key distinction is the normalization: PLS constrains the lengths of the weight vectors, while CCA constrains the variances of the resulting scores. These choices can lead to different directions when the variables have different scales or within-block covariance structures.
Writing the Objective Precisely
Because the columns are centered, both score vectors are centered. Their sample covariance is
Thus, the covariance-maximization formulation is
The unit-norm constraints prevent the objective from being made arbitrarily large just by multiplying a weight vector by a large constant. They constrain the weights, not the lengths or variances of the scores. In particular, the scores \(Xw\) and \(Yc\) need not have unit norm or unit variance.
This formulation assumes paired observations: row \(i\) of \(X\) and row \(i\) of \(Y\) must refer to the same sample. The covariance must also be defined; for a sample covariance, that requires at least two complete paired observations. Centering is part of the formulation, not a cosmetic step: without it, the cross-product generally mixes covariance with differences in the block means. Normality is not required to define or optimize this descriptive sample objective. Inferential claims about a population would require additional assumptions suited to the sampling and dependence structure.
Worked Example: A One-Variable PLS Covariance
Suppose four hypothetical paired samples have centered measurements \(x=(2,-2,2,-2)^\mathsf{T}\) and \(y=(1,-1,1,-1)^\mathsf{T}\). Each block has one variable, so the only unit-norm weights are \(w=1\) or \(-1\) and \(c=1\) or \(-1\). Calculate the covariance for \(w=c=1\).
The scores are \(t=x\) and \(u=y\). Their sample covariance is
The covariance is \(2.667\) measurement-units of \(x\) times measurement-units of \(y\). Reversing one weight changes it to \(-2.667\), so the largest absolute covariance is \(2.667\). The observations are paired and centered as required. This small example shows why covariance is not a unit-free measure: changing the units of either variable changes its numerical value.
How CCA Uses a Different Constraint
For comparison, let \(S_{xx}=X^\mathsf{T}X/(n-1)\) and \(S_{yy}=Y^\mathsf{T}Y/(n-1)\) be the within-block sample covariance matrices. The sample variances of \(t\) and \(u\) are \(w^\mathsf{T}S_{xx}w\) and \(c^\mathsf{T}S_{yy}c\). When both are positive, their sample correlation is
CCA maximizes this correlation. One equivalent formulation, when the score variances can be made positive, is to maximize \(w^\mathsf{T}S_{xy}c\) subject to \(w^\mathsf{T}S_{xx}w=1\) and \(c^\mathsf{T}S_{yy}c=1\). These constraints normalize the score variances, taking the covariance relative to the variability within each block. In PLS, by contrast, Euclidean weight lengths are fixed, and the variances induced by those weights are not held constant.
Worked Example: PLS and CCA Can Prefer Different Directions
Consider hypothetical centered data from \(n\geq 5\) paired samples with covariance matrices
These are valid covariance blocks: the paired coordinate blocks have determinants \(4(1)-0.6^2=3.64>0\) and \(1(1)-0.8^2=0.36>0\). For PLS, take the unit weight vectors \(w=(w_1,w_2)^\mathsf{T}\) and \(c=(c_1,c_2)^\mathsf{T}\). The covariance objective is
Since \(0.6w_1c_1+0.8w_2c_2\leq 0.8\bigl(|w_1c_1|+|w_2c_2|\bigr)\leq 0.8\), the largest absolute covariance is \(0.8\), attained by \(w=c=(0,1)^\mathsf{T}\).
For CCA, the coordinate-wise correlations are \(0.6/\sqrt{4\cdot1}=0.3\) and \(0.8/\sqrt{1\cdot1}=0.8\). More generally, because these covariance matrices are diagonal, CCA’s variance-normalized cross-block matrix is diagonal with entries \(0.3\) and \(0.8\); its largest correlation is \(0.8\), also attained in the second coordinate. The two methods agree on this example, but their objectives are different. The calculations use positive score variances and correctly paired, centered observations.
Why Scale Changes the PLS Objective
The next example isolates the effect of units. A change in units is a rescaling of a variable; it does not add new information about the samples. Yet because covariance changes under rescaling, it can change which direction maximizes the PLS objective. Correlation, by contrast, is unchanged by positive rescaling.
Worked Example: Rescaling a Feature Changes the PLS Choice
Suppose a different hypothetical paired data set has
As above, PLS selects the second coordinate because \(0.8>0.6\). Now express the first \(X\)-variable in units twice as large. Its values are multiplied by 2, so its variance is multiplied by \(2^2=4\), and its covariances with the \(Y\)-variables are multiplied by 2. The updated matrices are \(S_{xx}^{*}=\begin{pmatrix}16&0\\0&1\end{pmatrix}\) and \(S_{xy}^{*}=\begin{pmatrix}1.2&0\\0&0.8\end{pmatrix}\). The maximum PLS covariance is now \(1.2\), attained by the first coordinate.
The first coordinate’s correlation remains \(1.2/\sqrt{16\cdot1}=0.3\); the second remains \(0.8/\sqrt{1\cdot1}=0.8\). Thus, CCA still selects the second coordinate. This calculation is consistent with positive rescaling: the covariance and the variance change, but the corresponding correlation does not. It also shows that a larger PLS covariance after a unit conversion does not mean the underlying association became stronger.
Derivation and Interpretation
The covariance identity follows directly from the definition of the scores. Substitute \(t=Xw\) and \(u=Yc\) into their sample covariance. Matrix multiplication then moves the weights outside the cross-product of the data matrices:
The result clarifies both the target and its limits. PLS seeks directions with large observed cross-block covariance under its chosen weight constraint. It does not, from this objective alone, test whether an association is statistically significant, establish a causal relationship, or guarantee that the selected direction will predict well in new samples. Those claims require separate analysis.
Diagnostics, Alternatives, and Limitations
Before interpreting a fitted direction, check that rows identify the same paired samples, that columns have been centered consistently, and that missing values have not caused the two blocks to refer to different sample sets. Examine feature units and scales: variables measured in large numerical units can exert greater influence on a covariance objective. Inspect unusually influential observations as well, because a small number of extreme paired values can substantially affect a sample covariance. These are checks on the data and objective, not tests that guarantee a stable direction.
The main alternative in this comparison is CCA. It targets scale-normalized association and can therefore answer a different question from covariance-maximizing PLS. Standardizing individual variables before PLS is another possible preprocessing choice: it can reduce differences caused only by measurement units, but it does not generally make PLS equivalent to CCA. CCA accounts for the full within-block covariance of linear scores, not only the separate scales of the original variables.
| Feature | Two-block PLS | CCA |
|---|---|---|
| Target | Covariance between linear scores | Correlation between linear scores |
| Normalization | Unit Euclidean norms for the weight vectors | Unit variances for the score vectors |
| Effect of changing a variable’s units | Can change the selected direction | Positive rescaling alone does not change score correlation |
| Interpretation | Directions with large cross-block covariance under the chosen scale | Directions with strong scale-normalized linear association |
Neither objective is universally preferable. Covariance can be meaningful when the units and scales themselves are relevant to the scientific target; otherwise, scale sensitivity should be treated as a limitation and reported transparently. In high-dimensional data, covariance estimates and selected directions can also be unstable, especially when there are few samples relative to the number of features. The optimization objective alone does not solve that estimation problem.
Common Pitfall
A common mistake is to say, “The PLS scores are normalized, so PLS is just maximizing their correlation.” That reasoning confuses two different constraints. In the formulation here, the weights have unit Euclidean norm; the resulting scores can have different variances. Their covariance therefore need not equal their correlation. The correction is to state the constraint explicitly before interpreting the objective: unit-norm weights define this covariance-maximizing PLS formulation, while unit-variance scores define the corresponding CCA formulation.
Check Your Understanding
Answer each question, then compare with the concise solutions below.
- For centered paired data, write the sample covariance between \(Xw\) and \(Yc\) in terms of \(S_{xy}\).
- If the PLS weights have unit Euclidean norm, must the two score variances equal 1? Explain.
- A positive change of units doubles one \(X\)-variable. What happens to its variance and its covariance with a fixed \(Y\)-score?
- Why is maximizing covariance not generally the same as maximizing correlation?
Solutions. (1) The score covariance is \(w^\mathsf{T}S_{xy}c\), since \((Xw)^\mathsf{T}(Yc)/(n-1)=w^\mathsf{T}S_{xy}c\). (2) No. The score variance is \(w^\mathsf{T}S_{xx}w\), which depends on the within-block covariance matrix and need not be 1. (3) The variable’s variance is multiplied by \(2^2=4\), and its covariance with a fixed \(Y\)-score is multiplied by 2. (4) Correlation divides covariance by the product of the score standard deviations; those standard deviations can differ across candidate directions, so the maximizing directions can differ.