From Covariance Maximization to an SVD
Two-block partial least squares (PLS) seeks directions in two data blocks whose resulting scores have large covariance. The earlier material on Partial Least Squares as Covariance Maximization sets up this objective. Here the goal is to derive its solution and show why a singular value decomposition (SVD) gives the maximizing directions directly.
Let \(X\) be an \(n \times p\) data matrix and \(Y\) an \(n \times q\) data matrix, with rows referring to the same \(n\) observational units. Assume both matrices have been column-centered, so each column has sample mean zero, and that the paired observations are complete for the variables being used. Define the sample cross-block covariance matrix by \(S_{XY}=X^\top Y/(n-1)\). For weight vectors \(w \in \mathbb{R}^p\) and \(c \in \mathbb{R}^q\), the scores are \(Xw\) and \(Yc\).
This is a matrix optimization problem, not a probability model requiring normality. Its algebraic solution applies to any fixed, finite matrix \(S_{XY}\), provided the data blocks are paired and the objective and constraints are as stated. The directions depend on how variables are measured: changing units or scaling columns generally changes \(S_{XY}\), and thus can change the maximizing weights.
Deriving the Maximizer
Write a singular value decomposition of the cross-block covariance matrix as \(S_{XY}=U\Sigma V^\top\). Its singular values are ordered \(\sigma_1 \geq \sigma_2 \geq \cdots \geq 0\), and the corresponding columns of \(U\) and \(V\) are the left and right singular vectors. For any unit vectors \(w\) and \(c\), the objective can be bounded using the operator norm, which is the largest singular value.
The second inequality is Cauchy–Schwarz. The third uses \(\|S_{XY}c\|_2 \leq \sigma_1\|c\|_2\), which follows from the definition of the largest singular value. The bound is attained by setting \(c=v_1\) and \(w=u_1\), since \(S_{XY}v_1=\sigma_1u_1\). Therefore their score covariance is \(u_1^\top S_{XY}v_1=\sigma_1\).
The same conclusion follows by optimizing one weight at a time. For fixed \(c\), the best unit \(w\) points in the direction of \(S_{XY}c\), if that vector is nonzero. The resulting maximum is \(\|S_{XY}c\|_2\); maximizing this over unit \(c\) gives \(\sigma_1\). This view helps explain why both sides of the SVD matter: the right vector identifies a direction in the \(Y\)-variable space, and applying \(S_{XY}\) maps it to the matching direction in the \(X\)-variable space.
The singular vectors have a sign convention that matters for interpretation. If the largest singular value is positive and simple, the maximizing pair is unique apart from the paired sign choice \((u_1,v_1)\) or \((-u_1,-v_1)\). Reversing both signs leaves the covariance at \(\sigma_1\); reversing only one gives covariance \(-\sigma_1\), not the maximum of the signed objective. If the largest singular value is repeated, there are multiple maximizing pairs within the corresponding left and right singular subspaces. If \(S_{XY}=0\), every unit pair has covariance zero and is maximizing.
Numerical Verification
Worked Example: A Distinct Leading Singular Value
Suppose a hypothetical analysis of paired, centered measurements gives the cross-block covariance matrix, in the variables’ stated units, $$ S_{XY}= \begin{pmatrix} 3&0\\ 0&1 \end{pmatrix}. $$
Here \(S_{XY}^\top S_{XY}=\begin{pmatrix}9&0\\0&1\end{pmatrix}\), so its largest eigenvalue is \(9\), and the singular values of \(S_{XY}\) are \(3\) and \(1\). The leading left and right singular vectors are both \(e_1=(1,0)^\top\). Thus \(w=e_1,\ c=e_1\), and the maximum covariance is $$ w^\top S_{XY}c = (1,0) \begin{pmatrix} 3&0\\ 0&1 \end{pmatrix} \begin{pmatrix} 1\\0 \end{pmatrix} =3. $$
For comparison, choosing \(w=e_2,\ c=e_2\) gives covariance \(1\), and choosing \(w=e_1,\ c=e_2\) gives covariance \(0\). The leading singular-vector pair therefore attains the largest value among these candidates, as the theorem guarantees. The value 3 has units equal to the product of the units of the two score variables.
Worked Example: Checking the SVD From Hypothetical Centered Data
Consider three hypothetical paired observations with centered blocks $$ X= \begin{pmatrix} 1&1/\sqrt{3}\\ -1&1/\sqrt{3}\\ 0&-2/\sqrt{3} \end{pmatrix}, \qquad Y= \begin{pmatrix} 1+2/\sqrt{3}&2+1/\sqrt{3}\\ -1+2/\sqrt{3}&-2+1/\sqrt{3}\\ -4/\sqrt{3}&-2/\sqrt{3} \end{pmatrix}. $$
Each column sums to zero. Direct multiplication gives \(X^\top X/2=I\) and \(Y=X\begin{pmatrix}1&2\\2&1\end{pmatrix}\). Therefore, using \(n-1=2\), $$ S_{XY}=\frac{X^\top Y}{2} =\frac{X^\top X}{2} \begin{pmatrix}1&2\\2&1\end{pmatrix} = \begin{pmatrix}1&2\\2&1\end{pmatrix}. $$
This matrix is symmetric. Its unit eigenvectors \(u_1=v_1=(1,1)^\top/\sqrt{2}\) and \(u_2=v_2=(1,-1)^\top/\sqrt{2}\) have eigenvalues \(3\) and \(-1\), respectively. The singular values are the absolute values \(3\) and \(1\), so \(u_1,v_1\) are the leading singular vectors. Substitution verifies the maximum: $$ w^\top S_{XY}c = \frac{1}{\sqrt{2}}(1,1) \begin{pmatrix}1&2\\2&1\end{pmatrix} \frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\end{pmatrix} =3. $$
For the second pair \(u_2,v_2\), the signed covariance is \(-1\). This illustrates why an eigenvalue decomposition alone should not be confused with an SVD: a negative eigenvalue contributes a positive singular value, but its eigenvector pair does not exceed the leading signed covariance here. The SVD directly identifies the pair attaining the maximum.
Worked Example: Repeated Largest Singular Values
Suppose a hypothetical cross-block covariance matrix is \(S_{XY}=2I_2\). Its two singular values are both \(2\), so the largest one is not simple. For any unit vector \(a=(a_1,a_2)^\top\), choose \(w=a\) and \(c=a\). Then $$ w^\top S_{XY}c =a^\top(2I_2)a =2(a_1^2+a_2^2) =2. $$
For instance, both \(w=c=(1,0)^\top\) and \(w=c=(0,1)^\top\) maximize the objective; so does \(w=c=(1,1)^\top/\sqrt{2}\). The leading directions are not determined up to sign alone. More generally, if the largest singular value \(\sigma_1>0\) has multiplicity \(r\), any unit coefficient vector \(z\in\mathbb{R}^r\) produces a maximizing pair \(w=U_1z,\ c=V_1z\), where \(U_1,V_1\) contain the corresponding \(r\) leading singular vectors. Indeed, \(S_{XY}V_1z=\sigma_1U_1z\), so \(w^\top S_{XY}c=\sigma_1\).
Computational Checks and Common Failure Modes
An SVD is usually computed numerically, so the result should be checked against the defining objective rather than treated as a black box. For computed candidate vectors \(w,c\), verify that each has norm one, calculate \(w^\top S_{XY}c\), and compare that value with the reported leading singular value. Small discrepancies can arise from rounding; large discrepancies suggest an indexing, transposition, or scaling error.
Confirm that rows of \(X\) and \(Y\) correspond to the same units, apply a consistent complete-case rule if necessary, and center each column before forming the sample cross-covariance matrix.
Use \(S_{XY}=X^\top Y/(n-1)\). For \(S_{XY}=U\Sigma V^\top\), take the first columns of \(U\) and \(V\) as the candidate weights.
Check \(w^\top S_{XY}c\approx\sigma_1\), the weight norms, and the gap between the first and second singular values. A very small gap signals that directions may be sensitive to small data perturbations.
A common mistake is to treat the largest singular vectors as uniquely identified in every case. That is true only up to a paired sign when the largest singular value is simple and positive. With a repeated largest singular value, distinct directions in the leading singular subspaces also maximize covariance. Near-repetition is not exact non-uniqueness, but it can make estimated directions unstable.
Another frequent error is to interpret unit-norm weights as making the two score variances equal to one. The constraints are \(\|w\|_2=\|c\|_2=1\); in general, \(\operatorname{var}(Xw)=w^\top S_{XX}w\) and \(\operatorname{var}(Yc)=c^\top S_{YY}c\), neither of which is forced to equal one. This distinguishes the objective from canonical correlation analysis (CCA), which constrains score variances when maximizing correlation. It also means that raw-unit differences and scaling choices can strongly influence PLS covariance.
The derivation is exact for the chosen sample matrix and objective, but it does not by itself establish that a large observed covariance reflects a reproducible or scientifically important association. It does not supply an inferential test, guarantee stable weights in a small sample, or make the association causal. Those questions require additional design and validation considerations. Numerically, inspect the singular-value gap, check centering and variable scaling, and confirm the score covariance directly. A small leading singular value means the maximum covariance itself is small; a tied or nearly tied leading value raises a separate concern about direction identification.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If the largest singular value of \(S_{XY}\) is \(4\), what is the maximum signed score covariance under the unit-norm constraints?
- For \(S_{XY}=\begin{pmatrix}0&0\\0&5\end{pmatrix}\), identify one maximizing pair of weights.
- Why does \(\|w\|_2=1\) not imply that the score \(Xw\) has sample variance one?
- If the largest singular value has multiplicity two, are the maximizing directions unique up to a paired sign? Explain.
Solutions. (1) The maximum is \(\sigma_1=4\), in the units of the score covariance. (2) The leading left and right singular vectors are both \((0,1)^\top\), giving covariance \(5\). (3) The score variance is \(w^\top S_{XX}w\), which depends on the covariance structure of \(X\); the weight norm alone does not set it to one. (4) No. There are multiple maximizing pairs in the leading left and right singular subspaces, in addition to paired sign reversal.