PLS Regression Versus Two-Block PLS

Two PLS Methods, Two Questions

Partial least squares is a family of methods that uses covariance to construct latent scores. The name can refer to methods with different goals, however. Two-block PLS asks which linear combination of one variable block covaries most with a linear combination of a second block. PLS regression designates one block as predictors and another as the response, and constructs components to support prediction of that response. These questions may lead to the same first component, but they are not interchangeable interpretations.

Let $X$ be an $n\times p$ matrix and $Y$ an $n\times q$ matrix of measurements on the same $n$ subjects. Assume the rows are paired correctly and both blocks have been centered column by column. The sample cross-covariance matrix is

S_{XY}=\frac{X^\mathsf{T}Y}{n-1}.

For weight vectors $w\in\mathbb{R}^p$ and $c\in\mathbb{R}^q$, the score vectors are $t=Xw$ and $u=Yc$. Centering ensures that their sample covariance is $w^\mathsf{T}S_{XY}c$.

Definition: In the first-component two-block PLS-SVD formulation, choose weights of unit Euclidean length to maximize the covariance between the two scores: $$ \max_{\lVert w\rVert_2=1,\ \lVert c\rVert_2=1} w^\mathsf{T}S_{XY}c. $$ The sign of either weight can be reversed, so the equivalent sign-invariant objective is to maximize $\left|w^\mathsf{T}S_{XY}c\right|$. The resulting scores are a paired summary of the two blocks.

This constraint is on the lengths of the weights, not on the variances of the scores. Consequently, the criterion is sensitive to measurement units and to the scales of the variables unless preprocessing changes those scales. The method is algebraically defined for centered data without a normality assumption; its interpretation still depends on meaningful measurements and correctly matched rows.

Why the Two-Block Solution Uses Singular Vectors

The first-component solution follows directly from the singular value decomposition (SVD) of the cross-covariance matrix. Write $S_{XY}=U\Sigma V^\mathsf{T}$, where the diagonal entries of $\Sigma$ are nonnegative singular values in decreasing order. For any fixed unit $w$, the largest value of $w^\mathsf{T}S_{XY}c$ over unit $c$ is $\lVert S_{XY}^\mathsf{T}w\rVert_2$: this follows from the Cauchy–Schwarz inequality, with equality when $c$ points in the direction of $S_{XY}^\mathsf{T}w$.

Maximizing that quantity over unit $w$ gives the largest singular value of $S_{XY}$, denoted $\sigma_1$. The maximizing weights are corresponding leading left and right singular vectors, $u_1$ and $v_1$. Substitution verifies the achieved covariance: $u_1^\mathsf{T}S_{XY}v_1=\sigma_1$. If the largest singular value is repeated, there can be more than one maximizing pair of directions.

w=u_1,\qquad c=v_1,\qquad \operatorname{Cov}(Xw,Yc)=\sigma_1.

Thus two-block PLS gives a paired, covariance-maximizing description of $X$ and $Y$. It does not, by this objective alone, say that one block predicts the other well for new subjects. A large covariance between training scores is an association property, not a prediction-error guarantee.

Worked Example: The Strongest Paired Direction

Suppose a hypothetical centered analysis gives the cross-covariance matrix, in the variables' stated units,

S_{XY}= \begin{pmatrix} 2&0\\ 0&5 \end{pmatrix}.

Both blocks have two variables. Under the unit-length weight constraints, the singular values are 5 and 2. The leading left and right singular vectors are both $(0,1)^\mathsf{T}$, so the two-block scores use the second variable in each block. Their sample covariance is

(0,1) \begin{pmatrix} 2&0\\ 0&5 \end{pmatrix} \begin{pmatrix}0\\1\end{pmatrix} =5.

The alternative unit weights $(1,0)^\mathsf{T}$ in both blocks give covariance 2. Therefore the first pair is selected because it has the larger cross-block covariance, not because the second variable in $Y$ has been declared the prediction target. If the entries are, for example, measured in two different combinations of assay units, the numeric comparison also reflects those units.

PLS Regression Makes a Response the Target

In PLS regression, $X$ is designated as the predictor matrix and $Y$ as the response matrix. In the single-response case, write the centered response as $y\in\mathbb{R}^n$. The first PLS regression direction is commonly defined to make the predictor score $t=Xw$ covary strongly with $y$, subject to a weight normalization such as $\lVert w\rVert_2=1$. For this first component, maximizing the squared covariance is equivalent to maximizing its absolute value.

Definition: For single-response first-component PLS regression with centered $X$ and $y$, the covariance criterion is $$ \max_{\lVert w\rVert_2=1}\left|\,\frac{w^\mathsf{T}X^\mathsf{T}y}{n-1}\,\right|. $$ If $X^\mathsf{T}y\ne 0$, a maximizing direction is $w=X^\mathsf{T}y/\lVert X^\mathsf{T}y\rVert_2$, up to sign. The score $t=Xw$ is then used in a regression fit for the designated response.

The direction follows because the objective is the absolute inner product of $w$ with $X^\mathsf{T}y/(n-1)$; with $\lVert w\rVert_2=1$, Cauchy–Schwarz bounds it by $\lVert X^\mathsf{T}y/(n-1)\rVert_2$, attained by aligning $w$ with that vector. If $X^\mathsf{T}y=0$, every unit direction has zero covariance under this criterion, so no unique covariance-guided first direction is determined.

For multiple responses, PLS regression procedures can construct response-guided components from the cross-covariance between $X$ and $Y$, then use the resulting scores to predict $Y$. With the same centered matrices, unit-weight constraints, and first-component SVD criterion as two-block PLS, the first paired directions can be identical. The distinction is then the analysis goal and how components are used: paired representation of two blocks versus a fitted model whose designated purpose is response prediction. Across algorithms or later components, the procedures need not coincide.

Worked Example: A Single-Response PLS Direction and Fit

Consider four hypothetical subjects with centered predictor columns $x_1=(-1,-1,1,1)^\mathsf{T}$ and $x_2=(-2,2,-2,2)^\mathsf{T}$. Let the centered response be $y=2x_1+0.5x_2$. The predictor columns are orthogonal: $x_1^\mathsf{T}x_2=0$, with $x_1^\mathsf{T}x_1=4$ and $x_2^\mathsf{T}x_2=16$. Thus $X^\mathsf{T}y=(8,8)^\mathsf{T}$, and the unit first-component weight is

w=\frac{(8,8)^\mathsf{T}}{\sqrt{8^2+8^2}} =\frac{(1,1)^\mathsf{T}}{\sqrt{2}}.

The score is $t=(x_1+x_2)/\sqrt{2}$. Its sum of squares is $t^\mathsf{T}t=(4+16)/2=10$, while $t^\mathsf{T}y=(8+8)/\sqrt{2}=8\sqrt{2}$. The least-squares coefficient for regressing the centered response on this one score is therefore

\hat{\gamma}=\frac{t^\mathsf{T}y}{t^\mathsf{T}t} =\frac{8\sqrt{2}}{10} =\frac{4\sqrt{2}}{5}.

The one-component fitted response is $t\hat{\gamma}=(4/5)(x_1+x_2)$. In coefficient form on the original predictor columns, this is $0.8x_1+0.8x_2$, rather than the full response $2x_1+0.5x_2$. For example, the residual is $1.2x_1-0.3x_2$, whose squared length is $1.2^2(4)+0.3^2(16)=7.2$. This example shows why maximizing first-score covariance does not generally mean that one component reproduces the response exactly.

When the Methods Agree—and When They Do Not

If PLS regression has one response $y$, two-block PLS with $Y=y$ has only one possible unit response weight, $c=1$ or its sign reversal. Its first predictor weight is therefore the same covariance-maximizing direction as single-response PLS regression, given the same centering, scaling, and normalization. More generally, a multivariate-response first component can agree with the two-block PLS-SVD solution when both use the same cross-covariance matrix and constraints.

That agreement does not erase the difference in interpretation. In a genuinely paired-block analysis, neither block is inherently the response. In regression, the analyst specifies the prediction target, evaluates fitted responses, and may choose a number of components based on prediction performance. The two-block covariance objective does not itself select a component count to minimize prediction error.

Worked Example: A Designated Response Changes the Question

Return to the hypothetical cross-covariance matrix in the first example, and now designate the first column of $Y$, denoted $y_1$, as the single response. The covariance of the two $X$ variables with $y_1$ is the first column of $S_{XY}$, namely $(2,0)^\mathsf{T}$. Single-response PLS therefore chooses $w=(1,0)^\mathsf{T}$, up to sign; its score uses the first $X$ variable. Its covariance with $y_1$ is 2.

Two-block PLS instead uses all of $Y$ symmetrically. Its leading pair is the second variable of $X$ and the second variable of $Y$, with covariance 5. There is no contradiction: the methods answer different questions. If the task is to predict $y_1$, the covariance of the second $X$ variable with $y_2$ does not make that variable the preferred single-response direction. If both columns of $Y$ are treated as a response block under the same first-component SVD criterion, the first direction can again agree with two-block PLS; the prediction interpretation is then directed from $X$ toward the response block.

Diagnostics, Alternatives, and Limitations

Check whether the analysis matches its stated goal before interpreting a component. For two-block PLS, inspect the paired score covariance and the singular values, and confirm that the same subjects occur in the same row order in both matrices. Check variable units and preprocessing: a large-scale variable can dominate a covariance criterion. For PLS regression, examine prediction errors on data not used to fit the model, alongside residuals and the stability of predictions under reasonable data perturbations. A strong training association alone is not evidence of accurate out-of-sample prediction.

A common mistake is to say, “The first two-block PLS component has the largest covariance, so it must be the best predictor of the clinical response.” That conclusion does not follow. The maximizing covariance could be driven by a different response variable, and covariance is not a prediction-error measure. The correction is to name the objective explicitly: two-block PLS maximizes covariance between paired scores; PLS regression uses a designated response and should be assessed for prediction using an appropriate held-out evaluation.

Another important alternative is canonical correlation analysis (CCA), which constrains score variances rather than simply imposing unit lengths on the weight vectors. That difference changes the optimization problem; it should not be conflated with the PLS covariance criterion. Neither method removes the need to examine scaling, sample size, collinearity, or stability. With many variables and few subjects, estimated cross-covariances may be unstable. Components can also be difficult to interpret when variables are highly correlated or when preprocessing choices substantially change the result.

PLS regression is not a single universal multi-component algorithm: implementations differ in component extraction and deflation. Consequently, agreement of the first directions under matching objectives does not imply identical later components or fitted models. The treatment of those later directions is a separate methodological choice. For prediction, report the response definition, preprocessing, number of components, and validation design; for paired-block association, report how scores were defined and what covariance was maximized.

Key takeaway: Ask “which paired scores covary most?” for two-block PLS, and “how can $X$ predict the designated $Y$?” for PLS regression. The first-component formulas may coincide under matching definitions, but their goals and evaluation criteria remain distinct.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

What quantity does first-component two-block PLS maximize under unit-length weight constraints?
For a centered single-response PLS regression with $X^\mathsf{T}y=(3,4)^\mathsf{T}$, what is a unit maximizing weight vector?
Can the first component of PLS regression be identical to the first component of two-block PLS? State conditions under which this can occur.
A two-block analysis finds the largest score covariance for $Xw$ and $Yc$. What additional evidence is needed before claiming that $Xw$ predicts a clinical response well?

Solutions. (1) It maximizes $w^\mathsf{T}S_{XY}c$, equivalently its absolute value, over unit-length $w$ and $c$. (2) The norm of $(3,4)^\mathsf{T}$ is 5, so $w=(3/5,4/5)^\mathsf{T}$, up to sign. (3) Yes. With the same centered blocks, scaling, weight constraints, and cross-covariance SVD criterion, the first directions can coincide; for a single response, the response block has just one direction. (4) Evaluate predictions for the designated response on data not used to fit the model, using a suitable prediction-error measure, and check that the target was actually the response used to construct the regression model.