Orthogonality in the JIVE Decomposition

Why Orthogonality Matters in JIVE

In multi-omics data, a sample pattern can appear in several blocks, while other patterns may be confined to one block. Joint and Individual Variation Explained (JIVE) represents these contributions separately. The separation is meaningful only if the individual component cannot reuse a direction already assigned to the joint component. Orthogonality provides a precise constraint for that separation: it distinguishes sample directions, not necessarily feature directions or biological mechanisms.

We use a feature-by-sample convention. For block \(k\), \(X_k\) is a \(p_k \times n\) data matrix with the same \(n\) aligned samples across blocks and \(p_k\) features. After any chosen preprocessing, JIVE writes \(X_k=J_k+I_k+E_k\), where \(J_k\) is joint structure, \(I_k\) is block-specific individual structure, and \(E_k\) is residual variation. The focus here is the geometry of \(J_k\) and \(I_k\), with their ranks treated as fixed.

Definition: A JIVE representation has a common joint sample space spanned by the columns of \(V\), where \(V\) is an \(n \times r\) matrix with orthonormal columns. A convenient parameterization is \(J_k=C_kV^T\), with \(C_k\) a \(p_k \times r\) matrix. The individual component is \(I_k=D_kW_k^T\), where \(W_k\) is an \(n \times q_k\) matrix with orthonormal columns and \(D_k\) is \(p_k \times q_k\). The orthogonality constraint is \(V^TW_k=0\) for every block \(k\). Thus every individual sample direction is perpendicular to the full common joint sample space.

The stacked joint matrix \(J=[J_1^T,\ldots,J_K^T]^T\) has row space \(\operatorname{span}(V)\) when the vertically stacked coefficient matrix formed from the \(C_k\)'s has column rank \(r\). This full-rank condition matters: without it, the parameter \(V\) can include directions that do not actually occur in the stacked joint component. In that case the identifiable joint sample space is the row space of \(J\), and orthogonality should be defined against that actual space rather than redundant directions in a chosen parameterization.

The usual centered-data setup also requires the same samples to be aligned in every block. If each block is centered across samples, its fitted joint and individual sample directions should be centered as well: \(\mathbf{1}_n^TV=0\) and \(\mathbf{1}_n^TW_k=0\). Centering and scaling choices affect the fitted decomposition, but they do not change the meaning of the orthogonality constraint. It is a geometric restriction on the sample-score spaces.

What the Constraint Implies

Since the columns of \(V\) and \(W_k\) are orthonormal, \(VV^T\) is the orthogonal projector onto the joint sample space. The condition \(V^TW_k=0\) says that every column of \(W_k\) lies in the orthogonal complement of that space. Consequently, the joint and individual matrices in each block have zero Frobenius inner product. The Frobenius inner product of two same-sized matrices \(A\) and \(B\) is \(\langle A,B\rangle_F=\operatorname{tr}(A^TB)\).

\begin{aligned} \langle J_k,I_k\rangle_F &=\operatorname{tr}\!\left((C_kV^T)^T(D_kW_k^T)\right)\\ &=\operatorname{tr}\!\left(V C_k^T D_k W_k^T\right)\\ &=\operatorname{tr}\!\left(W_k^T V C_k^T D_k\right)=0, \end{aligned}

The final equality follows from \(W_k^TV=(V^TW_k)^T=0\). In particular, the squared Frobenius norm of the fitted signal \(J_k+I_k\) separates into the two squared norms:

\|J_k+I_k\|_F^2=\|J_k\|_F^2+\|I_k\|_F^2.

This is a consequence of the constraint, not a claim that the residual \(E_k\) is orthogonal to either fitted component. Nor does orthogonality mean that the joint and individual components have unrelated feature loadings. It means their sample directions occupy perpendicular subspaces.

Key result: \(V^TW_k=0\) guarantees both that the individual space is orthogonal to the full common joint space and that \(J_kI_k^T=0\). The matrix product condition is only a necessary consequence; by itself, it does not establish orthogonality to the full joint space.

Worked Examples

Worked Example: Verifying Orthogonal Sample Directions

Suppose a hypothetical analysis has three centered samples. Let the joint score direction be \(v=(1,-1,0)^T/\sqrt{2}\) and the individual score direction for one block be \(w=(1,1,-2)^T/\sqrt{6}\). Let that block have one feature, with \(C=2\) and \(D=3\), so \(J=2v^T\) and \(I=3w^T\).

Check the assumptions and constraint. Both score vectors have squared length one: \(v^Tv=2/2=1\) and \(w^Tw=6/6=1\). They are centered because their entries each sum to zero. Their inner product is \(v^Tw=(1-1+0)/\sqrt{12}=0\), so the specified individual direction is perpendicular to the joint direction.

Compute the components and their inner product. The joint row is \(J=(\sqrt{2},-\sqrt{2},0)\), and the individual row is \(I=(3/\sqrt{6},3/\sqrt{6},-6/\sqrt{6})\). Directly, \(JI^T=(\sqrt{2})(3/\sqrt{6})+(-\sqrt{2})(3/\sqrt{6})+0=0\). Equivalently, the general identity gives \(JI^T=2(v^Tw)3=0\).

The example verifies the constraint and its consequence. It does not establish that orthogonality of this one pair of component rows would always detect every overlap with a higher-dimensional joint space.

Worked Example: Why a Zero Product Can Miss Overlap

Consider two blocks and three centered samples, with \(v=(1,-1,0)^T/\sqrt{2}\). Suppose the stacked joint component has \(J_1=0\) in block 1 and \(J_2=v^T\) in block 2. Its full sample space is therefore \(\operatorname{span}(v)\). Now let block 1 have individual component \(I_1=v^T\).

The full-space orthogonality check fails: with \(V=v\) and \(W_1=v\), we obtain \(V^TW_1=v^Tv=1\), not zero. The individual direction in block 1 coincides with a direction in the stacked joint space.

Yet the blockwise product is \(J_1I_1^T=0\cdot v=0\), because block 1 has no joint component. A zero product here says nothing about whether \(I_1\) overlaps the joint direction contributed by block 2. This is why the correct diagnostic compares \(W_k\) with the full joint space, not merely with the joint component in the same block.

Worked Example: A Repeated Direction Makes a Split Ambiguous

For one hypothetical block with three centered samples, let \(v=(1,-1,0)^T/\sqrt{2}\), and suppose the observed signal is \(X=3v^T\). An unconstrained split could assign \(J=2v^T\) and \(I=v^T\). It reconstructs the data exactly, since \(J+I=3v^T=X\), but the proposed individual direction is the joint direction: \(v^Tv=1\).

The split is therefore not a valid orthogonal JIVE separation. The repeated direction can instead be represented as \(J=3v^T\) and \(I=0\), which also reconstructs \(X\) exactly and satisfies the orthogonality requirement (the zero individual component has no nonzero directions to check). The example illustrates the role of the constraint: exact reconstruction alone does not determine which part of a signal is joint versus individual. The orthogonality restriction rules out assigning the same sample direction to both parts.

Projection View and a Short Derivation

The geometry can be made explicit using the projector \(P_J=VV^T\) and its complement \(P_J^\perp=I_n-VV^T\), where \(I_n\) is the \(n\times n\) identity matrix. For an individual score vector \(w\), the constraint \(V^Tw=0\) is equivalent to \(P_Jw=0\). Indeed, \(V^Tw=0\) implies \(VV^Tw=0\); conversely, if \(VV^Tw=0\), then multiplying by \(V^T\) and using \(V^TV=I_r\) gives \(V^Tw=0\).

Thus a sample pattern \(z\) can be split into its projection onto the joint space and its orthogonal remainder:

$$ z=VV^Tz+(I_n-VV^T)z. $$

The two pieces are perpendicular because \((VV^Tz)^T(I_n-VV^T)z=z^TVV^T(I_n-VV^T)z=0\); here \(V^TV=I_r\) makes \(VV^T(I_n-VV^T)=0\). This establishes the projection-based justification for excluding joint directions from individual score spaces. It does not imply that every observed data matrix is perfectly reconstructed by the retained components; residual structure can remain in \(E_k\).

Diagnostics, Alternatives, and Limits

For a fitted model, inspect the geometry of the score bases directly. When their columns are orthonormal, calculate \(\|V^TW_k\|_F\) for each block. Exact orthogonality gives zero; in numerical computation, a small value relative to the calculation’s precision indicates approximate satisfaction. Also verify the basis conditions \(V^TV=I_r\) and \(W_k^TW_k=I_{q_k}\). If the stacked coefficient matrix is rank deficient, determine the actual row space of the stacked joint matrix before interpreting the purported full joint basis.

Large values of \(\|V^TW_k\|_F\) indicate that an individual score direction overlaps the joint space, or that the fitted bases have not been made orthogonal as required. A small \(J_kI_k^T\) is not a substitute: as the second example shows, it may be small or zero simply because the block’s own joint coefficient is zero or weak. Numerical near-zero checks should be interpreted with the scale and numerical tolerance of the fitted matrices in mind.

Approach	How it separates structure	What orthogonality contributes
JIVE with fixed ranks	Represents joint and block-specific components with separate low-rank terms.	Prevents individual sample directions from reusing the full joint sample space.
Unconstrained low-rank split	Fits multiple components without requiring their sample spaces to be perpendicular.	Without the constraint, a shared direction can be assigned to both terms, making the split ambiguous.
Separate PCA by block	Finds low-rank directions within each block independently.	Does not itself impose a common joint space or separate it from block-specific directions.

These approaches answer different questions; orthogonality does not guarantee that JIVE is the scientifically best representation. The constraint is a modeling choice that makes the joint/individual distinction precise, but results can still depend on preprocessing, the selected ranks, noise, and whether the shared sample patterns are well represented by a low-rank space. It does not prove that a direction is biologically shared in a causal sense. Rank selection and broader model diagnostics require additional decisions beyond checking orthogonality.

Common Mistakes

Checking only \(J_kI_k^T=0\). This condition follows from orthogonality, but its converse can fail, particularly when \(J_k\) is zero or does not represent every direction in the stacked joint space. Check \(V^TW_k\) against the full joint space.
Calling the components orthogonal without naming the space. The JIVE constraint concerns sample-score directions. It does not assert that feature loadings in different blocks are orthogonal.
Assuming reconstruction identifies the split. A decomposition can reconstruct \(X_k\) exactly while assigning the same pattern to both joint and individual terms. Reconstruction and orthogonal separation are distinct requirements.
Treating orthogonality as a biological conclusion. It is a property of the fitted representation, not evidence of independent mechanisms or absence of association.

Key takeaway: JIVE separates structure by requiring each block-specific sample-score space to be perpendicular to the full common joint sample space. This constraint implies zero joint–individual Frobenius inner product, but a zero blockwise matrix product alone does not verify the required full-space orthogonality.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In the feature-by-sample convention, what does \(V^TW_k=0\) assert geometrically?
Why does \(V^TW_k=0\) imply \(\langle J_k,I_k\rangle_F=0\) under the stated JIVE parameterization?
A fitted block has \(J_k=0\) and \(I_k\ne0\), while another block contributes a joint direction that overlaps the first block’s individual direction. Can \(J_kI_k^T=0\) still hold?
What rank condition makes the columns of \(V\) span the row space of the stacked joint matrix in the parameterization \(J_k=C_kV^T\)?

Solutions. (1) Every individual score direction is perpendicular to every direction in the common joint sample space. (2) Substitution gives \(\langle J_k,I_k\rangle_F=\operatorname{tr}(W_k^TV C_k^TD_k)=0\), since \(W_k^TV=0\). (3) Yes. The product is zero because \(J_k=0\), even though the individual direction can overlap a joint direction contributed by another block; the full-space check \(V^TW_k=0\) detects that overlap. (4) The vertically stacked coefficient matrix with blocks \(C_k\) must have column rank \(r\).