Deriving The Implied Cross-Omics Covariance

Why Derive the Covariance?

A multi-omics factor model describes each sample's measurements as combinations of latent scores and feature loadings. Its covariance implications make that description testable: they show which parts of the model account for variation within an omics view and which parts connect different views. In particular, shared factors can generate cross-omics covariance, while independent view-specific factors cannot.

The derivation depends on the model's independence assumptions. If view-specific factors are correlated across views, or if residual errors share a source, those terms can also contribute to cross-view covariance. The formulas therefore express both a model implication and a set of assumptions to assess.

Model and Assumptions

Consider \(M\) omics views measured on the same samples. For sample \(i\), let \(Y_{ij}\) be the \(p_j\)-dimensional feature vector in view \(j\), where \(j=1,\ldots,M\). Use the continuous, linear observation model

Y_{ij}=\mu_j+W_j Z_i+V_j H_{ij}+\varepsilon_{ij}.

Here, \(\mu_j\) is the mean vector; \(Z_i\) is a \(K\)-dimensional shared factor score; \(H_{ij}\) is a \(Q_j\)-dimensional factor score specific to view \(j\); and \(\varepsilon_{ij}\) is residual variation. The loading matrices \(W_j\) and \(V_j\) have dimensions \(p_j\times K\) and \(p_j\times Q_j\), respectively. Define \(\Sigma_Z=\operatorname{Cov}(Z_i)\), \(\Omega_j=\operatorname{Cov}(H_{ij})\), and \(\Psi_j=\operatorname{Cov}(\varepsilon_{ij})\). All scores and residuals have mean zero.

For the covariance formulas below, assume that the shared scores are uncorrelated with every view-specific score and residual; view-specific scores from different views are uncorrelated; residuals are uncorrelated with all scores; and residuals from different views are uncorrelated. These assumptions need not imply normality. The covariance calculations require finite second moments. The covariance matrices \(\Sigma_Z\), \(\Omega_j\), and \(\Psi_j\) are positive semidefinite; in a continuous factor model, \(\Psi_j\) is often taken to be diagonal, but diagonality is not needed for the derivation.

Definition: The within-view covariance matrix for view \(j\) is \(\operatorname{Cov}(Y_{ij})\). The cross-view covariance matrix for views \(j\) and \(\ell\) is \(\operatorname{Cov}(Y_{ij},Y_{i\ell})\), with entries equal to covariances between features in the two views. Its dimensions are \(p_j\times p_\ell\).

The centering by \(\mu_j\) does not affect covariance. It is useful to derive the result from \(Y_{ij}-\mu_j\), so that each term is a centered random vector.

Deriving the Covariance Blocks

For a within-view covariance, expand the covariance of the sum. The covariance of a sum includes each component's covariance and the cross-covariances between components. The stated uncorrelatedness assumptions make those cross terms zero.

\begin{aligned} \operatorname{Cov}(Y_{ij}) &=\operatorname{Cov}(W_j Z_i+V_j H_{ij}+\varepsilon_{ij})\\ &=W_j\Sigma_Z W_j^{T} +V_j\Omega_j V_j^{T} +\Psi_j. \end{aligned}

For example, \(\operatorname{Cov}(W_j Z_i)=W_j\operatorname{Cov}(Z_i)W_j^T\), by the covariance rule for a linear transformation. The other two terms follow by the same rule. The result separates variation associated with shared factors, view-specific factors, and residuals. All three terms are \(p_j\times p_j\) matrices.

For two distinct views \(j\) and \(\ell\), expand the cross-covariance. Under the assumptions, every cross-term involving a view-specific score or residual is zero. The shared factor is the only component present in both views:

\begin{aligned} \operatorname{Cov}(Y_{ij},Y_{i\ell}) &=\operatorname{Cov}(W_j Z_i+V_j H_{ij}+\varepsilon_{ij}, W_\ell Z_i+V_\ell H_{i\ell}+\varepsilon_{i\ell})\\ &=W_j\Sigma_Z W_\ell^{T}. \end{aligned}

This is the implied cross-omics covariance: shared scores create covariance through the two views' loadings on those scores. If \(\Sigma_Z=I_K\), the \(K\times K\) identity matrix, this reduces to \(W_jW_\ell^T\). That special case assumes uncorrelated shared scores with unit variance; it should not be silently substituted for a general \(\Sigma_Z\).

Derivation: For centered random vectors \(A\) and \(B\), \(\operatorname{Cov}(A,B)=E[AB^T]\). Thus the shared-factor contribution is \(E[(W_jZ_i)(W_\ell Z_i)^T]=W_jE[Z_iZ_i^T]W_\ell^T=W_j\Sigma_ZW_\ell^T\). For a view-specific contribution, independence or zero cross-covariance across views gives \(E[H_{ij}H_{i\ell}^T]=0\); the same argument removes the residual cross-term. Hence only the shared-factor term remains under the stated assumptions.

Worked Covariance Calculations

Worked Example: Within-View and Cross-View Matrices

Consider two hypothetical continuous omics views, each with two features, and one shared factor \(Z_i\) with variance \(3\). Let \(W_1=(2,1)^T\) and \(W_2=(1,-1)^T\). In view 1, let the specific-factor loading be \(V_1=(1,0)^T\), its score variance be \(2\), and \(\Psi_1=\operatorname{diag}(0.5,1)\). In view 2, let \(V_2=(0,1)^T\), its score variance be \(4\), and \(\Psi_2=\operatorname{diag}(1,0.25)\). Assume the shared score, specific scores, and residuals meet the independence conditions above.

For view 1, the shared contribution is

W_1(3)W_1^T =3\begin{pmatrix}2\\1\end{pmatrix}\begin{pmatrix}2&1\end{pmatrix} =\begin{pmatrix}12&6\\6&3\end{pmatrix}.

The specific contribution is \(V_1(2)V_1^T=\begin{pmatrix}2&0\\0&0\end{pmatrix}\). Adding it and the residual covariance gives

\operatorname{Cov}(Y_{i1}) =\begin{pmatrix}12&6\\6&3\end{pmatrix} +\begin{pmatrix}2&0\\0&0\end{pmatrix} +\begin{pmatrix}0.5&0\\0&1\end{pmatrix} =\begin{pmatrix}14.5&6\\6&4\end{pmatrix}.

For view 2, the three terms are

\begin{aligned} W_2(3)W_2^T&=\begin{pmatrix}3&-3\\-3&3\end{pmatrix},\\ V_2(4)V_2^T&=\begin{pmatrix}0&0\\0&4\end{pmatrix},\\ \operatorname{Cov}(Y_{i2})&=\begin{pmatrix}3&-3\\-3&3\end{pmatrix} +\begin{pmatrix}0&0\\0&4\end{pmatrix} +\begin{pmatrix}1&0\\0&0.25\end{pmatrix} =\begin{pmatrix}4&-3\\-3&7.25\end{pmatrix}. \end{aligned}

The cross-view covariance is determined by the shared score:

\operatorname{Cov}(Y_{i1},Y_{i2}) =W_1(3)W_2^T =3\begin{pmatrix}2\\1\end{pmatrix}\begin{pmatrix}1&-1\end{pmatrix} =\begin{pmatrix}6&-6\\3&-3\end{pmatrix}.

For instance, the covariance between feature 1 of view 1 and feature 2 of view 2 is \(-6\) in the product of their measurement units. Its negative sign follows from their opposite loading directions on the shared factor. The specific-factor and residual variances affect the within-view matrices but, under the assumptions, add nothing to this cross-view matrix.

Worked Example: A General Shared-Factor Covariance

Suppose a hypothetical pair of features, one from each of two views, loads on two shared factors. Their loading rows are \(w_1=(1,2)\) and \(w_2=(2,-1)\), and the shared-factor covariance is \(\Sigma_Z=\operatorname{diag}(4,1)\). Assume view-specific factors and residuals are uncorrelated across views.

The scalar cross-view covariance is

w_1\Sigma_Zw_2^T =(1,2)\begin{pmatrix}4&0\\0&1\end{pmatrix} \begin{pmatrix}2\\-1\end{pmatrix} =(1,2)\begin{pmatrix}8\\-1\end{pmatrix} =8-2=6.

The implied covariance is \(6\) in the product of the two feature units. Both factors contribute: the first contributes \(8\), while the second contributes \(-2\). A loading's sign alone does not determine the covariance; the contributions are combined, weighted by the factor covariance.

Worked Example: Zero Covariance Despite Shared Factors

Suppose two hypothetical features load on two independent shared factors, each with variance \(1\). Their loading rows are \(w_a=(1,1)\) and \(w_b=(1,-1)\). Then

\operatorname{Cov}(Y_a,Y_b) =w_a I_2 w_b^T =(1,1)\begin{pmatrix}1\\-1\end{pmatrix} =1-1=0.

Both features load on both shared factors, yet their contributions cancel in this pairwise covariance. Therefore, a zero cross-view covariance for a feature pair does not establish that the features have no shared latent drivers. Other feature pairs may still show nonzero covariance, and covariance is not a complete description of every possible dependence.

Block Structure and Interpretation

For two views, stacking their centered measurements yields a covariance matrix whose diagonal blocks are within-view covariances and whose off-diagonal blocks are cross-view covariances:

\operatorname{Cov} \begin{pmatrix}Y_{i1}\\Y_{i2}\end{pmatrix} = \begin{pmatrix} W_1\Sigma_ZW_1^T+V_1\Omega_1V_1^T+\Psi_1 & W_1\Sigma_ZW_2^T\\ W_2\Sigma_ZW_1^T & W_2\Sigma_ZW_2^T+V_2\Omega_2V_2^T+\Psi_2 \end{pmatrix}.

The lower-left block is the transpose of the upper-right block, as required for a covariance matrix. The full matrix must be positive semidefinite: for any vector \(a\), the variance \(a^T\operatorname{Cov}(Y_i)a\) cannot be negative. This provides a useful consistency check for a proposed or estimated block covariance structure.

A cross-view matrix is not generally symmetric, because its rows and columns index different feature sets. Its entry in row \(r\), column \(s\) is the covariance of feature \(r\) in view \(j\) with feature \(s\) in view \(\ell\). The units are the product of those features' units. If features are standardized, the covariance is expressed on the standardized scale; it is a correlation only when both variables have variance one.

Assumptions, Diagnostics, and Alternatives

The clean cross-view formula relies on shared samples and the absence of cross-view covariance among specific factors and residuals. If those conditions fail, the general expansion includes extra terms. For example, if specific scores in views \(j\) and \(\ell\) have covariance \(\Omega_{j\ell}\), then the cross-view matrix includes \(V_j\Omega_{j\ell}V_\ell^T\). Correlated residuals add \(\operatorname{Cov}(\varepsilon_{ij},\varepsilon_{i\ell})\) as well. A model that omits such terms may attribute their covariance to shared factors.

Check the measurement scale.
The linear covariance formulas apply directly to the continuous observation model stated here. For count or binary observations modeled through a nonlinear link, covariance of the observed data is not generally just the product of loading matrices; distinguish covariance on the latent scale from covariance on the observed scale.

Inspect residual cross-view structure.
After fitting the model, examine whether residual associations remain across views. Systematic residual covariance can indicate omitted shared structure, correlated errors, batch effects, or another violation of the assumed decomposition.

Compare implied and observed covariance patterns.
Large, structured discrepancies suggest the factor model does not capture the covariance adequately. Sampling noise can also create discrepancies, so assess them relative to sample size and uncertainty rather than treating each small difference as a failure.

An independent model for each view permits within-view covariance but implies zero cross-view covariance. It is appropriate only when cross-view dependence is not part of the target or is negligible. A joint model with shared and view-specific factors makes a structured decomposition of that dependence. A less structured joint covariance model can represent cross-view covariance without assigning it to shared factors, but gives less direct latent-factor interpretation and may require estimating many more covariance parameters. These approaches answer different modeling questions; the factor model's structure is an assumption, not a consequence of observing several omics blocks.

Common Mistake and Key Takeaway

Wrong reasoning: “The view-specific factors vary within each omics block, so they must contribute to covariance between blocks.” Correction: variation within one view does not itself create cross-view covariance. Under the model assumptions, a view-specific score is uncorrelated with the other view's specific score and residuals, so its cross-view terms are zero. If those scores or residuals are correlated across views, the zero-term conclusion no longer holds and the covariance formula must include the corresponding terms.

Key takeaway: Under a linear model with uncorrelated view-specific factors and residuals across views, within-view covariance is \(W_j\Sigma_ZW_j^T+V_j\Omega_jV_j^T+\Psi_j\), while cross-view covariance is \(W_j\Sigma_ZW_\ell^T\). The cross-view result follows from shared loadings and the model's independence assumptions together.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If \(\Sigma_Z=I_K\), what is the cross-view covariance between views \(j\) and \(\ell\)?
A pair of features has loading rows \((1,1)\) and \((1,-1)\) on two independent unit-variance shared factors. What is their implied cross-view covariance?
Suppose the specific scores in two views have nonzero covariance \(\Omega_{j\ell}\). What additional term enters the cross-view covariance?
Why is a cross-view covariance matrix not generally symmetric, and what are the units of its entries?

Solutions. (1) It is \(W_jW_\ell^T\), since multiplying by the identity leaves the loading product. (2) It is \(1(1)+1(-1)=0\); the two shared-factor contributions cancel. (3) The added term is \(V_j\Omega_{j\ell}V_\ell^T\). (4) The rows and columns index different feature sets, so swapping them produces the transpose block rather than the same block; each entry has units equal to the product of the two paired feature units.