Cross-Block Covariance and Correlation Matrices

Why Cross-Block Covariance Matters

In multi-omics data, two measurement blocks may describe the same biological samples while recording different features. For example, one block might contain gene-expression measurements and another metabolite abundances. Their within-block covariances summarize relationships among features inside each block; their cross-block covariance summarizes how features from one block vary with features from the other. These are descriptive building blocks for later multivariate analyses, not evidence by themselves that one feature causes another.

The essential requirement is that the rows in the two blocks refer to the same observations in the same order. Once the rows are matched, each cross-block covariance entry is an ordinary sample covariance between one feature in the first block and one feature in the second.

Definition: Let $X$ be an $n\times p$ data matrix and $Y$ an $n\times q$ data matrix. Row $i$ in both matrices must represent the same observation. Let $X_c$ and $Y_c$ be their column-centered versions, formed by subtracting each column's sample mean from its observations. The within-block sample covariance matrices and the cross-block sample covariance matrix are $$ S_{XX}=\frac{X_c^\mathsf{T}X_c}{n-1},\qquad S_{YY}=\frac{Y_c^\mathsf{T}Y_c}{n-1},\qquad S_{XY}=\frac{X_c^\mathsf{T}Y_c}{n-1}. $$ Here $S_{XX}$ is $p\times p$, $S_{YY}$ is $q\times q$, and $S_{XY}$ is $p\times q$. Entry $(j,k)$ of $S_{XY}$ is the sample covariance between feature $j$ of $X$ and feature $k$ of $Y$.

The sample covariance between two columns $x_j$ and $y_k$ is their centered cross-product divided by $n-1$. The denominator $n-1$ is the usual sample-covariance convention. The matrices $S_{XX}$ and $S_{YY}$ are symmetric; $S_{XY}$ generally is rectangular and need not be symmetric. Reversing the block order transposes the cross-block matrix: $S_{YX}=S_{XY}^\mathsf{T}$.

Calculating the Matrices

Worked Example: Two Features in Each Block

Consider three hypothetical matched samples. The two columns in $X$ are $x_1=(1,2,3)$ and $x_2=(2,0,1)$; the two columns in $Y$ are $y_1=(2,4,6)$ and $y_2=(3,1,2)$. Each row of $X$ is paired with the corresponding row of $Y$. Calculate the within-block and cross-block covariance matrices.

The column means are $\bar{x}_1=2,\ \bar{x}_2=1,\ \bar{y}_1=4,\ \bar{y}_2=2$. Thus the centered columns are $x_{1c}=(-1,0,1)$, $x_{2c}=(1,-1,0)$, $y_{1c}=(-2,0,2)$, and $y_{2c}=(1,-1,0)$. Since $n-1=2$, each centered cross-product is divided by 2.

For $S_{XX}$, the variances are $(1^2+0^2+1^2)/2=1$ and $(1^2+(-1)^2+0^2)/2=1$. The covariance is $\{(-1)(1)+0(-1)+1(0)\}/2=-0.5$. Therefore

S_{XX}= \begin{pmatrix} 1 & -0.5\\ -0.5 & 1 \end{pmatrix}.

For $S_{YY}$, the variances are $(4+0+4)/2=4$ and $(1+1+0)/2=1$. Their covariance is $\{(-2)(1)+0(-1)+2(0)\}/2=-1$, giving

S_{YY}= \begin{pmatrix} 4 & -1\\ -1 & 1 \end{pmatrix}.

The four cross-block entries are $(x_{1c}^\mathsf{T}y_{1c})/2=4/2=2$, $(x_{1c}^\mathsf{T}y_{2c})/2=-1/2=-0.5$, $(x_{2c}^\mathsf{T}y_{1c})/2=-2/2=-1$, and $(x_{2c}^\mathsf{T}y_{2c})/2=2/2=1$. Hence

S_{XY}= \begin{pmatrix} 2 & -0.5\\ -1 & 1 \end{pmatrix}.

For example, the first entry says that $x_1$ and $y_1$ have sample covariance 2 in their respective measurement units. In these particular data, $y_1=2x_1$ for every sample, so this positive covariance reflects an exact positive linear relationship in the three observed pairs.

Correlation Matrices and Units

Covariance retains the units of both variables: if one feature is measured in micromoles per litre and another in milligrams per litre, their covariance uses the product of those units. This matters for interpretation and means that changing a feature's scale changes its covariance. Correlation standardizes each covariance by the two sample standard deviations, making the result dimensionless and bounded between $-1$ and $1$.

Definition: When all included features have positive sample variance, let $D_X$ and $D_Y$ be diagonal matrices containing the diagonal entries of $S_{XX}$ and $S_{YY}$. The within-block sample correlation matrices and cross-block sample correlation matrix are $$ R_{XX}=D_X^{-1/2}S_{XX}D_X^{-1/2},\qquad R_{YY}=D_Y^{-1/2}S_{YY}D_Y^{-1/2},\qquad R_{XY}=D_X^{-1/2}S_{XY}D_Y^{-1/2}. $$ Each entry of $R_{XY}$ is the sample correlation for a pair of features, one from each block. The diagonal entries of each within-block correlation matrix are 1.

These formulas amount to dividing each covariance by the product of the relevant sample standard deviations. Correlation allows comparisons across measurement scales, but it does not retain the original units or indicate the size of a change in those units. Neither covariance nor correlation establishes a causal relationship.

Worked Example: A Cross-Block Matrix with Two Outcomes

For four hypothetical matched samples, suppose the centered $X$ column is $x_c=(-1.5,-0.5,0.5,1.5)$, and the two centered $Y$ columns are $y_{1c}=(-1.5,-2.5,0.5,-0.5)$ and $y_{2c}=(-2,0,0,2)$. Calculate $S_{XX}$, $S_{YY}$, and $S_{XY}$, then standardize the cross-block entries.

Here $n-1=3$. The within-$X$ variance is $(2.25+0.25+0.25+2.25)/3=5/3$. The within-$Y$ variances are $(2.25+6.25+0.25+0.25)/3=3$ and $(4+0+0+4)/3=8/3$. The covariance between the $Y$ columns is $\{(-1.5)(-2)+(-2.5)(0)+(0.5)(0)+(-0.5)(2)\}/3=2/3$. Thus

S_{XX}=(5/3),\qquad S_{YY}= \begin{pmatrix} 3 & 2/3\\ 2/3 & 8/3 \end{pmatrix}.

The two cross-products with $X$ are $(-1.5)(-1.5)+(-0.5)(-2.5)+(0.5)(0.5)+(1.5)(-0.5)=3$ and $(-1.5)(-2)+(-0.5)(0)+(0.5)(0)+(1.5)(2)=6$. After division by 3, $S_{XY}=(1,\ 2)$. The corresponding cross-correlations are

R_{XY}= \left( \frac{1}{\sqrt{(5/3)(3)}},\ \frac{2}{\sqrt{(5/3)(8/3)}} \right) = (0.4472,\ 0.9487).

In these four pairs, $X$ has a moderate positive sample correlation with $Y_1$ and a stronger positive sample correlation with $Y_2$. The entries are standardized associations; they do not imply that changing $X$ would change either $Y$ feature.

A Useful Matrix Check

The within- and cross-block matrices belong together in a single covariance matrix for the combined features. This gives a useful check on calculations and clarifies why the cross-block matrix has the orientation shown above.

Result: The sample covariance matrix of all $p+q$ features, with columns ordered as $X$ followed by $Y$, is $$ S= \begin{pmatrix} S_{XX} & S_{XY}\\ S_{XY}^\mathsf{T} & S_{YY} \end{pmatrix}. $$ It is symmetric and positive semidefinite: for every vector $a\in\mathbb{R}^{p+q}$, $a^\mathsf{T}Sa\geq 0$.

To justify the result, concatenate the centered matrices to form $Z_c=[X_c\ Y_c]$. By the sample covariance definition,

S=\frac{Z_c^\mathsf{T}Z_c}{n-1}.

Partitioning this matrix product into its $X$ and $Y$ columns gives the displayed blocks. For any $a$, the quadratic form is $a^\mathsf{T}Sa=(Z_ca)^\mathsf{T}(Z_ca)/(n-1)=\|Z_ca\|^2/(n-1)\geq0$. Thus symmetry and positive semidefiniteness follow directly from the construction. A computed combined covariance matrix with a materially negative quadratic direction signals an implementation or data-handling problem; tiny numerical discrepancies can arise from rounding.

Worked Example: Comparing Covariance and Correlation

Suppose one hypothetical $X$ feature is measured in arbitrary concentration units, with centered values $(-1.5,-0.5,0.5,1.5)$. Two centered $Y$ features have values $y_{1c}=(-0.5,1.5,-1.5,0.5)$ and $y_{2c}=(-15,-5,5,15)$. Calculate their cross-covariances and correlations with $X$.

There are four matched samples, so divide centered cross-products by 3. The variance of $X$ is $5/3$. The cross-product for $Y_1$ is $(-1.5)(-0.5)+(-0.5)(1.5)+(0.5)(-1.5)+(1.5)(0.5)=0$; therefore its covariance with $X$ is 0 and its correlation is 0.

For $Y_2$, the cross-product is $(-1.5)(-15)+(-0.5)(-5)+(0.5)(5)+(1.5)(15)=50$, so the covariance is $50/3$. The variance of $Y_2$ is $(225+25+25+225)/3=500/3$. Its correlation with $X$ is

\frac{50/3}{\sqrt{(5/3)(500/3)}}=1.

The covariance $50/3$ depends on the measurement scales. The correlation of 1 indicates a perfect positive linear relationship among these observed pairs: $Y_2$'s centered values are ten times $X$'s. Standardizing makes that relationship easy to compare with the zero correlation for $Y_1$, but the standardized value alone does not express the ten-to-one scale relationship.

Assumptions, Diagnostics, and Alternatives

For descriptive sample matrices, the formulas require finite, correctly paired observations and consistent feature definitions. To treat these matrices as estimates of population covariances, a standard setting is an independent, identically distributed sample of matched multivariate observations with finite second moments. Dependence among features within a sample is allowed; that dependence is what the matrices describe. Inference about population values requires additional methods and assumptions beyond simply calculating these sample matrices.

Before interpreting the entries, check that both blocks have the same number of rows and that row $i$ refers to the same sample in each. Confirm that features are consistently measured and that columns were centered separately. Then inspect the diagonal variances, extreme observations, and the scale and units of each feature. An exact or near-constant feature has zero or nearly zero variance: its correlation is undefined or numerically unstable, so it should not be standardized as if its standard deviation were positive.

Missing values require particular care. Calculating each covariance from a different subset of pairs can produce entries based on different sample sizes. The resulting assembled matrix need not be positive semidefinite, even if each individual covariance calculation is correct for its own subset. A common complete-case analysis preserves a common set of matched rows, but may discard substantial data; other missing-data approaches require an explicit model or justified imputation strategy.

Summary	Useful when	Important limitation
Covariance matrices	Measurement units and absolute scale matter, or the analysis uses covariance directly.	Values depend on the units and scales of the features.
Correlation matrices	Comparing linear associations across differently scaled features.	Standardization removes scale information and requires positive variances.
Pairwise covariance summaries	Exploring individual feature pairs when observations are missing.	Different pairs may use different samples, so the combined matrix may not be a valid covariance matrix.

A frequent mistake is to multiply $X_c^\mathsf{T}$ by $Y_c$ after the two blocks have been sorted independently, then interpret the result as biological cross-block covariance. The correction is to verify sample identifiers and align the rows first: each product in the cross-product sum must combine measurements from the same observation. Another mistake is to call a large covariance a strong association without checking scale. Report the units, inspect the corresponding correlation when appropriate, and remember that correlation describes linear association rather than causation.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

If $X$ has 5 columns and $Y$ has 3 columns, what are the dimensions of $S_{XX}$, $S_{YY}$, and $S_{XY}$?
Why does permuting the rows of $Y$ without applying the same permutation to $X$ invalidate the cross-block covariance?
A feature has sample variance zero. Can its correlation with another feature be calculated by standardizing its covariance? Explain.
In one sentence, distinguish what a covariance entry and its corresponding correlation entry communicate.

Solutions. (1) The dimensions are $5\times5$, $3\times3$, and $5\times3$, respectively. (2) The cross-product would combine measurements from different observations rather than matched pairs, so it would not estimate the intended cross-block relationship. (3) No. Standardization divides by the feature's sample standard deviation, which is zero, so the correlation is undefined. (4) Covariance gives the direction and magnitude of joint variation in the product of the features' units, while correlation gives their standardized, unitless linear association.