Estimating Cross-Omics Associations From Partial Overlap

Why the Overlap Population Matters

In a multi-omics study, one group of subjects may have measurements from one modality and another group from a second modality, while only some subjects have both. A cross-omics association requires paired values: each $X$ value must belong to the same subject as its corresponding $Y$ value. Measurements from different subjects cannot be paired merely because they appear in the same study.

The direct estimate from jointly observed subjects answers a clear question, but its scope is limited: it describes the association among subjects whose two modalities are observed. If those subjects differ systematically from the rest of the cohort, their association need not represent the association in the full cohort. The first step is therefore to define the target population, not to assume that complete pairs are representative.

Definition: For subject $i$, let $R_i=1$ if both modalities are observed and $R_i=0$ otherwise. The overlap population is the population distribution of subjects conditional on $R=1$. For features $X$ and $Y$, its covariance and correlation are $$ \operatorname{Cov}(X,Y\mid R=1) \quad\text{and}\quad \operatorname{Corr}(X,Y\mid R=1) = \frac{\operatorname{Cov}(X,Y\mid R=1)} {\sqrt{\operatorname{Var}(X\mid R=1)\operatorname{Var}(Y\mid R=1)}}. $$ The correlation is defined when both conditional variances are positive.

The covariance is expressed in the product of the units of $X$ and $Y$; the correlation is dimensionless and ranges from $-1$ to $1$. Both are descriptive of the overlap population. Neither, by definition alone, describes the full study population.

Estimating Association From Paired Measurements

Suppose there are $n_o$ subjects with $R_i=1$, and each has a measured pair $(X_i,Y_i)$. The pairwise-complete sample means are calculated using those same $n_o$ subjects. The sample covariance and Pearson sample correlation are

\bar{x}_o=\frac{1}{n_o}\sum_{i:R_i=1}X_i,\qquad \bar{y}_o=\frac{1}{n_o}\sum_{i:R_i=1}Y_i,

s_{XY,o}=\frac{1}{n_o-1}\sum_{i:R_i=1}(X_i-\bar{x}_o)(Y_i-\bar{y}_o), \qquad r_o= \frac{\sum_{i:R_i=1}(X_i-\bar{x}_o)(Y_i-\bar{y}_o)} {\sqrt{\sum_{i:R_i=1}(X_i-\bar{x}_o)^2 \sum_{i:R_i=1}(Y_i-\bar{y}_o)^2}}.

At least two paired subjects are needed to calculate a sample covariance, and the correlation also requires nonzero sample variation in both features. For statistical generalization from the observed pairs to the overlap population, a standard starting assumption is that the paired subjects are independent and identically distributed draws from that population, with finite second moments. Under these conditions, the sample covariance estimates the overlap covariance without bias, and both the sample covariance and sample correlation are consistent as $n_o$ grows. A sample correlation is not generally exactly unbiased in a finite sample.

Worked Example: Calculating a Pairwise-Complete Correlation

In a hypothetical study, five subjects have both an RNA feature $X$, measured in normalized units, and a protein feature $Y$, measured in normalized units. Their paired values are $(1,2),(2,1),(3,4),(4,3),(5,5)$. Assume these are independent subjects sampled from the overlap population, and that the feature measurements are on the same scales across the five subjects.

The sample means are $\bar{x}_o=3$ and $\bar{y}_o=3$. The deviations from each mean are $(-2,-1,0,1,2)$ for $X$, and $(-1,-2,1,0,2)$ for $Y$. Thus the sum of cross-products is $2+2+0+0+4=8$, and each sum of squared deviations is $4+1+0+1+4=10$.

Substitution gives the sample covariance and correlation:

s_{XY,o}=\frac{8}{5-1}=2.00 \quad\text{normalized-units}^2,\qquad r_o=\frac{8}{\sqrt{10\cdot10}}=0.800.

The paired observations show a positive linear association in these five subjects. This calculation estimates an overlap-population association only if the sampling assumption is reasonable; it does not establish that the same correlation holds among subjects missing one or both measurements.

How Selection Changes the Association

The overlap group is often a selected subset. Selection can change the distribution of either feature, and it can change the association even when the within-group relationship is stable. A useful way to see the issue is to condition on a group $S$, such as a measured severity category. The law of total covariance gives

\operatorname{Cov}(X,Y\mid R=1) = E\!\left[\operatorname{Cov}(X,Y\mid S,R=1)\mid R=1\right] + \operatorname{Cov}\!\left(E[X\mid S,R=1],E[Y\mid S,R=1]\mid R=1\right).

The first term averages within-group covariances using the group proportions among overlap subjects. The second term is the covariance between the group-specific means, also using the overlap group's composition. Consequently, knowing only the proportion of overlap subjects in each group is not enough to determine the overlap covariance. One also needs the conditional means and covariances among those subjects.

Justification: Write $X-\!E[X\mid R=1]$ as $(X-\!E[X\mid S,R=1])+(E[X\mid S,R=1]-E[X\mid R=1])$, and write the corresponding decomposition for $Y$. Taking the conditional expectation of the product yields the within-group covariance term and the covariance of the conditional means. The cross-terms have expectation zero because the residual from each group-specific mean has conditional expectation zero given $S,R=1$.

Worked Example: Selection by a Severity Group

Consider a hypothetical full population with a binary severity group $S$. Half the subjects are in each group. In group $S=0$, the means of $X$ and $Y$ are 0 and 0; in group $S=1$, they are 4 and 5. In both groups, the within-group covariance is 0.50, and the within-group variances of $X$ and $Y$ are 4 and 9. Assume selection into overlap is independent of $(X,Y)$ conditional on $S$. This assumption means the conditional means, covariances, and variances just stated also hold among overlap subjects within each severity group.

Suppose 80% of group $S=0$ and 20% of group $S=1$ have both modalities measured. Among overlap subjects, the proportion in group $S=1$ is

P(S=1\mid R=1) = \frac{0.50(0.20)}{0.50(0.80)+0.50(0.20)} = \frac{0.10}{0.50}=0.20.

Using the law of total covariance, the overlap covariance is the average within-group covariance plus the covariance of the group means. For a binary group with probability $q$ in group 1, the latter covariance is $q(1-q)(\mu_{X,1}-\mu_{X,0})(\mu_{Y,1}-\mu_{Y,0})$. Therefore,

\operatorname{Cov}(X,Y\mid R=1) =0.50+(0.20)(0.80)(4-0)(5-0) =3.70.

For comparison, the full-population covariance under the stated group distributions is $0.50+(0.50)(0.50)(4)(5)=5.50$. The overlap and full-population covariances differ because overlap subjects contain a smaller proportion of the high-severity group. Using the stated within-group variances, the overlap variances are $4+(0.20)(0.80)(4^2)=6.56$ for $X$ and $9+(0.20)(0.80)(5^2)=13.0$ for $Y$. The overlap correlation is consequently $3.70/\sqrt{6.56(13.0)}\approx0.401$. The full-population correlation is $5.50/\sqrt{8(15)}\approx0.502$. These are distinct targets, even though selection within each severity group leaves the conditional distributions unchanged.

The conditional independence assumption in that example is substantive. If selection depends on $X$ or $Y$ even after conditioning on severity, the overlap subjects' within-severity means or covariances may differ from the full population's. In that case the calculation using the full-population within-group quantities is not justified. The observed overlap proportion $q$ alone does not reveal those selected within-group quantities.

Worked Example: Why the Overlap Proportion Is Not Enough

Keep the severity-group means and the overlap proportion $q=0.20$ from the previous example. Suppose, however, that selection within severity groups is not independent of the measured features. In one hypothetical selection process, the overlap subjects' within-group covariance is 0.50 in each group. In another, it is 2.00 in each group. Assume for illustration that the overlap group-specific means remain 0 and 0 for $S=0$, and 4 and 5 for $S=1$, in both processes.

The between-group contribution remains $(0.20)(0.80)(4)(5)=3.20$. But the total overlap covariance differs:

\text{Process 1: }0.50+3.20=3.70, \qquad \text{Process 2: }2.00+3.20=5.20.

Both processes have the same overlap proportion and the same overlap group means, yet they yield different overlap covariances because their within-group covariances differ. If the group-specific means also changed with selection, the between-group contribution could change as well. Thus a group proportion cannot, by itself, supply the missing conditional information.

Interpretation, Diagnostics, and Alternatives

A pairwise-complete estimate is often the most transparent descriptive summary when the scientific target is explicitly the subjects with both modalities. To generalize to a broader target, the analyst must state and support additional assumptions about selection. For example, if overlap selection is independent of the measured pair conditional on observed covariates, conditional associations may be estimable within covariate strata; an overall target still requires accounting for the full population's covariate distribution. Inverse-probability weighting and imputation are alternative approaches covered in the earlier material on those subjects. They do not remove the need for defensible assumptions about modality availability.

Useful checks include reporting the number of paired subjects and the availability patterns; summarizing measured characteristics in the overlap and non-overlap groups; and checking whether the paired sample spans the ranges and groups relevant to the intended target. Examine scatterplots for nonlinear patterns, clusters, and influential observations, and verify that feature scaling and preprocessing are comparable across subjects. These checks can reveal sparse support or obvious selection differences, but they cannot prove that selection is independent of unobserved feature values.

In a high-dimensional analysis, a separate overlap set may be used for each feature pair. The resulting collection of pairwise covariances may not correspond to any single joint population covariance matrix and need not be positive semidefinite. This matters for procedures that require a coherent covariance matrix. Report the pair-specific sample sizes and targets rather than treating estimates from different overlaps as if all came from one common set of subjects. Dependence among observations, such as repeated measures or family clustering, also invalidates the simple independent-subject uncertainty calculations unless it is handled in the analysis.

Specify the target.
State whether the goal is association among jointly measured subjects or among a wider population.

Define the paired sample.
For each feature pair, identify subjects with both values observed and report the pair count.

Estimate and check.
Calculate the covariance or correlation using the paired values; assess variation, influential points, and observed differences between availability groups.

Limit the conclusion appropriately.
Describe the estimate as overlap-population evidence unless additional selection assumptions justify generalization.

A common mistake is to say, “Only 20% of the overlap subjects are in the severe group, so the covariance is determined by $q=0.20$.” That reasoning omits the within-group covariance and assumes, without justification, that selection leaves within-group feature distributions unchanged. The correction is to use the total-covariance decomposition and explicitly state any conditional selection assumption needed to identify its components. Another mistake is to combine an $X$ measurement from one subject with a $Y$ measurement from another. Such cross-subject pairing does not estimate a subject-level cross-omics association.

Key takeaway: Pairwise-complete covariance and correlation estimate association in the overlap population. Extending that interpretation to the full cohort requires assumptions about how joint modality availability relates to the features and to the target population's composition.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

What population does a covariance computed from subjects with both modalities observed describe by default?
In the severity example, why was it valid to use the same within-severity covariance among overlap subjects as in the full population?
With $q=0.25$, group-mean differences of 2 for $X$ and 3 for $Y$, and an overlap within-group covariance of 0.40 in both groups, calculate the overlap covariance.
Why can pairwise-complete estimates for many feature pairs fail to form a valid single covariance matrix?

Solutions. (1) It describes the overlap population, conditional on $R=1$, not automatically the full study population. (2) The example explicitly assumed selection into overlap was independent of $(X,Y)$ conditional on severity, so the conditional feature distributions, including their covariances, were unchanged within each severity group. (3) The within-group contribution is 0.40. The between-group contribution is $(0.25)(0.75)(2)(3)=1.125$. Thus the overlap covariance is $0.40+1.125=1.525$, in the product of the feature units. (4) Different feature pairs may use different subsets of subjects and therefore estimate associations for different overlap populations; the assembled entries need not describe one common joint distribution or yield a positive semidefinite matrix.

Estimating Cross-Omics Associations From Partial Overlap

What You'll Learn