Why modality coverage changes what can be validated
A validation cohort may include many participants and still be unable to test an important discovery claim if it did not measure the relevant omics modality. A cohort with RNA measurements but no protein or metabolite measurements can assess some transcript-level findings, but it cannot directly assess a protein association or a three-modality signature. The central planning task is therefore not simply to ask whether a cohort is large enough. It is to ask whether its measurements can test the same claim.
This distinction applies even when the validation cohort contains several modalities. A claim about the correlation between RNA and protein requires both measurements on the same observational units, whereas two separate groups measured for RNA and protein can support separate modality-specific analyses. Likewise, an integrated score that uses three modalities is not fully evaluated by calculating a score from only the two available modalities.
A modality-coverage matrix is a practical starting point. Let \(M_k\) denote the set of modalities measured in cohort \(k\), and let \(R(c)\) denote the modalities required by claim \(c\). The inclusion \(R(c)\subseteq M_k\) is a necessary check, not a guarantee: the measurements must also be on the relevant participants, the features must be usable, and the study design must support the claim.
For a claim involving a relationship between modalities, “measured in the cohort” is not enough: the relevant modalities generally must be measured on the same individuals. For a claim about an integrated predictor, the cohort must supply the inputs needed to compute the specified predictor, using transformations consistent with the validation plan. A missing modality is not an observed value of zero.
Assumptions and a decision rule
Before classifying a claim as directly testable, specify its target quantity, the feature or signature involved, the outcome or comparison, and the required measurement structure. Also state whether measurements are paired by participant. The validation cohort should be independent of the discovery data for an external validation claim, and its assays and sampling should be adequate for the target comparison. The measurement scale and preprocessing must permit a meaningful comparison with the original claim; a difference in population or protocol may affect transportability even when the modalities are present.
For a frozen signature, the required inputs and transformations matter as much as the modality labels. If the original score uses a feature that was not measured, the original score is not computable for that participant. Refitting a new score or changing its weights can answer a new question, but does not test the unchanged frozen score. This is the relevant distinction from the earlier material on frozen versus refit analyses.
A useful planning rule has three checks: observability (are all inputs available in the required participants?), estimand alignment (does the proposed analysis target the same quantity and contrast?), and design adequacy (can the cohort estimate that quantity with its sampling and measurement structure?). A “no” at any check means the original claim is not directly testable as stated. A narrower claim may still be testable, but it should be labelled as narrower rather than presented as full validation.
For example, distinguish a transcript’s association with disease status from a transcript–protein correlation or the performance of a multi-omics score.
Record which modalities and features the claim uses, and whether measurements must be available on the same participants.
Check modality presence, feature availability, participant overlap, and the outcome or comparison needed.
Label the claim directly testable, testable only in a clearly narrowed form, or not testable from that cohort’s measurements.
Why missing modalities block some claims
The limitation is not merely reduced precision. When a required modality is unobserved, a claim involving that modality may be unidentifiable from the observed data. For example, suppose a cohort records \(X\) and an outcome \(Y\), but not a metabolite measurement \(Z\). The observed data describe the distribution of \((X,Y)\). Different possible distributions of the unobserved \(Z\) can leave that observed distribution unchanged while implying different values of the RNA–metabolite association.
Consequently, the observed data alone cannot determine which value of that association holds. This is why an absent modality cannot be treated as evidence that its signal is absent, nor can a result for \(X\) alone establish a claim about the joint behavior of \(X\) and \(Z\). A proxy may be useful for another scientific question, but it does not reveal the unmeasured quantity without additional assumptions.
Worked Example: Sorting claims across two validation cohorts
Suppose a discovery analysis measured RNA, protein, and metabolites on the same participants. It made four claims: (1) RNA feature \(R\) is associated with disease status; (2) protein feature \(P\) is associated with disease status; (3) metabolite \(M\) is associated with disease status; and (4) \(R\) and \(P\) are correlated across participants. Validation cohort A measures RNA and protein on the same 180 participants. Cohort B measures RNA only on 140 participants. Assume each cohort has the same disease-status contrast, the stated features are measurable, and the samples are suitable for the planned analyses.
For claim (1), \(R(c)=\{\text{RNA}\}\). Both cohorts include RNA, so both can directly test the RNA–disease association, subject to comparable measurement and analysis.
For claim (2), \(R(c)=\{\text{protein}\}\). Cohort A can test the protein–disease association; cohort B cannot, because protein was not measured. The number 140 does not remedy the missing measurement.
For claim (3), \(R(c)=\{\text{metabolite}\}\). Neither cohort measured metabolites, so neither can directly test this claim.
For claim (4), the claim requires paired RNA and protein values. Cohort A has both modalities on the same 180 participants, so its measurement structure supports a direct test. Cohort B lacks protein and cannot test the correlation. In summary, the direct-test counts are \(2/2\) available cohorts for the RNA claim, \(1/2\) for the protein claim, \(0/2\) for the metabolite claim, and \(1/2\) for the RNA–protein correlation. These fractions describe coverage of the two named cohorts; they are not measures of statistical strength or validation success.
Worked examples: integrated scores and unpaired measurements
Worked Example: A partially observed multi-omics score
A discovery analysis defines a frozen score \(S=0.5Z_R+0.3Z_P+0.2Z_M\), where \(Z_R\), \(Z_P\), and \(Z_M\) are standardized RNA, protein, and metabolite inputs using the discovery reference means and standard deviations. In a validation cohort, RNA and protein are measured on the same participant, but the metabolite is not. For one hypothetical participant, \(Z_R=1.0\) and \(Z_P=0.5\).
The observed part of the score is \(0.5(1.0)+0.3(0.5)=0.50+0.15=0.65\). But the specified full score is \(S=0.65+0.2Z_M\), and \(Z_M\) is unavailable. Thus the full score cannot be calculated for this participant, and its validation-cohort performance cannot be evaluated as specified.
One might divide the observed contribution by the sum of its available weights: \(0.65/(0.5+0.3)=0.65/0.8=0.8125\). That value is a rescaled partial score, not the original score: it changes the weights and removes the metabolite contribution. It could support a separately defined partial-score analysis, but it does not directly validate the frozen three-modality score. This conclusion assumes the score is defined with all three stated terms; if the original score had a prespecified rule for missing inputs, that rule—not an improvised renormalization—would govern.
Worked Example: Separate assays do not establish a cross-modal relationship
A discovery claim states that RNA feature \(R\) and protein feature \(P\) are positively correlated within participants. A validation cohort measures RNA in 100 people and protein in 90 different people, with no participant measured for both. Assume the two groups come from the same target population and each assay is technically adequate.
The number of paired observations is the size of the intersection of the two participant sets: \(|I_R\cap I_P|=0\). Therefore, the observed paired fraction relative to the smaller assay group is \(0/90=0\). There are no participant-level \((R,P)\) pairs from which to estimate their correlation. The correlation claim is not directly testable in this cohort.
The cohort could still support separate RNA and protein analyses against a measured outcome, if those outcomes and contrasts are available. For illustration, suppose the estimated RNA log-fold change is \(0.30\) with \(SE=0.12\), and the estimated protein log-fold change is \(0.25\) with \(SE=0.10\), each under a suitable approximate normal sampling model. Their 95% intervals are, respectively, \(0.30\pm1.96(0.12)=0.30\pm0.2352=[0.0648,0.5352]\) and \(0.25\pm1.96(0.10)=0.25\pm0.196=[0.0540,0.446]\). These intervals describe uncertainty in the two separate effects, in log-fold-change units. They do not estimate or validate the within-person RNA–protein correlation.
Diagnostics, alternatives, and limitations
A coverage matrix should distinguish three different problems. A modality absent from the cohort makes claims requiring that modality unavailable. A modality may be present while a particular feature is missing or unusable, which blocks claims depending on that feature. Finally, two modalities may both be present but measured on nonoverlapping participants, which prevents direct testing of claims requiring paired values. Checking only a cohort-level list of assays can miss the third problem.
Several tempting substitutions change the scope of evidence:
- Using a proxy feature: a correlated or biologically related feature is not the original feature. Validation with a proxy is a separate claim and needs its own rationale.
- Dropping a score component: a reduced score or new set of weights defines a different predictor. It may be informative, but cannot stand in for the unchanged full score.
- Combining separate modality analyses into a joint claim: evidence that each feature has a marginal association does not establish their participant-level correlation, interaction, or joint predictive value.
- Treating “not measured” as “not associated”: no direct estimate was obtained, so the result is unvalidated rather than a negative finding.
The main alternatives are to validate the claims supported by the available modalities and label their scope, or to obtain a cohort with the missing measurements and required participant-level pairing. A proxy-based or partial-score analysis is another option only when its target is explicitly redefined. Cross-cohort pooling, including the earlier material on meta-analysis, cannot manufacture a missing modality: pooling modality-specific estimates does not by itself create the paired data needed for a cross-modal claim.
Useful diagnostics include a claim-by-cohort table, a feature-level inventory within each modality, counts of participants with each required combination of measurements, and a check that the outcome and contrast are observed. These checks diagnose whether a planned test matches the claim; they do not guarantee adequate power, measurement comparability, or transportability. Even when a claim is directly testable, differences in populations, protocols, or selection into the measured subset may limit interpretation. Coverage establishes an opportunity to test, not successful replication.
Check Your Understanding
For each question, identify what is directly testable and why. Solutions follow the questions.
- A discovery claim concerns a protein–outcome association. A validation cohort measures RNA and the outcome but no protein. Is the claim directly testable?
- A discovery claim concerns an RNA–protein correlation. A validation cohort measures both modalities, but on different participants. What blocks a direct test?
- A frozen score uses RNA and metabolite inputs. A cohort measures RNA only. Would evaluating an RNA-only score validate the frozen score?
- A cohort measures every modality required by a claim, but uses a different outcome contrast from discovery. Is modality coverage alone sufficient for direct validation?
Solutions. (1) No. Protein is required for the claim and was not measured; the RNA–outcome association, if estimable, is a different claim. (2) The lack of paired participant-level measurements prevents estimation of the within-person RNA–protein correlation. Separate modality measurements do not supply the pairs. (3) No. The RNA-only score omits a required input and defines a different score; it can be assessed only as a separately specified analysis. (4) No. Coverage is necessary but not sufficient. The different contrast may target a different quantity, so estimand alignment must be established before calling the result direct validation.