Why the Distinction Matters
A multi-omics analysis may produce both a scientific finding and a procedure for using that finding. For example, researchers may report a molecular subtype associated with a clinical outcome and develop a classifier that assigns new samples to that subtype. Applying the classifier to new data and asking whether the reported association appears again are related but different tasks.
The distinction matters because the claims are not interchangeable. Evidence that a classifier performs adequately in new samples supports a claim about that procedure under the evaluation conditions. Evidence that an association appears again in independent data supports a claim that the finding is corroborated. Neither claim, by itself, proves that the association is causal, universally applicable, or biologically meaningful.
These terms are not used identically in every field. Some reports use “validation” broadly, including work that could also be described as replication. The practical remedy is to state what was tested, what was considered new and independent, and what result would count as support. The name alone is not enough to establish the strength or type of evidence.
State the Claim Before Evaluating the Evidence
A prespecified claim is stated before the new data are used to choose or modify the analysis in response to its results. It may describe a finding, such as “the abundance of feature X differs between groups,” or a procedure, such as “this scoring rule assigns samples to the reported subtype.” A useful evaluation also identifies in advance what evidence will be considered supportive. Otherwise, many analyses can be tried and only the favourable one reported.
The key question is: what would a positive result support? If the analysis applies an already described procedure to new data and evaluates a stated criterion, it supports a validation claim about that procedure. If the analysis asks whether a previously reported scientific finding is seen again in independent data, it supports a replication claim about that finding. One study can address both, but each claim requires its own clear description and evidence.
| Question | Validation emphasis | Replication emphasis |
|---|---|---|
| What is being assessed? | A prespecified finding or procedure, often its performance in new data | A previously reported finding |
| What supports the claim? | The new-data result meets a stated evaluation criterion | Independent evidence is consistent with the stated finding |
| What does the claim not establish by itself? | That the finding is true in every setting or that a procedure explains a biological mechanism | That the finding is causal or that a procedure performs adequately for its intended use |
Worked Examples
Worked Example: Validating a Subtype Classifier
Suppose a discovery analysis defines subtype A and a classifier for assigning samples to A or not A. Before looking at a new hypothetical set of 80 samples, investigators specify that classifier performance will be evaluated against reference labels assigned independently of the classifier. The reference labels identify 20 samples as subtype A and 60 as not A. The classifier correctly assigns 18 of the 20 A samples and 54 of the 60 non-A samples.
The sensitivity, the proportion of reference A samples assigned to A, is
The specificity, the proportion of reference non-A samples assigned to not A, is
The overall proportion correctly assigned is \((18+54)/80=72/80=0.90\), or 90%. If the prespecified criterion were at least 85% overall agreement, this result meets that criterion. The supported conclusion is that the classifier met this particular performance criterion in these new samples, relative to the stated reference labels. This does not, on its own, replicate every biological claim about subtype A. The interpretation also depends on whether the reference labels are suitable and whether the classifier and evaluation rule were not adjusted in response to these results.
Worked Example: Replicating a Feature Association
In a hypothetical discovery data set, researchers report that feature X has higher average abundance in group Y than in group Z. The group means are 6.1 and 4.9 measured units, respectively. In a separate hypothetical data set, collected and analysed as an independent assessment of this finding, the corresponding means are 5.4 and 4.6 measured units.
In the discovery data, the observed difference is \(6.1-4.9=1.2\) measured units. In the separate data, it is \(5.4-4.6=0.8\) measured units. Both observed differences are positive, so the direction agrees with the reported finding. This is evidence consistent with independent replication of the direction of the association, provided the feature, groups, and comparison have the same relevant meaning in both analyses.
The calculation does not establish that the association is causal. Nor does direction alone show that the second estimate is precise or that the magnitude is practically important. A replication claim should match the criterion actually assessed; here, the arithmetic supports a statement about observed direction, not a broader claim that every aspect of the original result has been confirmed.
Worked Example: Why a Reanalysis of the Discovery Data Is Not Replication
Suppose a discovery analysis examines 500 molecular features and selects the one with the strongest association with a clinical outcome. The selected feature has an estimated difference of 1.5 units. An analyst then reruns the same association calculation on the same 500 participants and obtains the same difference, 1.5 units.
The arithmetic agrees exactly: \(1.5-1.5=0\) units separates the two reported estimates. But the second calculation uses no new independent evidence. It reuses the data that selected the feature, so it cannot independently corroborate the finding. At most, the rerun may check that the calculation was implemented consistently. Calling it replication would confuse computational repeatability with corroboration by independent data.
Even a new random split of the original data should not automatically be described as independent replication: both subsets come from the same original data collection, and the feature was selected using information from that collection. The split may provide a form of internal evaluation, but the claim should say that rather than imply independent corroboration.
Assumptions and a Short Justification
A validation or replication claim is interpretable only when the claim and evidence are sufficiently well defined. The central conditions are conceptual as well as statistical:
- The claim is stated clearly. The reader can tell whether the analysis concerns a finding, a procedure, or both.
- The assessment is prespecified enough to limit result-driven choices. The finding or procedure and the evaluation criterion are not repeatedly altered until the new data look favourable.
- The new evidence is genuinely new for the claim being made. Reusing the discovery observations cannot independently corroborate a discovery result.
- Measurements and comparisons are interpretable. The feature, group definitions, outcome, or reference labels have a sufficiently comparable meaning for the stated assessment.
- The conclusion is limited to what was evaluated. Meeting a performance criterion is not proof of mechanism, and observing an association again is not proof of causation.
The reason validation and replication support different conclusions follows from what each evaluates. Let \(C\) denote a prespecified procedure, \(D_{\mathrm{new}}\) new data, and \(R(C,D_{\mathrm{new}})\) the result of applying the procedure and scoring it by a stated rule. If \(R(C,D_{\mathrm{new}})\) meets the criterion, the direct evidence concerns \(C\)'s performance under that assessment. It does not logically imply that a scientific finding \(F\) is independently corroborated, unless the assessment also tests \(F\).
whereas replication asks whether new independent evidence supports \(F\). This distinction is about the object of the claim, not about whether the analysis uses sophisticated statistics. A simple calculation can contribute to either task, but its interpretation depends on what was specified and what evidence it uses.
Diagnostics, Comparisons, and Limitations
Several warning signs indicate that the label may be stronger than the evidence. Check whether the analysis was changed after examining the new results; whether the feature, outcome, or subgroup was selected anew rather than evaluated as specified; whether the same observations contributed to discovery and assessment; and whether the stated criterion matches the result actually reported. A result that appears supportive only after trying many definitions or thresholds needs cautious interpretation. These checks do not automatically invalidate the analysis, but they affect how confidently it can be described as prespecified validation or independent replication.
Validation and replication also differ from nearby forms of evidence. Repeating a computation on the same data can check reproducibility of the calculation, but does not supply new evidence about the finding. Resampling or splitting a data set can assess aspects of internal stability or performance, but does not automatically provide independent replication. Conversely, an independent analysis that finds an association again may support replication of that finding without validating a specific classifier or pipeline.
In multi-omics work, comparability can be difficult: assays may measure related but non-identical quantities, and cohorts may differ in collection or processing. These issues can make an apparent disagreement hard to interpret, or make apparent agreement less informative than it seems. This tutorial’s definitions do not resolve those design questions; they establish why a report should be explicit about what was assessed and should avoid treating “validation” or “replication” as a blanket certificate of generality.
When reporting, name the claim directly: for example, “We evaluated the prespecified classifier in new samples against reference labels,” or “We assessed whether the reported feature association was present in independent data.” State what the result supports and what remains untested. Such wording is more informative than using either label without explanation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A prespecified classifier assigns 45 of 50 reference-positive samples correctly. What sensitivity does this estimate, and what kind of claim does it support?
- A reported feature association is observed again in separate data, but no classifier is evaluated. Is this evidence primarily validation of a procedure or replication of a finding?
- An analyst tests a selected feature again using the same participants used to select it. Why is that not independent replication?
- A classifier meets a prespecified accuracy threshold in new data. Does that result establish that the associated subtype causes a clinical outcome? Explain.
- In a discovery data set, group means are 3.8 and 3.1 units; in independent data, they are 4.2 and 3.7 units in the same group order. Calculate each difference and state the narrow conclusion supported by the directions.
Solutions. (1) Sensitivity is \(45/50=0.90\), or 90%; it supports a validation claim about classifier performance relative to the reference labels in the assessed samples. (2) It is primarily replication evidence for the finding, because the association is assessed again; no procedure’s performance is being tested. (3) The data are not new independent evidence: the same participants contributed to selection and reassessment, so the second calculation cannot independently corroborate the selected result. (4) No. The result concerns classifier performance under the stated assessment; it does not establish causation. (5) The discovery difference is \(3.8-3.1=0.7\) units, and the independent-data difference is \(4.2-3.7=0.5\) units. Both are positive, supporting agreement in observed direction, but not by themselves a causal or universal claim.