Reporting Validation Design and Cohort Provenance

Why Design and Provenance Belong in a Validation Report

A validation result is interpretable only when readers can tell what data played the discovery role, what data played the validation role, and how those data were obtained. In multi-omics research, a cohort may be described by a study name, a hospital, a recruitment wave, or a data-release label. None of these labels alone establishes that its participants were independent of the discovery sample. Nor does a new assay, a new site, or a new analysis automatically create an independent validation cohort.

This tutorial focuses on reporting the design and provenance needed to evaluate a validation claim: cohort sources, recruitment periods, participant overlap checks, the basis for describing samples as independent, and the claim fixed before validation results were examined. It does not substitute provenance reporting for analysis or results reporting. The earlier material on replication criteria and failed replication supplies the context for why a clear claim matters; here the emphasis is on making the design auditable.

These definitions distinguish participant independence from other forms of separation. A validation cohort may be collected at another institution yet include participants recruited earlier into the discovery study. Conversely, two groups recruited at the same institution may have no participant overlap, though shared recruitment practices, staff, or processing infrastructure can still be relevant dependencies. Reports should state the evidence for independence, not use “independent” as an unexplained adjective.

What to Report About Cohort Provenance

For each discovery and validation cohort, identify the source in enough detail to distinguish it from other datasets: the parent study or registry, contributing institution or sites, the sampling frame, and how participants entered the data resource. State recruitment or sampling dates, preferably as the start and end month and year when available. If dates refer to specimen collection or data extraction rather than participant recruitment, label them accurately. A publication date or release date is not a substitute for a recruitment period.

Describe which data source supplied each role. For instance, a multi-omics project might use an earlier hospital-based cohort for discovery and a later community-based cohort for validation. The report should make clear whether the validation sample was prospectively recruited for the validation question, drawn from an existing resource, or assembled from a previously collected study. These facts help readers understand what “validation” means in the design without implying that the cohorts represent the same population.

Cohort provenance is not the same as a detailed table of participant characteristics or a modality-by-modality coverage summary. Those are separate reporting tasks. Here, give enough source and time information to establish the design and make overlap assessment possible, while keeping the discovery and validation roles unambiguous.

Participant Overlap and Independence

A useful overlap report gives the number of unique participants in each data set, the number found in both, the method used to identify matches, and the fraction of each data set represented by the overlap. If \(D\) is the set of unique discovery participants and \(V\) is the set of unique validation participants, then the overlap is \(D\cap V\). The overlap proportions are \(|D\cap V|/|D|\) for discovery and \(|D\cap V|/|V|\) for validation. State the denominators explicitly: the two proportions answer different questions.

For privacy-protected data, linkage may use encoded identifiers or an authorized trusted third party. A report need not reveal personal identifiers. It should, however, say what linkage keys or matching process were used at an appropriate level, whether matching was exact or probabilistic, and what uncertainty remains. “No overlap identified” is more informative when accompanied by the method and its limitations than when stated without explanation.

Participant-level checks also need to account for multiple records per person. Repeated visits, aliquots, specimens, or assay runs do not create new independent participants. If a participant contributes to discovery through one specimen and to validation through another, the validation data are not participant-independent from discovery. A cohort can include unique validation participants alongside a subset with prior contributions; the report should quantify and describe that situation rather than concealing it under a cohort label.

An overlap check can miss matches if identifiers are incomplete, inconsistent, or unavailable. Thus, a zero matched count supports a no-overlap statement only relative to the linkage method and records examined. If a data source supplies only aggregate records or lacks linkable identifiers, say that overlap could not be directly ruled out. A conservative and transparent qualification is preferable to an unjustified claim of independence.

Prespecifying the Validation Claim

A validation claim should be specific enough that it is possible to distinguish a supportive result from a result that does not meet the stated criterion. At minimum, identify the discovery finding being carried forward, the target population or data context to which the claim applies, the direction or form of the expected finding where relevant, and the evidence rule planned for evaluation. The claim should be documented before examining validation outcomes, for example in a dated protocol or analysis plan. This tutorial concerns the claim and design description; the full account of analysis choices and deviations belongs in its own reporting section.

The timing matters because choosing a target, direction, or success criterion after seeing validation results can make the evidence appear more confirmatory than it was. Prespecification does not guarantee that a study is unbiased, adequately powered, or transportable. It establishes what was intended to be tested and allows readers to compare that intention with the design and the eventual report.

The claim must also match the validation design. A statement about reproducing a direction of association in a new cohort is different from a claim that a signature will predict outcomes in a new clinical setting. A claim about a molecular feature measured in both cohorts is not automatically a claim about all features, all modalities, or all populations. State the boundary of the intended claim rather than allowing “validated” to imply more than the design can support.

Design Alternatives and Their Interpretations

Different validation designs provide different kinds of separation. None is automatically superior for every question. State the design actually used and avoid implying that one form of separation establishes another.

A random split can be appropriate for a particular evaluation, but it should not be described as external validation. A temporal or geographic cohort may offer a more distinct source, but its provenance still needs to be checked and its scope stated. Assay orthogonality is another design dimension: a different assay can strengthen a measurement-specific confirmation, but it does not settle whether participants overlap or whether the cohorts are independent.

Justifying the Overlap Counts

The basic count used in an overlap report follows from set counting. Adding the two cohort sizes counts each shared participant twice; subtracting the intersection once gives the number of unique people in their union. This identity is useful as a consistency check on reported counts.

For the hypothetical counts above, \(|D\cup V|=240+180-12=408\). If a report instead claimed 420 unique people while retaining those same source totals and overlap count, the values would be inconsistent. The identity does not prove that the linkage found every shared participant; it checks arithmetic given the identified sets.

Diagnostics, Limitations, and Common Pitfalls

Before finalizing a design description, check that the source counts refer to unique people rather than specimens or rows; that recruitment dates are labeled by event; that overlap percentages use explicit denominators; and that a statement of independence is supported by a described check. Confirm that the validation claim can be found in a record dated before validation outcomes were inspected. A mismatch among cohort names, counts, periods, and roles is a signal to resolve the provenance before interpreting the validation.

Common mistake: “The validation cohort came from a different hospital, so it is independent.” This reasoning treats site separation as proof of participant separation. Participants may attend multiple sites, move between studies, or contribute data to a shared registry. The correction is to report site provenance and participant-linkage evidence separately. If a reliable linkage was not possible, state that limitation and qualify the independence claim.

Another limitation is that independence is not a single binary property covering every source of dependence. A no-overlap check addresses shared participants to the extent the linkage can detect them. It does not rule out shared recruitment mechanisms, common laboratory processing, or other design features that may affect comparability. Nor does independent provenance alone establish that a finding generalizes to a target population. Reports should specify what was checked and avoid extending the conclusion beyond that check.

Good provenance reporting makes a validation design auditable, not automatically strong. It helps readers distinguish a genuinely new participant sample from a relabeled or partially overlapping data set, understand what the prespecified claim covered, and assess whether the evidence addresses that claim. That clarity is essential even when the eventual result is inconclusive.