What Makes a Replication Cohort Independent?
A replication analysis is informative only if its evidence is not simply a second calculation on substantially the same observations. Independence is therefore a design property to establish and document, not a label implied by a different institution, recruitment date, or dataset name. The aim is a replication cohort whose participants and measurements provide new information about the association under study.
The earlier material on replication estimands distinguishes the target effect, its direction, and the population to which it refers. Cohort design must support that target: participants need to come from a relevant population, the phenotype and genetic exposure must be measured comparably, and the analysis must account for dependence among observations. A new cohort need not have identical demographics to the discovery cohort, but differences should be understood rather than hidden.
For a quantitative phenotype, a typical association model may be written as \(Y_i=\beta_0+\beta_G G_i+C_i^\mathsf{T}\gamma+\varepsilon_i\), where \(Y_i\) is the phenotype, \(G_i\) is genetic dosage, and \(C_i\) contains adjustment variables. For a binary phenotype, a corresponding model is \(\operatorname{logit}\{P(Y_i=1\mid G_i,C_i)\}=\beta_0+\beta_GG_i+C_i^\mathsf{T}\gamma\). The design should make the model’s assumptions plausible: the phenotype and dosage are defined consistently, the model is suitable for the outcome, relevant population structure and covariates are handled, and the assumed independence of observations is either reasonable or replaced by a dependence-aware analysis.
In particular, “independent cohort” does not guarantee independent observations within that cohort. Siblings or other relatives can share genetic background and environmental influences. Likewise, a replication dataset can overlap with discovery even if its records carry a different study label. Both issues need an explicit plan before results are interpreted.
Audit Participant Overlap Before Analysis
Overlap can arise through participants enrolled in multiple studies, shared biobank recruitment, transferred samples, or merged datasets. Names and study labels are not reliable checks: participants may appear under different identifiers, while the same identifier may not be globally unique. A defensible audit uses available consented linkage information, such as harmonized participant identifiers, documented sample provenance, and genotype-based checks where permitted. The audit should record its method, the number of possible matches, and what was done with them.
The cleanest design is to exclude discovery participants from replication before estimating the replication association. If overlap cannot be resolved, the dependence must be accounted for by an analysis designed for it; simply treating estimates as independent is not justified. A small overlap is still an overlap. Its impact depends on the number and influence of shared participants, so a fixed percentage is not a universal threshold for safety.
Worked Example: Turning an Overlap Audit Into an Analysis Set
Suppose a discovery analysis contains 1,200 people. A proposed replication cohort contains 900 people, and a permitted linkage audit identifies 45 of them as also present in discovery. The replication protocol requires no shared participants.
The proportion of the proposed replication cohort identified as overlapping is \(45/900=0.05\), or 5%. Excluding the 45 overlapping participants leaves \(900-45=855\) participants for replication. The replication analysis must use those 855 people, not the original 900.
This arithmetic does not establish that 855 is sufficient for the study’s scientific objective; it only gives the eligible count after resolving known overlap. The remaining cohort should also pass the planned phenotype, genotype, and covariate checks. If the audit were incomplete, the count of 855 would not establish independence: unrecognized shared participants could still make the replication estimate statistically dependent on discovery.
Overlap can be missed when study records are assembled from several contributing cohorts. A practical audit therefore traces participants back to recruitment sources and sample identifiers, not only to the final analysis file. Keep a reproducible record of exclusions so that the reported sample size can be reconciled with the original enrollment count.
Related Participants Require a Separate Plan
After removing discovery overlap, assess relatedness within the replication cohort. If relatives are analyzed as though every observation were independent, standard errors can be wrong because outcomes and genotypes may be correlated within families. The response should be chosen before analysis: either select a subset with no close relatives for a design that requires unrelated participants, or retain relatives and use a method that models the dependence. Excluding relatives can reduce sample size and may alter the cohort composition; retaining them does not help if the analysis ignores their relationship.
One simple illustration of the cost of clustering comes from estimating a mean. Suppose there are \(K\) equally sized families with \(m\) participants per family, each outcome has variance \(\sigma^2\), and the correlation between any two outcomes within a family is \(\rho\), while different families are independent. Then the variance of the overall mean is
The factor \(1+(m-1)\rho\) is the design effect for this particular mean-estimation setup. It shows why counting every relative as a fully independent observation can overstate information. It is not a general replacement for a genetic association power calculation: the appropriate precision depends on the genetic model, relatedness pattern, and analysis method. In association analysis, a family-aware model may use a covariance structure based on relatedness, rather than relying on a single effective-sample-size adjustment.
Worked Example: Understanding the Cost of Family Clustering
Consider 300 sibling pairs, so \(K=300\) families and \(m=2\) people per family, for 600 people total. For a simplified mean-estimation illustration, suppose the within-pair outcome correlation is \(\rho=0.20\). The design effect is
The corresponding effective sample size for estimating this mean is \(600/1.20=500\). Thus 600 clustered observations have the same mean-estimation variance as 500 independent observations under the stated assumptions. This does not mean the genetic association analysis should automatically be run as though there were exactly 500 people. It means the naive independent-observation calculation is not appropriate; the study should either restrict the sample to unrelated participants or use a dependence-aware association model and plan its precision accordingly.
The assumptions behind the illustration matter: family size is equal, within-family correlation is exchangeable, and families are independent. Real cohorts may contain pedigrees of different sizes and degrees of relatedness. Genotype similarity and phenotype similarity also need not have the same pattern. Diagnose relatedness using appropriate genetic and pedigree information, then verify that the chosen software and model handle the observed structure.
Make Phenotypes and Measurements Comparable
A replication cohort should measure the same scientific construct as discovery. This requires more than matching variable names. Check case definitions, control eligibility, diagnostic codes, assessment windows, instruments, units, transformations, and missing-data rules. For a quantitative phenotype, confirm that the scale and transformation support a meaningful comparison. For a binary phenotype, confirm that cases and controls meet comparable criteria. A mismatch can change who is classified or what a one-unit effect means, even when both files use the same column label.
If procedures differ, harmonize them using a prespecified and documented rule where scientifically defensible. A conversion between units may be straightforward; a change in diagnostic definition may not be. Do not assume that statistical adjustment can repair a phenotype that measures a different construct. When cohort-specific procedures cannot be made sufficiently comparable, describe that limitation and reconsider whether the cohort is suitable for the intended replication.
Worked Example: Checking Case-Definition Agreement
Suppose a proposed replication cohort has 400 participants classified under its local definition. A review against the common study definition finds that 360 classifications agree and 40 do not. The crude agreement is \(360/400=0.90\), or 90%.
That result is a useful warning and audit summary, not proof that the definitions are interchangeable. The 40 disagreements could be concentrated among cases, among controls, or in a clinically important subgroup. The team should inspect the discrepancy table, apply the common definition if the necessary source data are available, and recalculate the eligible case and control counts. If they are not available, the study should state that phenotype classification differs and consider whether the resulting association still addresses the intended estimand.
For example, if applying the common definition reclassifies 25 of the 40 discrepant participants as ineligible, the analysis set falls from 400 to \(400-25=375\), subject to any other exclusions. It would be misleading to report 400 as the analyzed sample size or to treat the original 90% agreement as if it eliminated measurement concerns.
Comparability also concerns genetic measurements and covariates. Confirm that allele dosage refers to the same allele, that genetic quality checks are applied consistently, and that adjustment variables have compatible definitions. Differences in ancestry composition can affect allele frequencies and patterns of genetic correlation; this does not automatically invalidate replication, but it requires suitable control of population structure and careful interpretation of the target population.
Plan Usable Sample Size, Not Just Enrollment
The number recruited is not necessarily the number contributing to the association estimate. Eligibility restrictions, unresolved overlap, relatedness handling, missing phenotypes, genotype quality control, and missing adjustment variables can all reduce the analysis set. Build a participant flow before analysis: start with enrollment, list each exclusion criterion, and estimate the number remaining after each stage. Keep exclusions disaggregated where possible; a single overall attrition percentage can conceal a specific bottleneck.
Adequacy depends on the scientific target and analysis, not on a universal minimum sample size. A cohort with many eligible participants may still provide limited information for a low-frequency allele, a rare outcome, or a poorly measured phenotype. Conversely, sample-size planning needs the expected effect and variability, allele frequency or outcome prevalence, model, and error criterion. The design question here is to ensure those calculations use the anticipated usable cohort, with realistic exclusions and a suitable analysis for relatedness. A recruitment target should be increased when planned losses would otherwise leave too few analyzable participants.
Design Workflow, Alternatives, and Diagnostics
Ensure the proposed cohort can inform the effect and population of interest, and document material differences from discovery.
Trace recruitment sources, audit possible discovery overlap, and resolve matches before analysis.
Either exclude relatives under a stated rule or use an association method that models their dependence; do not silently assume independence.
Compare phenotype definitions, units, instruments, covariates, and genetic coding, and document any irreducible differences.
Apply anticipated exclusions in sequence and use the resulting analyzable count for study planning and reporting.
The main alternatives are a strict unrelated, nonoverlapping sample and a broader sample analyzed with dependence-aware methods. The strict approach is easier to describe and may fit methods that assume unrelated observations, but it discards information and can change the sample’s composition. A family-aware method can retain relatives, but its assumptions, covariance model, and implementation must fit the data. Neither approach excuses unresolved overlap with discovery: within-cohort relatedness and cross-study participant overlap are distinct problems.
Useful diagnostics include a participant-flow table, counts of related pairs or pedigrees, overlap-audit summaries, phenotype discrepancy tables, allele and unit checks, and checks that model inputs have the expected distributions and missingness. Warning signs include unexplained sample-count changes, a study labeled independent despite shared recruitment, relatives analyzed under an unrelated-sample model, large phenotype-definition disagreements, or different units treated as identical. These are design failures, not merely reporting details.
Common Mistake and Key Takeaway
- Wrong reasoning: “The replication data came from another hospital, so the estimate is independent.”
- Correction: Participants can appear in multiple hospitals’ datasets or be linked through shared recruitment. Audit participant provenance and resolve overlap; institutional separation alone does not establish statistical independence.
- Wrong reasoning: “We have 600 people, so our analysis has 600 independent observations.”
- Correction: Related participants can induce dependence. Use an unrelated subset or an appropriate dependence-aware method, and plan with the resulting information rather than the raw headcount.
- Wrong reasoning: “The phenotype column has the same name, so it is comparable.”
- Correction: Compare definitions, measurement procedures, units, and eligibility rules. Matching labels do not guarantee matching constructs.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A replication dataset includes 720 people, of whom 18 are found in the discovery dataset. If the protocol excludes all overlap, how many remain, and does the different dataset name establish independence?
- In a cohort of 200 sibling pairs, what assumptions are needed to use the equal-cluster design-effect formula for a mean? Why should its effective sample size not be treated as a universal genetic-association sample size?
- A phenotype is recorded in kilograms in discovery and pounds in replication. What should be checked before comparing effects?
- Give two reasons a planned recruitment count may exceed the final analysis count.
Solutions. (1) \(720-18=702\) participants remain after excluding the identified overlap. A different dataset name is not evidence of independence; participant provenance and possible additional overlap still require an audit. (2) The simple formula assumes equal cluster size, an exchangeable within-family correlation, and independence between families. Genetic association precision also depends on the genetic model, genotype correlations, and analysis method, so the mean-estimation adjustment is not universal. (3) Confirm the unit conversion is correct and the same phenotype definition and measurement procedure were used; effects must be on a compatible scale before comparison. (4) Examples include exclusions for overlap, relatives under an unrelated-sample design, missing phenotype or covariate data, and genotype quality-control failures.