Assessing Cohort Eligibility and Comparability

Why Eligibility and Comparability Matter

A validation result is meaningful only in relation to the people and procedures to which the validation claim applies. Two cohorts can have similarly named diagnoses yet differ in disease stage, treatment status, or how the diagnosis was established. They can also recruit from different clinical settings or handle specimens differently. Such differences may change who enters each cohort or what the recorded data represent. Before interpreting a validation analysis, check whether the cohorts meet the intended eligibility rules and whether their important features are sufficiently comparable for the claim being made.

Eligibility and comparability are related but distinct. Eligibility asks whether each person or specimen satisfies prespecified inclusion and exclusion criteria. Comparability asks how the eligible cohorts differ in relevant characteristics and procedures. A cohort may apply the eligibility criteria correctly but recruit a different mix of eligible participants. Conversely, cohorts may look similar in summary characteristics while using clinical definitions that classify people differently.

Make Eligibility Rules Operational

Write criteria so that a reviewer could apply them to either cohort without guessing. For example, “early disease” is not operational until the qualifying stages or diagnostic rule are specified. Criteria should identify the unit being selected, the qualifying diagnosis and its time point, relevant disease features, treatment status if important, and any exclusions. Define how to handle unknown or conflicting records; a criterion that cannot be checked consistently can create different eligible populations even when its wording is identical.

A useful formal representation is an eligibility indicator \(E_i\) for participant \(i\). It equals 1 when that participant satisfies every required inclusion criterion and no exclusion criterion, and 0 otherwise. If a criterion cannot be assessed, record that status explicitly rather than silently treating it as satisfied. The same logical rule should be applied in both cohorts. If a cohort’s data do not support the rule, that is a limitation of eligibility assessment, not proof that its participants qualify.

Compare Relevant Cohort Characteristics

After checking eligibility, summarize characteristics that could change the scope or interpretation of the validation claim. These may include age, disease stage, treatment history, recruitment setting, and other features relevant to the target population. For each variable, report its definition, units or categories, denominator, and a suitable summary in each cohort. A mean age is not interpretable without knowing whether it is based on all eligible participants or only those with recorded age. Different missingness can make apparently similar summaries misleading.

For a continuous characteristic, a standardized mean difference (SMD) expresses the difference in cohort means in units of a common descriptive scale. One common version is:

Here \(\bar{x}_A\) and \(\bar{x}_B\) are cohort sample means, and \(s_A\) and \(s_B\) are sample standard deviations, measured in the same units. The denominator is the square root of the average of the two sample variances. Dividing by this scale makes the difference unitless, which helps compare characteristics measured on different scales. It is a descriptive comparison, not a test of a null hypothesis. It assumes the characteristic has been defined and measured comparably in both cohorts; the formula cannot repair incompatible definitions or poor measurement.

For a binary characteristic, one descriptive SMD uses the pooled average of the two Bernoulli variances:

The proportions must refer to the same category and be calculated using clearly stated denominators. If a characteristic has several categories, display the category proportions and their denominators rather than collapsing categories in a way that hides relevant differences. As with the continuous SMD, the binary version is descriptive and depends on comparable definitions.

Inspect Definitions, Recruitment, and Sample Handling

Numerical summaries are not enough. Check whether the clinical rules behind the summaries match. A diagnosis may be based on central pathology review in one cohort and a registry code in another; disease stage may be recorded at different time points; treatment status may be defined differently. These differences can change eligibility or the composition of an ostensibly identical category. When definitions differ, state the difference and determine whether the cohorts can be restricted or recoded using a defensible common rule. If they cannot, limit the claim rather than implying that the labels are equivalent.

Recruitment context also matters. A specialist referral centre and a community clinic may draw different eligible participants even when the same written criteria are used. Compare setting, recruitment period, and route into the cohort when these features bear on the target population. The goal is not to make every feature identical; it is to identify differences that could make the validation evidence apply to a narrower or different population than claimed.

Review sample handling as a potential source of selection or interpretation differences. Relevant records may include time from collection to processing, storage conditions, and whether handling information is available for the same proportion of samples. At this stage, the question is whether handling differs enough to qualify the cohort comparison; assessing cross-platform measurement comparability is a separate issue. If processing delays are common in one cohort but rare in another, the cohorts do not have equivalent handling histories merely because both satisfy the clinical eligibility rule.

Justification, Diagnostics, and Alternatives

Standardized differences are useful because a raw difference depends on units. The same numerical mean difference can be large on one scale and small on another. Dividing by a standard deviation scale expresses the observed gap relative to variation in the characteristic. This is why an SMD can support a consistent descriptive scan across variables. But it cannot supply a universal threshold for acceptable comparability: the relevance of an age or stage difference depends on the target population and the validation claim.

A common alternative is a significance test comparing cohort characteristics. Such a test addresses evidence against a specified equality under its model assumptions; it does not establish that the cohorts are sufficiently similar. Its result also depends on sample size: a small difference may be statistically significant in very large cohorts, while a consequential difference may not be significant in small cohorts. Therefore, report descriptive summaries and judge relevance in context rather than declaring comparability because a test is nonsignificant.

Raw differences are often easier to explain in context, such as “7 years” or “15 percentage points.” SMDs add a unitless scale for comparison across variables. Neither replaces careful definition checks. Matching or restriction can sometimes create a more comparable analysis group, but they may exclude eligible participants and narrow the population to which the result applies. Any such choice must be prespecified or clearly disclosed; it does not make an incompatible clinical definition equivalent.

Look for practical warning signs: different denominators for summaries, substantial or patterned missingness, categories that do not map cleanly, eligibility rules applied at different time points, recruitment settings that differ from the claimed target, and sample-handling records available in only one cohort. Recalculate summaries using a clearly defined common eligible group when possible, and document what could not be assessed. Do not infer that an unreported difference is absent.