Validation Designs for Portability Comparisons

Why Validation Design Matters

A score can appear to perform differently across ancestry groups because of differences in the score’s transportability, but also because the groups were measured or sampled differently. For example, one group might have an outcome confirmed through clinical records while another is classified from self-report; one group might contain many relatives of people used to estimate the score’s weights; or one validation set might include people whose data helped select the score. These design differences can distort a comparison before any performance statistic is calculated.

A portability comparison therefore needs more than a score and two sets of results. It needs a prespecified validation design: a clear target population, a fixed score, comparable outcome ascertainment, and explicit safeguards against overlap and leakage. The goal is to make the contrast between ancestry strata reflect performance in those target populations rather than avoidable differences in how the validation data were assembled.

Specify the Comparison Before Looking at Results

Let \(S_i\) denote the score for participant \(i\), \(Y_i\) the outcome under study, and \(A_i\) the participant’s prespecified ancestry stratum. Let \(D\) denote the data used to develop or select the score and \(V\) the validation data. A clean validation design fixes the score using \(D\), then evaluates it in \(V\) without using validation outcomes to revise the score or choose which result to report.

Before accessing validation outcomes, specify the target populations, ancestry-stratum definitions, outcome definition, assessment window, score version, performance measures, inclusion rules, and primary comparison. The stratum definitions should be appropriate to the target-population question, rather than created after examining which grouping produces the most favorable results. Ancestry labels and genetic principal components can assist with defining and describing populations, but no single label should be treated as a complete description of genetic or social variation.

The same score means the same weights and allele coding are applied in every stratum. If the score is recalibrated or otherwise modified separately by ancestry, that is a different analysis from validating a fixed transported score. A study can report both questions, but should identify them as separate analyses and specify each procedure in advance. Similarly, score standardization should be planned and described: standardizing within each stratum changes the unit of a score, even though a positive linear rescaling does not change its rank ordering.

Independence, Overlap, and Relatedness

An independent validation set should not contribute participants’ outcomes to the score’s weight estimation, hyperparameter selection, feature selection, or choice among competing score versions. Overlap can arise when discovery and validation samples draw on the same biobank, when cohorts are combined in a meta-analysis, or when a nominally separate dataset contains previously analyzed participants. A sample may be independent at the cohort-name level but not at the participant level.

The practical safeguard is to compare participant identifiers across all contributing sources and document the overlap checks. If participants are shared, remove overlapping individuals from validation or use a genuinely independent dataset. If removal is not possible, clearly characterize the overlap and avoid presenting the result as fully independent. Outcome-informed choices made using validation data also compromise independence, even when no participant appears in the discovery sample.

Relatedness requires a separate decision. A validation set can contain relatives without having been used to develop the score, but close relatives can make a random individual-level split misleading: a person in the evaluation set may have a close relative in the data used for score selection or tuning. For an internal holdout, assign whole family or relatedness clusters to one side of the split. For an external validation sample, either define a sufficiently unrelated analytic set for the primary comparison or retain relatives and use methods that account for clustering when quantifying uncertainty. State which approach was used; do not imply that relatives were removed if they were merely accounted for in an analysis.

Make Phenotypes and Sampling Comparable

A common outcome label is not enough to ensure a common phenotype. Comparability depends on the operational definition: diagnostic criteria, data source, assessment procedure, time horizon, and rules for deciding who is a case or control. Apply the same definition across ancestry strata where possible. If measurements come from different sources, describe the difference and assess whether it could change who is classified as a case or control.

Controls also need a clear definition. For a disease outcome, controls should meet the prespecified criteria for being classified as noncases over the relevant assessment window. If age, follow-up time, or screening intensity affects the chance that disease has been detected, those factors may make nominal controls differ across strata. Excluding prevalent cases, requiring adequate follow-up, or defining disease-free status at a common time point may be necessary for the intended target question.

Ascertainment can differ even when the phenotype definition is identical. A clinic-based sample, a population cohort, and a case-control study can represent different participant distributions. In a case-control study, the proportion of cases is often set by design rather than reflecting population prevalence. Consequently, quantities that depend on prevalence, such as positive predictive value, cannot be interpreted as population values from the sampled case fraction alone. For a fair comparison, use the same sampling logic across strata or explicitly account for differing sampling schemes when the target quantity requires it.

One useful distinction is between ranking and prevalence-dependent quantities. Under specific conditions, case-control sampling can preserve a ranking measure such as the area under the receiver operating characteristic curve (AUC), while changing the case proportion and therefore predictive values. The conditions matter: cases and controls must represent the corresponding target case and noncase score distributions, and selection into the sample must not further depend on score within outcome group.

Why Case-Control Sampling Can Preserve AUC but Change Predictive Value

For a score \(S\), the AUC can be defined as the probability that a randomly selected case has a higher score than a randomly selected noncase, with a standard half-credit convention for ties. If sampling depends only on outcome status, the sampled score distribution among cases remains the case score distribution, and the sampled score distribution among controls remains the noncase score distribution. The case-control sampling fractions therefore do not change the pairwise comparison defining AUC.

In contrast, positive predictive value depends on the proportion of cases in the population being evaluated. If a threshold has sensitivity \(Se\), specificity \(Sp\), and disease prevalence \(\pi\), then Bayes’ rule gives:

Changing the case fraction changes \(\pi\) in this expression. Thus a case-control sample may support a comparison of ranking under the stated assumptions, but its observed positive predictive value should not be reported as a population predictive value without appropriate sampling information and adjustment.

Worked Design for a Direct Portability Comparison

A useful comparison applies the same outcome and analysis rules in every stratum. If one group’s outcome is assessed over a different time horizon, or if one group’s controls are selected differently, the resulting difference cannot be attributed solely to score portability. The design should state whether the target is a common clinical population, a particular cohort, or another well-defined population, because conclusions apply to the populations represented by the validation data.

Common Mistakes and Practical Reporting

• Calling a cohort independent because it has a different name. Participant-level overlap can remain across datasets. Check identifiers and document exclusions or unresolved overlap.
• Splitting relatives at random between development and evaluation. A close relative in the development data can make an internal holdout less independent. Allocate related individuals together, or state and address relatedness in the analysis.
• Using similar labels for different outcomes. “Disease case” may mean a confirmed diagnosis in one stratum and a self-report in another. Compare operational definitions, not just variable names.
• Comparing predictive values from differently ascertained samples. Case-control sampling changes the sampled prevalence. Do not interpret the sample case fraction as the population prevalence unless the design justifies it.
• Changing the score after inspecting validation results. A score revised using validation outcomes is no longer being evaluated as a fixed, independent score in that same analysis. Label any such work as development or tuning, and seek separate validation.
• Interpreting a performance gap as a causal effect of ancestry. A stratified contrast is descriptive of the defined populations and design. It does not isolate ancestry from all differences in environments, sampling, measurement, or other characteristics.

A concise report should identify the target population and ancestry-stratum definition; the score version and whether it was fixed in advance; the validation source and participant-overlap checks; the relatedness strategy; the phenotype and follow-up rules; and the sampling design in each stratum. It should also state which performance measures were compared and whether any scores were transformed or recalibrated. These details let readers judge whether the strata support a fair comparison rather than merely showing two numbers.