Why the Population Definition Matters
A portability estimate describes performance in a population, not in an abstract category called “another ancestry.” The population depends on who is included and how its members are described. Self-identified categories, genetic principal components (PCs), and a study’s sampling frame can each define a different population or subgroup. They are related sources of information, but they are not interchangeable definitions.
This distinction matters because prediction performance can vary across the people and settings being summarized. A score’s mean squared prediction error, for example, can differ between subgroups and can also change when the proportions of those subgroups change. Consequently, two analyses of the same score may report different portability estimates without either calculation being erroneous: they may be estimating performance in different target populations.
For a performance measure represented by \(g(S,Y)\), where \(S\) is a score and \(Y\) is an outcome, write its population value as \(\theta(T)=E_T[g(S,Y)]\). The subscript \(T\) identifies the population distribution over which the expectation is taken. A portability gap between two targets is a contrast such as \(\theta(T_2)-\theta(T_1)\). The meaning of the contrast depends on the definitions of both \(T_1\) and \(T_2\), as well as on using a comparable outcome scale and performance definition.
Three Different Ways to Describe a Target
Self-identified categories record how people describe their own identity, often using categories chosen by a survey, health system, or study. Such categories can be important for describing social experience, access to care, and inequities. They do not, by themselves, specify a person’s genetic similarity to a particular reference group. Categories may be broad, change across settings, allow multiple selections, or leave some identities unrepresented. A study must state how responses were collected and how multiple or missing responses were handled.
Genetic PCs are coordinates obtained by summarizing genetic variation in a selected set of genotyped individuals and variants. They can describe patterns of genetic similarity in that data and are often used as covariates or to define analysis strata. A PC coordinate is not a universal ancestry label: its orientation, scale, and meaning depend on choices such as the individuals and variants included, data processing, and the reference used. A cutoff on a PC therefore defines a group relative to a particular analysis, not a timeless or natural boundary between populations.
A sampling frame is the mechanism or list through which people could be recruited or observed—for example, patients at particular clinics, volunteers in a biobank, or residents eligible for a specified survey. It restricts who is represented before any ancestry grouping is applied. Two studies that use the same group label may still describe different populations if their settings, eligibility rules, recruitment, participation, or observation periods differ.
| Population description | What it defines | What it does not establish on its own |
|---|---|---|
| Self-identified category | A group based on respondents’ reported identity under specified response rules | Genetic homogeneity or a unique genetic ancestry |
| PC-based group | A region or stratum in a particular genetic coordinate system | A universally comparable identity category or a fixed boundary across datasets |
| Sampling frame | People eligible or reachable under a study’s recruitment process | Representativeness of everyone outside that frame |
Worked Example: One Sample, Two Different Groupings
Hypothetical data. Suppose a validation sample contains 100 people. Investigators calculate mean squared error (MSE), measured in outcome-units squared, within four cross-classified cells. The cells contain 40 people who self-identify as A and fall in PC stratum 1, 20 who identify as A and fall in stratum 2, 10 who identify as B and fall in stratum 1, and 30 who identify as B and fall in stratum 2. Their respective MSEs are 10, 30, 20, and 40 units squared.
For self-identified category A, the weighted MSE is \( (40\times10+20\times30)/60=1{,}000/60=16.67 \) units squared. For category B, it is \( (10\times20+30\times40)/40=1{,}400/40=35.00 \) units squared.
For PC stratum 1, the MSE is \( (40\times10+10\times20)/50=600/50=12.00 \) units squared. For PC stratum 2, it is \( (20\times30+30\times40)/50=1{,}800/50=36.00 \) units squared.
These are four summaries using the same hypothetical participants, but they answer different questions: one compares self-identified categories and the other compares PC-defined strata. The overall MSE is 24.00 units squared under either partition, as it must be when the same 100 people are divided into exhaustive groups and the cell values are combined with their sample proportions. The subgroup comparisons, however, are not interchangeable.
PC Rules Are Part of the Population Definition
A reported PC-based target is incomplete if it says only “PC1 high” or “PC-defined ancestry.” To make the group interpretable, specify the genotype data and PC construction in enough detail to identify the coordinate system, the rule for assigning people to groups, and the reference to which the rule applies. If a cutoff is used, give the cutoff and the PC scale. If groups are defined by clustering or a reference assignment procedure, describe that procedure rather than presenting its labels as self-evident.
PCs are often useful for describing structure in genetic data, but their usefulness does not make them equivalent to self-identification. A PC-defined subgroup may include people with different self-identifications, and people with the same self-identification may occupy different regions of PC space. The appropriate description depends on the question: identity-based questions call for an identity measure; questions about genetic similarity or score performance across genetic structure may use genetic data, with the construction and limitations reported.
Worked Example: Interpreting a PC Cutoff
Hypothetical data. A validation team computes PC1 in a sample of 200 participants using a specified genotype set and reference panel. In that fitted coordinate system, it defines stratum L as PC1 less than 0 and stratum H as PC1 at least 0. Of the participants, 120 have PC1 less than 0 and 80 have PC1 at least 0. The team reports an MSE of 18 units squared in L and 26 units squared in H.
The rule assigns \(120/200=0.600\), or 60.0%, of the sample to L and \(80/200=0.400\), or 40.0%, to H. The sample-wide MSE is \(0.600\times18+0.400\times26=10.8+10.4=21.2\) units squared.
The supported descriptions are “participants in PC1 stratum L” and “participants in PC1 stratum H,” as defined in this analysis. The cutoff does not establish that L and H are universal ancestry groups, nor that PC1 equal to 0 has the same meaning in another dataset. If the PCs are recomputed with different individuals or variants, the resulting axes or cutoff may not define the same groups. Therefore, the two subgroup MSEs describe these specified strata; applying them elsewhere needs a justified link between the new target and these strata.
Sampling Frames and Transporting Performance
A study’s observed sample is not automatically the target population. Recruitment from specialty clinics, participation in a volunteer biobank, or restriction to people with complete genotypes may affect which individuals are observed. The sample can estimate performance for the observed participants, subject to the statistical analysis, but a broader target requires additional support. In particular, each important target subgroup must be represented or handled through a defensible method, and differences between the sample and target must not make the relevant performance information unusable.
One simple way to see the issue is to express overall performance as a weighted average of subgroup performance. Suppose a target is partitioned into mutually exclusive, exhaustive groups \(G_1,\ldots,G_K\). If \(q_k=P_T(G_k)\) is the proportion of the target in group \(k\), then
This identity follows by conditioning the expectation on the group: each group contributes its conditional mean multiplied by its share of the target. It also shows why a sample-wide estimate can change when subgroup proportions change, even if every subgroup’s conditional performance stays the same. Matching group proportions is not always sufficient for transport: performance within a group may also differ across settings or across people with different characteristics.
If subgroup-specific performance is available from a study, a target-weighted estimate can be calculated as \(\sum_k q_k\hat{\theta}_k\), where \(q_k\) is the target proportion and \(\hat{\theta}_k\) estimates performance in group \(k\). This calculation relies on the measured groups being defined consistently in sample and target, adequate representation of each target group, and the assumption that the estimated within-group performance applies to the target members of that group. When those assumptions are doubtful, the weighted value should not be presented as established target performance.
Worked Example: A Clinic Sample and a Broader Target
Hypothetical data. A clinic validation sample contains 80% of participants in group A and 20% in group B. Its estimated MSEs are 12 and 28 outcome-units squared, respectively. A health-system target population has 50% in each group. Assume, for this calculation only, that the group definitions are consistent and that the group-specific MSEs from the clinic apply to the target.
The clinic-weighted MSE is \(0.80\times12+0.20\times28=9.6+5.6=15.2\) units squared. The target-weighted MSE is \(0.50\times12+0.50\times28=6+14=20.0\) units squared.
Thus 15.2 units squared describes the clinic sample’s group mixture, while 20.0 units squared is the estimate for the specified health-system mixture under the stated assumptions. The difference, \(20.0-15.2=4.8\) units squared, arises from changing the group weights in this example; it does not show that the score’s within-group performance changed. If the clinic and target also differ within groups, the weighted value may not describe the target.
Specify What a Portability Estimate Supports
A clear report names the target populations being compared and explains how each was identified. It should distinguish the group variable from the recruitment setting and state the outcome scale and performance metric. When a sample-wide estimate is presented, report the sample’s relevant composition rather than implying that the estimate represents all people sharing a broad label. If groups are defined by PCs, describe the construction and assignment rule; if they are defined by self-identification, describe the response categories and handling of multiple selections.
A portability estimate from a particular sample can directly describe that sample’s defined population and setting. It may support a broader target after appropriate standardization or other adjustment, but that extension requires assumptions about representation, measurement, and the applicability of conditional performance. A label alone cannot establish those assumptions. In particular, a result for one self-identified category, one PC-defined stratum, or one recruitment frame should not be generalized to another merely because both are casually described using the same ancestry term.
Specify who is included, where and when they are situated, and whether the target is a sample, a subgroup, or a broader population.
Identify whether membership uses self-identification, genetic PCs, or another rule, and explain relevant response or assignment details.
Report how people could enter the study and which eligibility, recruitment, or participation restrictions define the observed sample.
State whether the estimate describes the observed sample or a target reached through weighting or adjustment, and name the assumptions behind any extension.
Common Mistake
Wrong reasoning: “The study reports performance in its ‘ancestry group,’ so the estimate applies to everyone with that ancestry, and a PC cutoff can be used as an equivalent label.”
Correction: First determine what the study’s group actually means: self-identification under specified response rules, a region of a particular PC coordinate system, or membership in a sampling frame. Then define the intended target and assess whether the sample represents it. These descriptions can overlap, but none automatically substitutes for the others. A portability conclusion should name the population supported by the analysis and avoid extending beyond it without explicit justification.
Check Your Understanding
For each question, identify the population definition or assumption that matters most.
- Why can two estimates from the same validation sample differ when one uses self-identified categories and the other uses PC strata?
- What information is needed to interpret a group defined by a PC cutoff?
- A volunteer biobank estimate is reported for a broad population. What sampling-frame issue should be considered before interpreting it as representative?
- Suppose within-group performance is stable, but target group proportions differ from sample proportions. How can the target-wide performance estimate change?
- What assumptions are needed before weighting clinic-based subgroup estimates to describe a broader health-system population?