What Does Portability Measure?
A polygenic score can predict an outcome well in one population and less well in another. Cross-ancestry portability quantifies that difference in prediction performance. It is a comparison of measured performance, not an explanation of why performance differs. A numerical portability estimate does not by itself identify the contributions of allele frequencies, linkage disequilibrium, effect-size differences, phenotype measurement, or other factors.
A meaningful comparison requires a clearly specified prediction metric and evaluation design. The score should be fixed before evaluation, and the discovery and target performance estimates should be calculated using the same metric, outcome definition, scale, and evaluation rules. For example, comparing an out-of-sample \(R^2\) in one population with an in-sample \(R^2\) in another does not isolate a population difference: the evaluation procedures differ as well. The estimates should also be accompanied by uncertainty, since a gap between two noisy estimates is not necessarily evidence of a real performance difference.
The signed gap and the ratio answer related but different questions. The gap reports the change in the metric's own units: for example, a difference of \(-0.03\) in \(R^2\). The ratio describes target performance relative to discovery performance: a ratio of \(0.70\) means that the target metric is 70% of the discovery metric, or that 30% is lost on this ratio scale. Neither measure should be interpreted without naming the metric and the populations being compared.
These definitions assume that larger metric values represent better performance. For a metric where smaller values are better, such as prediction error, the sign and ratio need a different convention. One option is to report the target-minus-discovery error difference directly, where a positive value indicates worse target performance. Another is to transform the metric to a clearly defined benefit scale before calculating a retention ratio. State the convention rather than silently reversing signs.
Choose a Comparable Metric
For a quantitative outcome, a commonly used metric is out-of-sample \(R^2\), provided it is defined identically in both populations. A difference in \(R^2\) is in \(R^2\) units; a ratio compares the two \(R^2\) values. The interpretation remains specific to the evaluation setup. For example, differences in outcome variance or in the prediction baseline can affect \(R^2\), even when the same score is used.
For a binary outcome, the area under the receiver operating characteristic curve (AUC) measures discrimination: the probability that a randomly selected case receives a higher score than a randomly selected non-case, with ties handled by the metric's standard convention. AUC has a chance baseline of 0.5. Consequently, a raw AUC ratio is usually difficult to interpret as “performance retained”: an AUC of 0.60 is not simply twice as useful as an AUC of 0.30, and the chance baseline is not zero.
If a ratio is desired for a metric with a nonzero chance baseline \(c\), an excess-over-baseline ratio can be defined as \((M_T-c)/(M_D-c)\), when \(M_D>c\). For AUC, \(c=0.5\). This ratio describes retention of performance above the specified baseline, but it is unstable when \(M_D\) is close to \(c\), and it does not replace reporting the original metric and its absolute difference. AUC, \(R^2\), calibration measures, and error rates address different properties; they should not be combined into one ratio as though they were interchangeable.
Uncertainty in Gaps and Ratios
Write \(\widehat M_D\) and \(\widehat M_T\) for estimated performances, with standard errors \(SE_D\) and \(SE_T\). The estimated gap is \(\widehat\Delta=\widehat M_T-\widehat M_D\). Its variance depends on whether the estimates are independent. In general,
If the two estimates are independent, their covariance is zero, so \(SE(\widehat\Delta)=\sqrt{SE_T^2+SE_D^2}\). If they share individuals, folds, or other sources of sampling variation, the covariance may not be zero. In that case, simply adding the two squared standard errors can give the wrong uncertainty. A bootstrap that repeats the full evaluation procedure while preserving the dependence structure is often a practical alternative.
For a positive ratio \(\widehat R=\widehat M_T/\widehat M_D\), uncertainty is often calculated on the log scale. The following approximation uses the delta method: for estimates sufficiently concentrated away from zero, the variance of a smooth function can be approximated by applying its derivative to the estimates' variance and covariance.
An approximate 95% interval for the ratio is obtained by calculating \(\log(\widehat R)\pm1.96\,\widehat{\operatorname{SE}}\{\log(\widehat R)\}\), then exponentiating both endpoints. This gives an interval with positive endpoints and accommodates the skew that ratios can have. The approximation can be poor if the denominator is near zero, if estimates are highly skewed, or if metric constraints matter substantially. In those cases, a bootstrap interval may better represent the sampling distribution.
Worked Example: Retention of Out-of-Sample \(R^2\)
Suppose a fixed score is evaluated on independent, hypothetical held-out samples. Its estimated out-of-sample \(R^2\) is \(0.080\) in the discovery population, with \(SE_D=0.006\), and \(0.048\) in the target population, with \(SE_T=0.005\). Assume the same \(R^2\) definition and evaluation protocol were used in both populations, and that the standard errors adequately describe sampling uncertainty.
The estimated absolute gap is
Because the estimates are assumed independent, the standard error of the gap is
An approximate 95% interval is \(-0.032\pm1.96(0.007810)\), or approximately \((-0.0473,-0.0167)\). Under the stated sampling assumptions, the target-population \(R^2\) is estimated to be 0.032 lower, with an approximate 95% interval for the difference from 0.0473 lower to 0.0167 lower.
The retention ratio is \(0.048/0.080=0.600\): the target \(R^2\) is 60.0% of the discovery \(R^2\), corresponding to a 40.0% relative loss on this ratio scale. For its interval, the log-scale standard error is
Since \(\log(0.600)\approx-0.5108\), the log-scale interval is approximately \((-0.7624,-0.2592)\). Exponentiating gives a ratio interval of approximately \((0.4665,0.7717)\). Thus, under the approximation, the target metric is estimated to be about 46.7% to 77.2% of the discovery metric. These are uncertainty intervals for population performance contrasts, not probabilities that the fixed, unknown contrast lies inside a particular interval.
Worked Comparisons for Binary Outcomes
Worked Example: Comparing AUC Without a Misleading Raw Ratio
In a hypothetical evaluation, the discovery-population AUC is estimated as \(0.74\), with \(SE_D=0.018\), and the target-population AUC as \(0.68\), with \(SE_T=0.022\). Assume the samples are independent and both AUCs were calculated using the same outcome definition and evaluation protocol.
The target-minus-discovery difference is \(0.68-0.74=-0.060\). Its standard error is
The approximate 95% interval is \(-0.060\pm1.96(0.02843)\), or approximately \((-0.1157,-0.0043)\). The target AUC is estimated to be 0.060 lower; the interval describes uncertainty in that absolute AUC difference. A raw ratio, \(0.68/0.74\approx0.919\), can also be computed, but describing it as “91.9% of discrimination retained” ignores the AUC chance baseline of 0.5.
On an excess-over-chance scale, the point estimate is \((0.68-0.50)/(0.74-0.50)=0.18/0.24=0.750\). This says the target estimate retains 75.0% of the discovery estimate's AUC above chance. It is a different estimand from the raw AUC ratio and requires its own uncertainty interval. Here, the denominator is positive, but if the discovery AUC were close to 0.50, small changes in either AUC could make this ratio very unstable.
Worked Example: Bootstrap Uncertainty for a Portability Ratio
Suppose a hypothetical evaluation estimates discovery and target \(R^2\) values of \(0.071\) and \(0.052\), respectively. The point estimate of retention is
A researcher obtains an interval by resampling individuals separately within each ancestry group, recalculating both \(R^2\) values and their ratio in each of 2,000 bootstrap replicates. This procedure is appropriate only if individuals are the independent sampling units; relatedness, clusters, or a different sampling design would require resampling at the corresponding independent unit. Assume, for this hypothetical example, that the 2.5th and 97.5th percentiles of the resulting ratios are \(0.58\) and \(0.91\).
The reported estimate is therefore \(0.732\), with a 95% bootstrap percentile interval of \((0.58,0.91)\). In this example, the target \(R^2\) is about 73.2% of the discovery \(R^2\), and the bootstrap interval represents sampling uncertainty under the resampling scheme. The interval's endpoints are stated as the hypothetical bootstrap results; they cannot be derived from the two point estimates alone. The resampling must repeat any data-dependent steps that are part of the intended uncertainty assessment, rather than treating a selected score as if it had been selected independently of all data.
Conditions, Diagnostics, and Interpretation
Before reporting a portability estimate, check the conditions that make the comparison interpretable. In particular, verify that the score and metric are fixed consistently, the outcome is defined and scaled comparably, and both populations use comparable evaluation procedures. If covariate adjustment is part of the metric, use the same covariate specification and state the performance quantity being compared. A target sample with few cases or limited outcome variation may yield an unstable estimate even when the point estimate looks precise.
Inspect the uncertainty as well as the point estimate. A ratio interval that is very wide or has a lower endpoint near zero may indicate that the data do not pin down the degree of retention. A denominator near zero makes a ratio especially unstable. For bounded measures such as AUC, an interval calculation based on a normal approximation may produce implausible endpoints; use a method that respects the metric's sampling distribution when needed. Compare the ratio interval with the difference interval, since a ratio alone can conceal the scale of the absolute performance gap.
If estimates are dependent, check whether the uncertainty calculation reflects that dependence. For example, comparing overlapping evaluation samples or estimates that share fitted components can induce covariance. A joint bootstrap should resample the appropriate units and recalculate both metrics together. Conversely, when independent cohorts are evaluated using a locked score, the independent-estimate formula can be reasonable if its other assumptions are adequate.
A portability contrast describes population-level predictive performance in the evaluated samples. It does not establish that ancestry causes a performance difference, nor does it describe the performance for every individual in either population. Ancestry labels summarize heterogeneous populations and should not be treated as precise measures of an individual's genetic or social circumstances. The estimate also does not diagnose the mechanism behind a gap; it quantifies the gap so that subsequent scientific investigation can address possible sources.
Common Pitfalls and Alternatives
- Reporting only a ratio. A ratio can hide a small or large absolute difference. Report the metric values and absolute gap alongside it.
- Using a raw ratio for a metric with a nonzero baseline. AUC has chance level 0.5, so a raw AUC ratio does not represent retention above chance. If using an excess-over-baseline ratio, name the baseline and report the original AUC difference too.
- Comparing unlike estimates. Differences in outcome definition, scale, covariate adjustment, or evaluation design can masquerade as portability loss. Harmonize these choices or explicitly limit the interpretation.
- Ignoring uncertainty or dependence. A ratio of point estimates is not a precision measure. Use a justified standard-error calculation or resampling method that preserves the data structure.
- Treating a portability estimate as an explanation. A measured gap does not identify its cause. It is a performance comparison, not a decomposition of mechanisms.
The main alternatives are complementary rather than interchangeable. An absolute difference is clearest when the metric has a meaningful scale and the practical size of the gap matters. A retention ratio is useful when relative performance is the question and the denominator is well away from zero. An excess-over-baseline ratio can be useful for metrics such as AUC, but can be unstable near the baseline. Reporting more than one measure, with uncertainty for each, often makes the comparison easiest to interpret.
Check Your Understanding
Answer each question before reading the short solutions.
- A score has \(R^2=0.10\) in a discovery population and \(R^2=0.07\) in a target population. What are the target-minus-discovery gap and retention ratio?
- Why can a raw ratio of target AUC to discovery AUC be hard to interpret as retained discrimination?
- Two performance estimates are independent, with standard errors 0.02 and 0.03. What is the standard error of their difference?
- What is one reason to use a bootstrap rather than a log-scale delta-method interval for a ratio?
- Does a lower target-population metric establish that ancestry caused the performance loss? Explain.
Solutions. (1) The gap is \(0.07-0.10=-0.03\), and the ratio is \(0.07/0.10=0.70\), so the target \(R^2\) is 70% of the discovery \(R^2\). (2) AUC has a chance baseline of 0.5, not zero; an excess-over-chance ratio may be more relevant if its denominator is safely above zero. (3) The standard error is \(\sqrt{0.02^2+0.03^2}=\sqrt{0.0013}\approx0.0361\). (4) A bootstrap can better accommodate skewed ratio distributions or metric constraints, provided it resamples the appropriate units and repeats the relevant evaluation procedure. (5) No. The comparison quantifies predictive performance in the evaluated populations; it does not establish a causal explanation.