Why Compare Several Evaluations?
A polygenic score (PGS) can appear to perform differently when the people evaluated, the way they entered a biobank, or the disease prevalence differs from the intended setting. A single performance number rarely identifies which feature is responsible. Comparing carefully defined results is a diagnostic exercise: it reveals which conclusions are sensitive to ancestry composition, participation, or prevalence, while the assumptions behind each comparison determine what can be inferred.
The examples below use a binary outcome and a binary score category, “high” versus “low,” to make the arithmetic transparent. The score category is only a teaching device; real evaluations often use the continuous score and report measures such as an association coefficient, area under the receiver operating characteristic curve (AUC), or calibration. A risk difference here means the outcome risk in the high-score group minus the risk in the low-score group. It is an association, not evidence that the score causes the outcome.
Start With the Unadjusted and Ancestry-Adjusted Results
An unadjusted result mixes the score–outcome relationship within ancestry groups with differences in the ancestry composition of the score groups. If ancestry is associated with both score distribution and outcome risk, that mixture can weaken or even reverse the within-group association. The first diagnostic is therefore to report the marginal contrast alongside contrasts within relevant ancestry groups. Ancestry adjustment is informative only where there is adequate score variation within groups; an adjustment cannot manufacture information absent from the data.
Worked Example: An Unadjusted Association Reverses Within Ancestry Groups
Suppose a hypothetical biobank evaluation includes the following participants. “Cases” have the binary outcome, and each row gives counts by ancestry group and score category.
| Ancestry | Score category | Cases | Participants | Risk |
|---|---|---|---|---|
| A | High | 100 | 200 | 100/200 = 0.50 |
| A | Low | 320 | 800 | 320/800 = 0.40 |
| B | High | 120 | 800 | 120/800 = 0.15 |
| B | Low | 10 | 200 | 10/200 = 0.05 |
The unadjusted high-score risk is \(220/1000=0.22\), and the unadjusted low-score risk is \(330/1000=0.33\). Thus the marginal risk difference is
In this biobank sample, the high-score group has an outcome risk 11 percentage points lower than the low-score group. But the within-ancestry risk differences go in the opposite direction:
In both ancestry groups, the high-score risk is 10 percentage points higher. The reversal occurs because 80% of high-score participants are in group B, where risks are lower, whereas 80% of low-score participants are in group A, where risks are higher. The marginal contrast therefore combines different ancestry-specific risks with very different ancestry compositions. It does not describe the within-group association.
This example uses risk differences to expose the composition problem; a real analysis may use a regression model with ancestry covariates and, when justified, ancestry-by-score interactions. If the score association differs across ancestry groups, one common adjusted coefficient can conceal that heterogeneity. The stratum-specific estimates and their uncertainty should then be examined rather than assuming a single shared effect.
Use Participation Weights as a Separate Diagnostic
Ancestry adjustment and participation weighting address different concerns. Ancestry adjustment accounts for measured ancestry-related differences in the score and outcome. Inverse-probability weighting instead aims to recover an evaluation for a specified source population when participation is selective. For participant \(i\), a basic weight is the inverse of the probability of participation conditional on measured variables \(Z_i\):
where \(S_i=1\) indicates participation. In practice, weights may be stabilized and their distribution checked. A weighted comparison is credible only if the participation model is adequate, the variables needed to explain selection are measured, and every relevant covariate pattern in the source population has a positive probability of participation. If participation depends on unmeasured outcome-related factors after conditioning on \(Z_i\), weighting may not remove the bias.
Worked Example: Recomputing Within-Ancestry Risks With Participation Weights
Continue with the hypothetical counts above. Suppose the estimated participation probabilities are 0.50 for group A, high-score cases and 0.80 for group A, high-score noncases; 0.80 for both outcome categories in group A, low score and group B, high score; and 0.50 for group B, low-score cases and 0.80 for group B, low-score noncases. These probabilities are illustrative and assumed known for this calculation. Each participant receives weight \(1/P(S=1\mid Z)\).
For group A, high score, the weighted case total is \(100(1/0.50)=200\), and the weighted noncase total is \(100(1/0.80)=125\). The weighted risk is therefore
For group A, low score, cases and noncases both have participation probability 0.80. The weighted totals are \(320/0.80=400\) cases and \(480/0.80=600\) noncases, giving a weighted risk of \(400/(400+600)=0.4000\). The weighted risk difference in group A is \(0.6154-0.4000=0.2154\).
For group B, high score, both outcome categories have probability 0.80. The weighted totals are \(120/0.80=150\) cases and \(680/0.80=850\) noncases, so the weighted risk is \(150/1000=0.1500\). For group B, low score, the weighted totals are \(10/0.50=20\) cases and \(190/0.80=237.5\) noncases. Its weighted risk is \(20/(20+237.5)=0.07767\), giving a weighted risk difference of \(0.1500-0.07767=0.07233\).
Weighting changes the within-group contrasts from 0.1000 to 0.2154 in group A and from 0.1000 to 0.07233 in group B. Under the assumed selection probabilities, this is evidence that participation selection affects these evaluations. It is not proof that the weights have removed all bias: that interpretation depends on the selection model and measured covariates being adequate. Nor should these stratum-specific results be confused with a single result for a chosen target population.
Separate Prevalence Calibration From Association
A case-control evaluation can have an outcome prevalence chosen by the sampling design rather than reflecting the target setting. When sampling depends on outcome status but, conditional on outcome, is independent of the score, the sample-to-target change in prevalence can be represented by a logistic intercept shift. The assumption matters: if selection also depends on the score within outcome categories, this simple correction need not be valid.
Let \(p_s\) be the sample prevalence and \(p_t\) the target prevalence. Since the odds are \(p/(1-p)\), the intercept offset is the log target odds minus the log sample odds:
For a model with sample-scale predicted probability \(p_s(x)\) at score and covariates \(x\), the corrected probability is obtained by adding \(\Delta\) to the logit and transforming back:
Worked Example: Correcting Predicted Risk for a Prevalence Shift
Consider a separate hypothetical case-control evaluation in which the sample prevalence is 0.275 and the target prevalence is 0.10. Suppose, for this simple intercept-only illustration, every participant receives the sample-scale predicted risk 0.275. The sample odds are \(0.275/0.725=11/29\), while the target odds are \(0.10/0.90=1/9\). Keeping the ratio exact until the final rounding gives
The sample predicted odds are \(11/29\). After applying the offset, the odds become
Thus the corrected predicted risk is 10% in this intercept-only example, matching the specified target prevalence. With varying predictions, a prevalence correction shifts the logit of each prediction by the same amount under the stated sampling assumptions. It changes absolute predicted risks, but a common intercept shift preserves the ordering of individuals and therefore does not change the AUC. It does not fix a distorted score ranking, ancestry-related contamination, or selection that depends on the score beyond the assumed case-control mechanism.
A Practical Diagnostic Sequence
The comparisons are most useful when treated as a sequence of questions, not as a contest to find the most favorable performance number. Each estimate should be accompanied by its population, outcome definition, score scale, uncertainty, and assumptions.
Report ancestry composition, outcome prevalence, participation criteria, and how the score and outcome were measured. Check whether score distributions differ across ancestry groups.
Compare the marginal association with within-ancestry estimates. Material changes indicate sensitivity to ancestry composition; they do not alone identify every source of structure or prove which estimate transports.
Specify the participation model and its covariates, inspect weight extremes and overlap, and compare weighted with unweighted results. Interpret the weighted result only under the model’s selection and positivity assumptions.
Ask whether the sample-to-target prevalence difference follows the assumed outcome-dependent sampling mechanism. If so, report the calibration adjustment separately from discrimination and ancestry-adjusted association results.
Common Mistakes and Interpretation
- Treating an adjusted result as automatically unbiased. Ancestry covariates can reduce measured population-structure confounding, but do not correct unmeasured structure, inadequate overlap, or participation selection by themselves.
- Assuming weighting is a universal repair. Inverse-probability weighting addresses selection only when the participation probabilities are correctly modeled from sufficient measured variables and positivity holds. Extreme weights can also make estimates unstable.
- Calling prevalence calibration a performance improvement. An intercept correction can align risk levels with a target prevalence under its assumptions. It does not improve ranking or establish that the score transports across ancestry groups.
- Comparing numbers with different meanings. A risk difference, AUC, calibration intercept, and predicted absolute risk answer different questions. State which quantity changed before interpreting the change.
- Attributing every change to one mechanism. Ancestry composition and participation can operate together. A sensitivity analysis indicates dependence on assumptions; it rarely isolates a mechanism without appropriate data and a defensible design.
Check Your Understanding
Answer each question using the definitions and calculations above.
- In the first example, why is the unadjusted risk difference negative even though both within-ancestry risk differences are positive?
- What assumptions are needed to interpret inverse-probability-weighted risks as estimates for a source population?
- For what selection mechanism is a prevalence-based logistic intercept correction appropriate, and what does it leave unchanged?
- Why does a change after ancestry adjustment indicate sensitivity rather than automatically establish that the adjusted result is unbiased?
- Give one reason a large or unstable participation weight should prompt caution when interpreting weighted performance.