Why PRS Studies Can Fail in Different Ways
A polygenic risk score (PRS) combines information across many genetic variants, but a large score or a statistically significant association does not by itself establish useful prediction. A score can be computed incorrectly, perform well only in the data used to construct it, give inaccurate absolute risks, or be described as portable when evidence covers only a narrow target population. These failures have different causes and need different checks; recalibration, for example, cannot repair a score whose alleles were counted in the wrong direction.
The case studies below treat each score as a weighted sum of genotype dosages. For person \(i\), let \(G_{ij}\) be the number of copies, from 0 to 2, of the specified effect allele at variant \(j\), and let \(\beta_j\) be its weight. The score is \(S_i=\sum_j \beta_jG_{ij}\). For a binary outcome, weights may be log-odds effects; the score can then enter a logistic model with an intercept and, where appropriate, clinical covariates. The examples are hypothetical and are designed to isolate diagnostic reasoning, not to represent actual study findings.
Allele Misalignment: A Score Can Be Wrong Before It Is Evaluated
Each weight must be multiplied by dosage of the allele to which that weight refers. Thus, score construction assumes that variant identifiers match, alleles are represented consistently, and the dosage column counts the specified effect allele. A strand flip, an effect-allele swap, or a mistaken dosage label can violate this assumption. The error may affect only some variants, making it hard to spot from the overall score distribution.
Worked Example: A Dosage Column Counts the Other Allele
Suppose a two-variant score has weights \(0.20\) per copy of allele A at variant 1 and \(0.30\) per copy of allele G at variant 2. For one hypothetical person, the target data report one copy of A at variant 1 and two copies of C at variant 2. Assume each site is biallelic, diploid, and the reported alleles refer to the same genome build and variant positions as the weights.
At variant 1 the dosage already counts the effect allele A, so its contribution is \(0.20(1)=0.20\). At variant 2, C is the other allele: two copies of C mean zero copies of G. The correctly aligned score is therefore \(0.20(1)+0.30(0)=0.20\).
If the C dosage is incorrectly treated as a G dosage, the computed score becomes \(0.20(1)+0.30(2)=0.80\), an error of \(0.60\) log-odds units. Under the stated logistic interpretation, the incorrect score changes the implied odds by a factor of \(\exp(0.60)=1.822\), relative to the correctly computed score with the same intercept and covariates. This arithmetic demonstrates why allele orientation must be checked, not inferred from a score’s plausible-looking range.
A useful audit compares variant identifiers, reference and alternate alleles, effect alleles, genome build, strand, and the meaning of the target dosage field. Variants with ambiguous strand orientation, especially A/T and C/G, need particular care because allele labels alone may not reveal a strand flip. Frequency comparisons can help flag inconsistencies, but are not a substitute for resolving the representation.
LD Mismatch: A Tag Variant May Not Tag the Same Signal
A score can be harmonized correctly and still transfer poorly because a weighted variant may be a proxy for causal genetic signal rather than the causal variant itself. The proxy’s relationship to that signal depends on linkage disequilibrium (LD). The key assumptions in the simplified calculation below are that genotypes are standardized, the phenotype follows a linear genetic component plus independent residual noise, and the proxy’s weight is held fixed across populations. Real scores can involve many variants and more complicated LD patterns, but the example shows the mechanism.
Worked Example: A Tag Variant Transfers Less Well
Suppose the causal genotype has effect \(\beta=0.40\) on a standardized continuous outcome, and a score uses one standardized tag genotype with weight \(w=0.40\). In a discovery population, assume the tag–causal correlation is \(0.80\). The score–outcome covariance attributable to this signal is \(0.40(0.40)(0.80)=0.128\).
In a target population, suppose the correlation is \(0.20\), while the effect and score weight are unchanged. The corresponding covariance is \(0.40(0.40)(0.20)=0.032\). It is one quarter of the discovery value, because \(0.032/0.128=0.25\). This is not a complete prediction metric: score and outcome variances, other variants, and noise also matter. It does show why applying the same tag weight does not guarantee the same predictive contribution.
A practical diagnosis checks whether variants were directly measured or imputed, whether imputation quality is adequate, whether the relevant LD structure differs in the target data, and whether effects are heterogeneous. Compare performance and uncertainty across suitably defined target groups rather than attributing every loss to LD alone. The earlier material on diagnosing portability loss from frequency and LD differences gives a fuller decomposition; here, the important failure-mode lesson is not to interpret a transferred score’s weakness as automatically fixable by changing its calibration.
Overfitting and Reuse of Evaluation Data
Overfitting occurs when choices are adapted to random features of the data used to estimate or select a score. Choices can include variants, tuning parameters, ancestry-specific weights, covariates, and the performance metric or subgroup highlighted in the report. If many candidate models are evaluated on one validation set and only the best is reported, that set has become part of model selection; its winning performance is not an independent estimate of future performance. A final test set is informative only if it was held out from all such decisions.
Worked Example: A Tuning Set Becomes an Optimistic Estimate
A team evaluates ten hypothetical PRS configurations in a tuning sample of 600 people. The selected configuration has an observed area under the receiver operating characteristic curve (AUC) of \(0.74\) in that sample. The team then reports this same \(0.74\) as its estimate of performance in new patients. In a separate, untouched test sample of 800 people, the selected score has AUC \(0.69\).
The observed difference is \(0.74-0.69=0.05\) AUC units. This does not prove that exactly \(0.05\) was caused by overfitting: the two samples also have sampling variability and may differ in case mix. But selecting the maximum of ten noisy estimates tends to favor a configuration with positive noise in the tuning sample. Reporting the tuning AUC as independent confirmation therefore overstates what the data establish.
The correction is to keep model development and evaluation distinct. Use resampling or a designated development set to choose the model, freeze the choices, and evaluate once in an independent test set; if the available sample is limited, use a validation design that accounts for the full selection procedure. Report the population and uncertainty for the final estimate. A small train–test gap is not proof of no leakage, and a large gap can also reflect distribution shift, so the analysis history and data partitioning must be checked.
Calibration Failure and Its Distinction From Discrimination
Discrimination concerns how well predictions rank people with and without the outcome. Calibration concerns agreement between predicted risks and observed outcome frequencies. A score may rank individuals reasonably well yet systematically underpredict risk. For binary outcomes, if \(p_i\) is the predicted risk, a common calibration model is \(\operatorname{logit}P(Y_i=1)=a+b\operatorname{logit}(p_i)\) in validation data. Ideal calibration has intercept \(a=0\) and slope \(b=1\); deviations indicate systematic level or spread errors. Calibration estimates require enough outcome information and should be reported with uncertainty.
Worked Example: Detecting Underprediction in a Validation Group
Suppose a frozen score assigns every person in a hypothetical validation group a predicted risk of \(0.10\). Among 100 people, 16 experience the outcome. The predicted number of events is \(100(0.10)=10\), compared with 16 observed, so the observed-to-expected ratio is \(16/10=1.60\). In this group, predicted risk is too low on average; this comparison alone does not tell us whether the score ranks people well.
For an illustrative intercept-only recalibration with slope fixed at 1, the adjustment is $$ a=\operatorname{logit}(0.16)-\operatorname{logit}(0.10) =\log(0.16/0.84)-\log(0.10/0.90) =-1.6582-(-2.1972)=0.5390. $$ Adding this intercept changes a predicted risk of \(0.10\) to \(\operatorname{logit}^{-1}[-2.1972+0.5390]=0.160\), matching the observed rate in this deliberately homogeneous example. It does not establish that 0.160 is the right risk for future patients, nor does it show that the calibration slope is 1. Real recalibration should be assessed in appropriate data with uncertainty and should not be mistaken for new evidence of discrimination.
Portability Claims Must Match the Evidence
“Portable” is not a property established by a single successful evaluation. A defensible claim specifies the target population, outcome definition, prediction horizon, comparator, and performance measure. It distinguishes association from absolute-risk calibration and reports uncertainty, particularly when subgroup sample sizes or event counts are small. An ancestry label is also not a complete description of a target population: populations within a label can differ, and socially assigned race is not interchangeable with genetic ancestry. Differences in health systems, ascertainment, environment, and baseline risk can affect validation results as well.
Worked Example: A Narrow Evaluation Does Not Support a Broad Claim
A hypothetical study tests a score in 120 volunteers recruited at one clinic. The sample includes 40 people assigned to a second ancestry group by the study’s genetic-ancestry procedure. The score’s AUC is \(0.70\) in those 40 people, and the authors describe it as “portable across ancestry groups.” That conclusion exceeds the evidence: the estimate concerns one small subgroup in one recruitment setting, and an AUC alone says nothing about whether its absolute risks are calibrated.
A better conclusion states the measured result and its scope: “In this clinic sample, the estimated AUC in the defined subgroup was 0.70; the sample was small, and calibration and performance in other target populations remain unestablished.” The study should report an uncertainty interval and, where feasible, examine calibration and relevant comparators. A broad portability claim requires validation evidence across the intended settings, not just a statistically nonzero association or a favorable point estimate in one subgroup.
Diagnostic Sequence, Remedies, and Limitations
A good response to a disappointing or unusually strong result is to locate the failure before changing the score. The following sequence separates checks that address computation from checks that address prediction and interpretation.
Verify variant matching, effect-allele orientation, dosage meaning, genome build, missingness, and quality control. Recompute a small set of individual scores by hand or with an independent implementation.
Confirm that tuning, selection, and preprocessing did not use the final test outcomes. Check sample overlap, relatedness where relevant, outcome ascertainment, and whether the test sample represents the stated target.
Report discrimination and calibration as distinct properties, with uncertainty and a relevant comparator. A favorable ranking metric does not certify accurate absolute risks.
Fix allele errors before recalibration; investigate LD and effect differences before asserting transfer; use independent evaluation to measure selection optimism; recalibrate only when the score is otherwise suitable and target-population calibration is the problem.
| Observed warning | Potential cause | Appropriate next check |
|---|---|---|
| Unexpected score direction or extreme values | Allele or dosage misalignment | Audit effect alleles and recompute selected scores |
| Lower target performance despite correct harmonization | LD differences, effect heterogeneity, or target-data quality | Check imputation, LD, subgroup estimates, and uncertainty |
| Strong tuning result but weaker independent test | Selection optimism or population shift | Review the selection history and compare target samples |
| Good ranking but observed risks exceed predictions | Calibration error or changed baseline risk | Estimate calibration with adequate validation data |
| Broad claim from one narrow sample | Overgeneralization | Restrict the claim or obtain validation in intended populations |
These checks have limits. A clean allele audit cannot rule out model misspecification; an independent sample can still differ from the intended deployment population; and subgroup estimates can be imprecise. Recalibration can adjust a risk scale but cannot create information absent from the score or guarantee transfer elsewhere. Likewise, changing weights to improve a small target sample can overfit that sample. Every remedy should be followed by evaluation in data not used to choose it.
Common Mistake
Wrong reasoning: “The PRS is associated with the outcome and has a good AUC, so it is correctly implemented, calibrated, and portable.” Each clause makes a separate claim. Association and AUC do not verify allele alignment, absolute-risk calibration, or performance in populations not evaluated. Correction: audit score construction, report distinct prediction metrics, quantify uncertainty, and limit generalization to the populations and settings supported by independent validation.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A score weight is defined per copy of allele T, but the target dosage field counts allele C at a biallelic site. What must be checked before using the weight?
- In the LD example, what would happen to the tag’s covariance contribution if its correlation with the causal genotype were zero, under the stated assumptions?
- A model is selected from 20 candidates using one validation set. Can its best score on that set be reported as an independent test estimate? Why or why not?
- A PRS has good discrimination but predicts risks that are systematically too low. Which property is failing, and what does the discrimination result not establish?
- A study evaluates one small subgroup and claims portability to all populations. State one limitation of the claim and one type of additional evidence needed.
Solutions: (1) Confirm allele orientation and what the dosage counts; if C is the other allele and the site is correctly harmonized, convert the dosage to T dosage before multiplying by the T weight. (2) The covariance contribution is \(w\beta(0)=0\), so that tag contributes no covariance with the causal genetic component in this simplified model. (3) No. Choosing the maximum over candidates uses that set for selection, so its winning estimate can be optimistic; evaluate the frozen model in data untouched by selection. (4) Calibration is failing. Good discrimination does not establish that predicted absolute risks match observed frequencies. (5) The subgroup may not represent other populations and its estimate may be imprecise; validation in additional intended target populations, with uncertainty and calibration assessment, is needed.