Why Score Selection and Score Evaluation Must Be Separate
A polygenic risk score (PRS) depends on analysis choices as well as genetic data. For example, a clumping-and-thresholding score depends on a \(p\)-value threshold and clumping settings; a shrinkage score may depend on prior or model parameters. Different choices can produce different predictions for the same participants. If the choice is made by comparing performance on a final test set, that set has helped build the score rather than providing an independent assessment of it.
The goal is to use one set of data to choose among prespecified score specifications and a separate set to estimate the selected score's performance. A discovery sample supplies association information used to construct candidate scores. A tuning sample compares those candidates. A final test sample is held aside until the score and analysis plan are fixed. The estimate from that test sample then concerns the selected procedure, evaluated in that sample's target population—not a hypothetical score chosen without data.
For participant \(i\), a weighted score can be written as \(S_i=\sum_{j=1}^{m}G_{ij}\hat{\beta}_j\), where \(G_{ij}\) is the genotype dosage at variant \(j\), \(\hat{\beta}_j\) is its estimated effect, and \(m\) is the number of included variants. A hyperparameter choice can change which variants enter the sum or how their effects are estimated. It does not change the meaning of the score as a weighted sum.
The split is defined by participants and data roles, not merely by files. The discovery data used to estimate effects should not contain the tuning or test participants if the goal is independent evaluation. If discovery GWAS data overlap with the test cohort, the test outcomes may have contributed to the effect estimates; apparent performance can then be optimistic. Related participants should generally be kept in the same partition, because close relatives across partitions can make evaluation less independent. The partitions should also represent the intended target population and measurement setting closely enough that the test result is relevant to the intended use.
Which Choices Are Tuned?
The distinction is about how a setting was chosen, not its name. A \(p\)-value threshold fixed in advance from a scientific rationale is not selected using the tuning sample. If several thresholds are compared and the best-performing one is kept, the threshold has been tuned. The same applies to choices such as clumping cutoffs, shrinkage parameters, and the method used to combine score components.
The tuning sample may also be used to choose among candidate model families, provided that the alternatives and the selection rule are specified before inspecting the final test results. Choices that can quietly become tuning decisions include how to define the phenotype, which covariates to include, how to handle missing data, which performance metric to emphasize, and which subgroup result to report. Repeatedly changing these decisions in response to test performance makes the test set part of model selection.
The performance metric should match the prediction task and be fixed before comparison. For a quantitative phenotype, one possible metric is \(R^2=1-\mathrm{SSE}/\mathrm{SST}\), where \(\mathrm{SSE}\) is the sum of squared prediction errors and \(\mathrm{SST}\) is the sum of squared deviations from the observed mean. \(R^2\) is dimensionless. Mean squared error, in contrast, has squared outcome units. The choice of metric can affect which candidate is selected, so it is part of the analysis plan, not a neutral afterthought.
Worked Example: Separating Three Data Roles
Suppose investigators have an external discovery GWAS and a target cohort of 2,000 unrelated participants with measured outcomes. They plan to compare three clumping-and-thresholding settings. They randomly assign 800 target participants to tuning and 1,200 to a final test set, keeping all records from each participant together.
The discovery GWAS supplies the association estimates for all three candidate scores. The 800 tuning participants are used to calculate the prespecified performance metric for each candidate and select one setting. The 1,200 test participants are not used to select settings, alter score construction, or change the metric.
Suppose tuning \(R^2\) values are \(0.071\), \(0.084\), and \(0.079\). The second setting is selected because \(0.084\) is the largest. After the score and evaluation plan are locked, its predictions are evaluated once in the test sample. If the test \(R^2\) is \(0.076\), the reported result is a dimensionless \(R^2\) of \(0.076\) for those test participants. It is not measured in squared-outcome units.
This design is appropriate if the partitions are independent at the participant level, the discovery GWAS does not overlap with the target cohort, and the test sample reflects the population to which the result is meant to apply. If relatives crossed the tuning/test boundary or the discovery GWAS included the test participants, the stated independence would be compromised and should be addressed or disclosed.
A Practical Selection and Evaluation Workflow
A defensible workflow makes the roles of the data explicit before looking at comparative performance. The following sequence is sufficient for a simple holdout design; more complex validation designs require additional care.
State the phenotype, target population, and one primary performance measure. Specify how the metric is calculated and which direction indicates better performance.
Identify discovery data for effect estimation, tuning data for choosing settings, and a separate test sample for final evaluation. Check participant overlap and relatedness across partitions.
Use discovery data to create candidate scores under the planned settings. Compare their tuning performance using the same metric and the same rules for all candidates.
Record the chosen setting, score construction, phenotype definition, covariates, and metric before accessing test performance. Do not revise them in response to the test result.
Apply the locked procedure to the test participants and report the resulting metric, sample size, and relevant uncertainty. Describe the test set as an evaluation sample, not as another tuning resource.
Sometimes investigators refit parts of the procedure after choosing settings, for example by re-estimating quantities using additional development data. That refit must be specified as part of the procedure, and the test set must remain untouched. A test result for one fitted score does not automatically evaluate a materially different score refitted after the result was seen.
Worked Example: Selecting a Threshold Without Reading the Test Results
Assume a discovery GWAS has supplied the effect estimates and three candidate \(p\)-value thresholds are \(0.001\), \(0.01\), and \(0.05\). The prespecified metric is \(R^2\) for a quantitative phenotype. In a hypothetical tuning sample of 500 participants, the observed values are \(0.081\), \(0.094\), and \(0.089\), respectively.
The selection rule is to choose the threshold with the largest tuning \(R^2\). Since \(0.094>0.089>0.081\), the selected threshold is \(0.01\). This choice uses only the tuning data and the fixed rule. Assume the discovery GWAS does not overlap with either target partition, the tuning and test samples contain different unrelated participants, and the same outcome definition and metric are used throughout.
The selected score is then evaluated in a separate hypothetical test sample of 700 participants. Suppose its test \(R^2\) is \(0.087\). Report the dimensionless value \(R^2=0.087\) for the test sample; do not describe it as \(0.087\) squared-outcome units. The tuning \(R^2\) of \(0.094\) served selection, while the test \(R^2\) estimates performance for the selected procedure in new participants from the relevant target population.
If the investigators also calculated all three test \(R^2\) values and chose whichever was largest, the test set would no longer provide an independent evaluation of a prespecified selected score. Even if the chosen test value happened to be \(0.087\), the procedure used to obtain it would be different and more optimistic on average.
Why Reusing the Test Set Causes Optimism
A performance estimate varies from sample to sample. When several candidates are evaluated on the same test sample and the one with the best observed value is reported, the selection tends to favor candidates whose estimates were unusually high because of sampling variation. The selected candidate's reported performance is therefore generally higher than its true performance, even if the candidates are equally good.
A simple derivation makes the mechanism visible. Let two candidates have the same true performance \(M\), and let their estimated performances be \(\hat{M}_1=M+\epsilon_1\) and \(\hat{M}_2=M+\epsilon_2\). Suppose, for illustration, that the errors are independent and each has a normal distribution with mean zero and standard deviation \(\sigma\). Selecting the larger estimate gives:
Because each error has mean zero, the expected contribution from \((\epsilon_1+\epsilon_2)/2\) is zero. Also, \(\epsilon_1-\epsilon_2\) has a normal distribution with mean zero and standard deviation \(\sqrt{2}\sigma\); the mean absolute value of a zero-mean normal variable with standard deviation \(\tau\) is \(\tau\sqrt{2/\pi}\). Therefore:
Thus, in this illustrative setting, choosing the larger of two noisy estimates creates expected optimism of \(\sigma/\sqrt{\pi}\). With \(\sigma=0.04\), that amount is \(0.04/\sqrt{\pi}\approx0.0226\). If both candidates truly have \(R^2=0.10\), the expected larger observed value is about \(0.123\), not \(0.10\). This is a simplified calculation, not a universal correction: real metrics may be bounded, candidate estimates may be correlated, and candidates may have different true performance. The general selection effect remains: choosing based on favorable noise biases the selected estimate upward on average.
Worked Example: Repeated Test-Set Inspection
Suppose a team compares four PRS specifications on a test set and observes \(R^2\) values of \(0.102\), \(0.108\), \(0.105\), and \(0.111\). It selects the fourth and reports \(0.111\) as the score's final test performance. The arithmetic maximum is \(0.111\), but it is also the maximum of four estimates that were used to make the selection.
The problem is not that \(0.111\) was calculated incorrectly. The problem is that the data were used both to choose the winning specification and to estimate the winner's performance. Sampling variation helped determine which value was largest, so the reported maximum is expected to overstate the performance of the selected procedure.
If the four specifications had been compared on tuning data and the fourth selected there, a test result calculated once for that locked specification would serve a different role. The test result could still vary by chance, and its precision would depend on the sample, but it would not have been selected for being the largest among these four test estimates.
Diagnostics, Alternatives, and Limitations
A clean separation is not guaranteed by naming a file “test.” Review the analysis history and ask whether test outcomes influenced any choice, including threshold, covariates, transformations, missing-data rules, subgroup emphasis, or which result to report. Check participant identifiers across discovery, tuning, and test sources; check for relatedness across partitions; and verify that no test-derived summary was used to revise the score. A test set that has been repeatedly inspected is no longer a pristine final test set, even if no single formal optimization routine was run.
A single tuning split is straightforward but can yield a setting that depends strongly on which participants happened to enter the tuning group. Repeated resampling or cross-validation within development data can help characterize or stabilize tuning decisions, but it does not make it acceptable to choose a model using final test results. The central comparison is not “holdout versus resampling” in isolation: it is whether the data used for final evaluation were kept out of all selection decisions. More elaborate nested validation addresses the organization of repeated tuning and evaluation, but the basic separation principle is the same.
There are practical limits. A smaller tuning sample may make candidate rankings unstable, and a small test sample may give an imprecise performance estimate. An independent test sample may also differ from the population where the score will be used, so its result may not transport to that setting. No split can repair biased discovery effects, poor phenotype measurement, or a mismatch between the target population and the data. Report the data roles and any overlap transparently rather than implying independence that the design does not support.
- Wrong: “We tried several thresholds on the test set and reported the one with the highest \(R^2\), so that \(R^2\) is an unbiased test estimate.” Correction: The threshold was selected using the test outcomes; the maximum is subject to selection optimism.
- Wrong: “The test set was not used to fit SNP effects, so it could not leak information.” Correction: Using test outcomes to choose thresholds, covariates, or reporting decisions also reuses test information.
- Wrong: “The test \(R^2\) is measured in squared phenotype units.” Correction: \(R^2\) is dimensionless; mean squared error has squared phenotype units.
Check Your Understanding
Answer each question, then compare with the solutions.
- A team compares six shrinkage settings using a tuning sample, selects the best, and evaluates it once in an untouched test sample. Which sample selects the setting, and which estimates final performance?
- Why is selecting the largest test \(R^2\) among several thresholds likely to overstate the selected score's performance?
- A score has \(R^2=0.14\). Is this value dimensionless or in squared-outcome units?
- Discovery GWAS summary statistics include participants from the final test cohort. What concern does this create?
- Name two choices besides SNP effect estimation that can constitute tuning if they are changed in response to observed performance.
Solutions. (1) The tuning sample selects the setting; the untouched test sample evaluates the selected score. (2) The maximum tends to include favorable sampling error, because the estimates were compared and the largest was chosen. (3) It is dimensionless. (4) The test outcomes may have contributed to discovery effect estimates, compromising independent evaluation and potentially inflating apparent performance. (5) Examples include the phenotype definition, covariate set, clumping settings, missing-data rule, metric, or subgroup chosen for emphasis.