Why the Best Reported Result Can Be Too Optimistic
A polygenic score evaluation may compare several scores, phenotypes, subgroups, or performance metrics. If investigators report only whichever result looks strongest, the reported value reflects two things: the score’s underlying predictive performance and the random error in its estimate. The selected result is more likely than an unselected result to have benefited from a favorable error. This is selection-induced optimism.
The issue arises even when every analysis is conducted correctly and each performance estimate would be unbiased if considered on its own. Selection changes the question: instead of asking how one prespecified evaluation performed, the report describes the largest observed result among a collection. The size of the resulting optimism depends on how many comparisons were effectively made, how noisy their estimates are, and how strongly their errors are correlated.
To reason clearly, first define the candidate analyses and a common performance estimand. Write \(\theta_j\) for the true target-population performance of candidate analysis \(j\), and \(\hat{\theta}_j\) for its estimate. Orient the metric so that larger values mean better performance. For an error measure where smaller is better, this can be done by considering its negative, while retaining the original units and explaining the direction.
The common-performance setting isolates the selection effect: there is no genuine difference among candidates, so any advantage of the reported maximum comes from estimation error. In real evaluations, candidates may genuinely differ. Then selection can identify a better candidate, but the reported maximum can still overstate that selected candidate’s performance. The simple calculations below quantify a basic case; they are not a universal correction for every analysis pipeline.
A Two-Candidate Derivation
Suppose two candidates have the same true performance \(\theta\), and their estimates have independent, normally distributed errors with standard deviation \(\tau\):
Each estimate is unbiased before selection: \(E(\hat{\theta}_j)=\theta\). But the larger estimate is not unbiased for \(\theta\). To derive the amount of optimism, use the identity \(\max(a,b)=(a+b+|a-b|)/2\):
The second equality follows because \(Z_1-Z_2\sim N(0,2)\), and a mean-zero normal variable with standard deviation \(\sqrt{2}\) has expected absolute value \(\sqrt{2}\sqrt{2/\pi}\). Thus the upward bias of the selected maximum is about \(0.5642\tau\), despite both candidate estimates being unbiased individually. This is a repeated-sampling expectation, not a claim that a particular reported estimate must exceed the truth by that amount.
Worked Example: Selecting the Better of Two Scores
Suppose two hypothetical polygenic scores have the same true AUC of \(0.68\) in a target population. Each estimated AUC has standard error \(0.04\), and, for this calculation, the errors are treated as independent and approximately normal. Investigators report the larger estimate.
The expected selection optimism is:
Thus, under these assumptions, the expected reported maximum is approximately \(0.68+0.02257=0.7026\). AUC is unitless, so the difference is about 0.0226 AUC units. This is not a correction that should automatically be subtracted from any observed maximum: the calculation assumes equal true AUCs, independent normal errors, and a known standard error.
For instance, if the two observed AUC estimates were \(0.71\) and \(0.68\), the selected estimate would be \(0.71\). The model says the selection procedure has positive expected optimism across repetitions; it does not establish that the true AUC is \(0.71-0.02257\), nor does it determine the true value from these two observations.
More Candidates, More Opportunity to Select Noise
With \(K\) candidates sharing the same true performance and independent normal errors with common standard error \(\tau\), the selected maximum has the form \(\theta+\tau\max_j Z_j\), where each \(Z_j\) is standard normal. Its expected optimism is:
where \(\phi\) and \(\Phi\) are the standard normal density and cumulative distribution function. The integral follows because the maximum has cumulative distribution function \(P(\max_j Z_j\leq z)=\Phi(z)^K\), whose density is \(K\phi(z)\Phi(z)^{K-1}\). For \(K>1\), the expectation is positive and grows as more candidates are considered. The growth is not linear in \(K\), but a larger search generally creates more opportunity for a favorable error to win.
Worked Example: Comparing Four Candidate Phenotypes
In a hypothetical evaluation, investigators consider four phenotypes, all assumed for illustration to have the same true score performance. Each estimated performance has standard error \(0.02\), and the four errors are treated as independent and normal. The largest estimate is selected for reporting.
For four independent standard normal variables, the expected maximum is approximately \(1.0294\). Therefore, the expected optimism in the selected performance estimate is:
If the common true performance were \(0.10\) on a unitless metric such as \(R^2\), the expected selected maximum under this model would be about \(0.1206\). The difference, \(0.0206\) \(R^2\) units, is selection optimism in expectation. It does not mean the observed maximum in every four-phenotype evaluation will be exactly \(0.1206\), or that the four phenotypes truly have equal performance.
The independence assumption matters. Phenotypes measured in the same participants, or scores built from related weights, can produce correlated estimation errors. Positive correlation often reduces the amount of independent information in the search relative to four independent comparisons, but the effect depends on the full dependence structure. Counting the number of reported results alone does not recover that structure.
What Counts as a Comparison?
The relevant set of candidates is the set that could have influenced the reporting choice, not merely the results eventually shown in a table. A search may involve different score constructions, phenotypes, ancestry groups, sex or age subgroups, covariate adjustments, evaluation metrics, and analysis choices. If investigators inspect these results and then highlight the most favorable one, each choice can contribute to selection.
The scale also matters. Selecting the largest AUC is not the same decision as selecting the largest \(R^2\), and the metrics may target different aspects of prediction. A subgroup result may be scientifically useful, but comparing many subgroups and reporting only the strongest one does not estimate the prespecified performance for the overall population. The target quantity and the selection rule should therefore be stated together.
A nominal confidence interval or p-value calculated as if the selected analysis had been specified in advance does not generally account for this search. Nor is the number of comparisons always obvious: exploratory analyses may be modified in response to earlier results. A defensible account describes the search process and distinguishes confirmatory analyses, specified before seeing evaluation results, from exploratory analyses chosen after inspecting them.
Worked Example: A Prespecified Metric and a Separate Evaluation Set
Consider a hypothetical study that develops several score versions in a discovery sample. Before examining evaluation outcomes, investigators specify one score, one phenotype, the evaluation population, and AUC as the primary performance metric. They then lock the score and evaluate it once in a separate sample that did not contribute to score construction or selection.
Because the score and primary evaluation were chosen without using the separate sample’s outcome results, the investigators have not selected the reported AUC from that sample’s competing estimates. This design removes the optimism specifically caused by choosing the best result on the evaluation data. It does not guarantee that every other source of bias is absent: the evaluation sample must still be relevant to the stated target, and the AUC estimator and study procedures must be appropriate.
If investigators instead inspect AUCs for five score versions in that evaluation sample and report only the largest, the sample has become part of the selection process. A confidence interval calculated as though that winning score had been the only candidate can be too optimistic or too narrow for the reporting procedure. A fresh, independent evaluation of the locked winner can provide a cleaner estimate, provided the second sample was not used to alter the score or choose what to report.
Prespecification and Selection-Aware Inference
The most direct way to prevent selection-induced optimism in a primary result is to specify the analysis before examining the relevant evaluation outcomes. A useful prespecification identifies the score, phenotype, subgroup or population, performance metric, direction of improvement, and primary comparison. Other analyses can still be reported, but should be labeled as exploratory rather than presented as if they were the sole planned test.
When selection has already occurred, one option is inference that accounts for the full candidate set. For example, if each of \(K\) estimates has a valid approximately normal standard error \(SE\), a Bonferroni simultaneous two-sided confidence procedure uses critical value \(z_{1-\alpha/(2K)}\). For each candidate \(j\), form:
If each marginal interval has coverage at least \(1-\alpha/K\), the probability that at least one of the \(K\) intervals fails to cover its corresponding true value is at most the sum of their failure probabilities, which is at most \(\alpha\). This follows from the union bound and does not require independence among the estimates. Consequently, with probability at least \(1-\alpha\), all candidate intervals cover simultaneously. The interval belonging to the selected candidate then covers its true performance as part of that simultaneous event.
Worked Example: A Simultaneous Interval for the Selected Result
Suppose five hypothetical candidate evaluations are compared, and the selected estimate is \(0.12\) on a unitless performance scale. Its standard error is \(0.025\). Assume the five estimates have valid approximately normal sampling distributions and use a familywise confidence level of 95%. A two-sided Bonferroni critical value is \(z_{1-0.05/(2\times5)}=z_{0.995}\approx2.5758\).
The margin of error is:
The interval for the selected result is therefore:
Under the stated assumptions, this is the selected candidate’s Bonferroni simultaneous interval: the procedure covers all five true performances with probability at least 95% across repeated samples, so it also covers the selected candidate’s true performance on that event. This is generally wider than an ordinary interval for one prespecified estimate. It does not remove all possible bias in the performance estimates or fix a mismatch between the evaluation sample and the intended population.
Bonferroni adjustment is transparent and does not rely on independence, but it can be conservative, especially when candidate results are strongly correlated. Other selection-aware approaches can use the joint sampling distribution, resampling that repeats the entire selection process, or a fresh validation sample. Any such method must represent the actual choices that led to the reported result. Adjusting only for the comparisons disclosed after the fact may not address an unrecorded or adaptive search.
Common Mistakes and Practical Interpretation
- “Each estimate is unbiased, so the best one is unbiased.” Unbiasedness applies to each estimate considered before selection. Taking a maximum changes its expected value; the two-candidate derivation shows a positive difference even when both estimates are unbiased.
- “The expected optimism is the amount to subtract from my result.” A model-based expectation is not a known correction for one observed study. Subtracting it as if it were exact requires assumptions about the candidate set, error distribution, standard errors, and dependence that may not hold.
- “I only reported one result, so there was only one comparison.” The number reported is not necessarily the number considered. Results inspected and then omitted may also have influenced the choice.
- “A significant or strong result proves the score predicts well.” A favorable selected estimate does not by itself establish performance in a target population. Selection adjustment addresses one aspect of uncertainty; it does not establish transportability, remove population-structure contamination, or demonstrate causation.
- “A Bonferroni interval validates the whole evaluation.” It controls simultaneous coverage under its stated model and valid standard errors. It does not account automatically for unmodeled data reuse, invalid metric estimation, or selection decisions outside the declared family.
The practical goal is not to prohibit comparisons. Comparing scores, phenotypes, and subgroups can reveal useful patterns. The essential distinction is whether the reported quantity is a prespecified evaluation or the winner of a search. Report the candidate set and selection rule, give uncertainty appropriate to that process, and use independent evaluation when feasible. This makes the strength of a reported polygenic-score result easier to interpret without confusing a favorable selected estimate with a guaranteed level of future performance.
Check Your Understanding
Answer these questions using the definitions and assumptions stated above.
- Two equally performing score evaluations have independent normal errors with standard error \(0.03\). What is the expected optimism of their selected maximum under the two-candidate model?
- Why can each of several performance estimates be unbiased while the largest one is upwardly biased?
- In a search across scores, phenotypes, and subgroups, what determines which comparisons are relevant to selection-aware inference?
- What is the coverage interpretation of Bonferroni simultaneous intervals for a selected candidate?
- Name one form of optimism that a separate, prespecified evaluation sample can reduce, and one important concern it does not automatically resolve.