Why the P-Value Cutoff Must Be Validated
A clumping-and-thresholding score includes variants whose discovery-study association p-values pass a chosen cutoff. A stringent cutoff includes fewer variants; a more permissive one can include many more. Because the score’s variant set changes with the cutoff, its performance may change too. The cutoff is therefore a tuning choice: it should be selected by comparing candidate scores against outcomes in a validation cohort, not by assuming that the smallest discovery p-values necessarily produce the best score.
A fair comparison holds the rest of the analysis fixed. Use the same validation individuals, outcome definition, covariates, outcome-model form, and performance metric for every candidate cutoff. Otherwise, a difference attributed to the cutoff could instead arise from a different sample, model, or evaluation rule.
This definition treats the clumped variant set, discovery effects, and allele coding as fixed while the p-value cutoff varies. In an actual analysis, the candidate cutoffs should also be stated in advance or generated by a rule chosen without consulting the validation outcomes. For example, one might compare \(10^{-8}\), \(10^{-5}\), \(10^{-3}\), and \(1\). A cutoff of \(1\) includes all variants that passed the fixed clumping and quality-control steps.
Specify the Comparison Before Fitting
For a quantitative outcome, a common comparison fits the same regression model at every cutoff:
Here \(Y_i\) is the outcome, \(\mathbf{c}_i\) is the vector of prespecified covariates, and \(S_{it}\) is the score at cutoff \(t\). The intercept and regression coefficients may differ across cutoffs because each score differs. The covariate definitions and the model form do not. If prediction is the goal, a metric such as mean squared error (MSE) or validation \(R^2\) can be specified in advance. Smaller MSE is better; larger \(R^2\) is better.
For a binary outcome, one option is the same logistic model form at every cutoff: the log odds of the outcome are modeled as a linear function of the covariates and score. The metric might be log loss, which assesses predicted probabilities, or the area under the receiver operating characteristic curve (AUC), which measures ranking. These metrics answer different questions. The choice should follow the intended use of the score, not whichever metric happens to favor a preferred cutoff.
The validation cohort should represent the population and outcome setting in which the score is intended to be used. Individuals must be independent for the usual interpretation of row-wise validation metrics; related individuals or other clustering can make apparent performance more favorable than performance in new, independent individuals. The discovery data must also be kept separate from the validation outcomes for this comparison: if the same outcomes helped determine the discovery effects and are then used to select the cutoff, the comparison is no longer an independent validation.
Worked Example: Comparing Quantitative-Outcome Scores
Worked Example: Comparing Quantitative-Outcome Scores
Consider a small, entirely hypothetical validation set of five individuals. Let the quantitative outcome be \(Y=(1,2,3,4,5)\). For illustration, compare two candidate scores using the same model at each cutoff: \(Y_i=\alpha_t+\beta_t S_{it}+\varepsilon_{it}\), with an intercept and no additional covariates. The candidate scores are \(S_A=(-2,-1,0,1,2)\) and \(S_B=(-1,1,-1,1,0)\). Suppose these are the scores produced by two different p-value cutoffs after the clumping step.
The outcome mean is \(\bar{Y}=3\), and its total sum of squares is \(SST=\sum_i(Y_i-\bar{Y})^2=4+1+0+1+4=10\). For an intercept-only comparison with one score, the validation \(R^2\) is \(1-SSE/SST\), where \(SSE=\sum_i(Y_i-\widehat{Y}_i)^2\).
For score \(A\), the mean score is zero, the fitted slope is \(\hat{\beta}_A=\sum_i(S_{iA}-\bar{S}_A)(Y_i-\bar{Y})/\sum_i(S_{iA}-\bar{S}_A)^2=10/10=1\), and the fitted intercept is \(\hat{\alpha}_A=3\). The predictions are \(1,2,3,4,5\), so \(SSE_A=0\) and \(R^2_A=1-0/10=1.00\).
For score \(B\), its mean is also zero. The cross-product with the centered outcome is \(2-1+0+1+0=2\), and \(\sum_i S_{iB}^2=4\), so \(\hat{\beta}_B=2/4=0.5\) and \(\hat{\alpha}_B=3\). Its predictions are \(2.5,3.5,2.5,3.5,3\). Thus \(SSE_B=(-1.5)^2+(-1.5)^2+(0.5)^2+(0.5)^2+2^2=9\), giving \(R^2_B=1-9/10=0.10\). Under the specified model and metric, cutoff \(A\) performs better in this hypothetical validation set. The perfect fit for \(A\) is not evidence of perfect performance in future individuals: this example has only five observations and is used to show the calculation.
Why the Shared Metric Makes the Comparison Meaningful
For a quantitative outcome, let \(SST=\sum_i(Y_i-\bar{Y})^2\) be the total variation in the validation outcomes. If every candidate is evaluated on the same individuals, \(SST\) is fixed. Since \(R^2_t=1-SSE_t/SST\), maximizing \(R^2_t\) is then equivalent to minimizing \(SSE_t\). Dividing by the same sample size \(n\) shows that minimizing validation MSE, \(MSE_t=SSE_t/n\), gives the same ranking.
This equivalence is useful, but it does not make every metric interchangeable. For example, a model can have lower MSE but worse calibration at clinically important risk levels. For binary outcomes, AUC evaluates ranking and does not directly assess whether predicted probabilities are accurate. Log loss does assess probabilities and penalizes confident predictions that are wrong.
Worked Example: Selecting From a Cutoff Grid
Worked Example: Selecting From a Cutoff Grid
Suppose a hypothetical validation cohort has \(n=100\) individuals with the same quantitative outcome for each candidate. The analysis team has prespecified a linear regression with the same covariates and one score at each cutoff, and uses validation \(R^2\) as its selection metric. The outcome’s total sum of squares is \(200\). After fitting the four models, the recorded results are:
| Cutoff \(t\) | Variants in score | Validation \(SSE\) | Validation \(R^2=1-SSE/200\) |
|---|---|---|---|
| \(10^{-8}\) | 12 | 150 | 0.250 |
| \(10^{-5}\) | 48 | 140 | 0.300 |
| \(10^{-3}\) | 230 | 146 | 0.270 |
| 1 | 1,100 | 160 | 0.200 |
For \(t=10^{-5}\), the metric is \(1-140/200=0.300\). The other values are \(1-150/200=0.250\), \(1-146/200=0.270\), and \(1-160/200=0.200\). The largest validation \(R^2\) is \(0.300\), so \(10^{-5}\) is selected from this grid. In this cohort and model, the score explains 30.0% of the outcome variation relative to the total sum of squares; that is a description of the observed validation fit, not a guarantee of the same performance in a new cohort. The variant counts help characterize the selected score but do not determine the selection.
Worked Example: A Metric for a Binary Outcome
Worked Example: A Metric for a Binary Outcome
In a second hypothetical validation set, four individuals have binary outcomes \(Y=(1,1,0,0)\). A logistic regression of the same form, with the same covariates, is fitted for each of two candidate cutoffs. Suppose the resulting predicted probabilities of \(Y=1\) are \(q_A=(0.8,0.7,0.3,0.2)\) and \(q_B=(0.9,0.6,0.4,0.1)\). Use mean log loss, defined as \(-\frac{1}{n}\sum_i[Y_i\log(q_i)+(1-Y_i)\log(1-q_i)]\); smaller is better.
For score \(A\), the two cases contribute \(-\log(0.8)\) and \(-\log(0.7)\), while the two controls contribute \(-\log(0.7)\) and \(-\log(0.8)\). Thus the mean loss is \((0.2231+0.3567+0.3567+0.2231)/4=0.2899\).
For score \(B\), the contributions are \(-\log(0.9)\), \(-\log(0.6)\), \(-\log(0.6)\), and \(-\log(0.9)\). Its mean loss is \((0.1054+0.5108+0.5108+0.1054)/4=0.3081\).
Because lower log loss is better, cutoff \(A\) is preferred under this prespecified metric. Both sets of probabilities rank the two cases above the two controls, so their AUCs are both 1 in this tiny example. The metrics therefore distinguish probability accuracy from ranking: log loss favors \(A\), while AUC does not distinguish the scores.
Diagnostics, Alternatives, and Limitations
A cutoff comparison is only as credible as the process that produced it. Inspect the full metric-versus-cutoff pattern, not just the winner. A sharp isolated peak may reflect sampling noise, especially when the validation cohort is small or the grid contains many cutoffs. Report the number of retained variants, the direction and magnitude of score effects under the fixed model, and whether the results are sensitive to a few influential observations. Check that score construction uses consistent allele harmonization and the same clumping reference and parameters across candidates. Differences in ancestry, linkage disequilibrium, phenotype measurement, or covariate distributions between discovery and validation can affect performance; a cutoff comparison does not repair those mismatches.
The main alternative is to choose a cutoff using discovery significance alone, such as retaining only genome-wide significant variants. That rule is simple, but discovery p-values measure evidence against a null association, not how well a particular weighted score predicts the validation outcome. A very stringent threshold may omit many individually weak signals; a permissive threshold may add noisy effects. Validation performance directly compares these trade-offs for the chosen outcome and population. It does not establish that the best cutoff is universally optimal.
A crucial limitation is that selecting the best of many cutoffs makes the winning validation metric optimistic as an estimate of future performance. The same cohort has been used to choose the winner, so chance fluctuations can help the selected cutoff. If the goal is an unbiased estimate of performance after selection, evaluate the locked score in data that were not used to choose the cutoff or fit any outcome model. The selected validation score is useful for tuning; it should not be presented as an independent final test result.
Check Your Understanding
Answer each question, then check the brief solutions below.
- Two cutoffs are evaluated on the same quantitative outcomes, and their \(SSE\) values are 80 and 90. The common \(SST\) is 200. Which has the larger validation \(R^2\), and what are both values?
- Why should the covariates and outcome-model form be held fixed when comparing cutoffs?
- A cutoff has the best validation AUC but worse log loss than another cutoff. Is this necessarily a contradiction? Explain.
- What is the problem with reporting the winning validation metric as an unbiased estimate of future performance?
- A researcher chooses the cutoff with the most retained variants without evaluating validation outcomes. What does that rule fail to establish?
Solutions. (1) The first cutoff has \(R^2=1-80/200=0.60\); the second has \(R^2=1-90/200=0.55\), so the first is larger. (2) Holding them fixed makes the cutoff the intended changing factor; otherwise model or covariate changes could explain performance differences. (3) No. AUC assesses ranking, whereas log loss assesses the accuracy of predicted probabilities, so the metrics can rank models differently. (4) The same validation outcomes were used to choose the winner, making its observed performance optimistic as an estimate for new individuals. (5) Variant count alone does not establish better predictive performance under any specified validation metric.