Why Select Variants for a Joint Model?
A locus can contain several correlated candidate variants, while a joint regression model estimates their associations together. Including every candidate may make estimates imprecise or the model difficult to interpret; selecting too few may leave relevant conditional signals out. A selection rule provides a systematic way to choose a model, but the model it selects depends on that rule. It is therefore important to distinguish the model-building decision from the evidence subsequently reported for its variants.
The approaches here are tools for constructing locus models, not procedures for identifying causal variants. A variant retained in a joint association model is associated with the outcome conditional on the other included variables, under the specified model. That result alone does not establish a causal effect.
Model and Selection Setup
Suppose \(Y_i\) is a quantitative phenotype for individual \(i\), \(C_{ij}\) are prespecified covariates, and \(G_{im}\) is the genotype dosage for candidate variant \(m\). A common starting model is a linear regression with additive dosage effects:
Here \(S\) is the selected set of variants. The usual likelihood-based comparisons assume that the outcome model is correctly specified, residuals are independent and normally distributed with common variance, and the compared models use the same individuals and covariates. The genotype terms must be estimable: exact linear dependence among columns prevents separate coefficients from being identified. In practice, the design and residual assumptions should be examined; a selection criterion cannot repair a misspecified model.
Forward selection begins with the covariate-only model and considers adding candidate variants. Backward selection begins with a specified larger model and considers removing variants. At each step, the method uses a chosen rule—such as an information criterion or a conditional significance test—to decide whether the model should change. The outcome can depend on the starting model, the candidate set, the decision rule, and the order in which comparisons are made.
Information Criteria: AIC and BIC
Information criteria balance fit against model size. Let \(\ell\) denote the maximized log-likelihood and let \(k\) be the number of estimated parameters. Smaller values are preferred among the candidate models being compared:
For Gaussian regression, \(k\) ordinarily includes the intercept, all estimated covariate and genotype coefficients, and the residual variance parameter. This convention matters: counting only genotype coefficients would give incorrect criteria. If all models share the same covariates and variance parameter, adding one variant increases \(k\) by one. AIC and BIC are relative criteria; their absolute values do not have a direct interpretation as evidence that a model is true.
AIC uses a fixed penalty of 2 per additional parameter. BIC penalizes each additional parameter by \(\log(n)\), which exceeds 2 when \(n\) is greater than about 7.4. Thus BIC often favors smaller models than AIC, though the data can still favor a larger model if its improved fit is sufficient.
Worked Example: Comparing Candidate Models by AIC and BIC
Suppose a hypothetical analysis includes \(n=300\) individuals, an intercept, three prespecified covariate slopes, and a Gaussian residual variance. Candidate models contain none, one, two, or three variants. The following maximized likelihood summaries are hypothetical; the same individuals and covariates are used in every fit.
| Model | \(-2\ell\) | \(k\) |
|---|---|---|
| Covariates only | 1000.0 | 5 |
| Covariates plus \(G_1\) | 996.5 | 6 |
| Covariates plus \(G_1,G_2\) | 986.3 | 7 |
| Covariates plus \(G_1,G_2,G_3\) | 984.8 | 8 |
The parameter counts include the intercept, three covariate coefficients, the residual variance, and one coefficient per included variant. For \(n=300\), \(\log(300)\approx5.7038\). For the model with \(G_1\) and \(G_2\), for example:
Applying the same calculations to each row gives:
| Model | AIC | BIC |
|---|---|---|
| Covariates only | 1010.000 | 1028.519 |
| Covariates plus \(G_1\) | 1008.500 | 1030.723 |
| Covariates plus \(G_1,G_2\) | 1000.300 | 1026.226 |
| Covariates plus \(G_1,G_2,G_3\) | 1000.800 | 1030.430 |
Among these four models, AIC and BIC both have their smallest value for the model with \(G_1\) and \(G_2\). The model with \(G_1\) alone improves fit relative to the covariate-only model, but its BIC is larger because the fit improvement does not offset the BIC penalty. This comparison is among the listed models; it does not establish that the selected model is correct or that the variants have causal effects.
How the Criterion Changes the Stepwise Decision
The trade-off can be made explicit using the likelihood-ratio statistic. For nested models, let \(\ell_{\text{large}}\) and \(\ell_{\text{small}}\) be their maximized log-likelihoods, with the larger model adding \(q\) parameters. The statistic is \(2(\ell_{\text{large}}-\ell_{\text{small}})\). Under the null that the added parameters are zero, and under the regularity conditions for the likelihood-ratio test, it is approximately chi-square distributed with \(q\) degrees of freedom.
Adding the larger model decreases BIC precisely when its improvement in fit exceeds its additional penalty:
or, equivalently, when \(2(\ell_{\text{large}}-\ell_{\text{small}})>q\log(n)\). For AIC, the corresponding requirement is that the likelihood-ratio statistic exceed \(2q\). These are criterion-based decisions, not universal significance tests. In particular, BIC’s penalty depends on sample size, while a fixed significance threshold such as 0.05 uses a different rule.
For the hypothetical \(n=300\) comparison above, adding one parameter decreases BIC only if the likelihood-ratio statistic exceeds \(\log(300)\approx5.704\). For one degree of freedom, that corresponds to a chi-square tail probability of about 0.017 under the test’s null approximation. The AIC threshold is 2, corresponding to a much less stringent one-degree-of-freedom tail probability of about 0.157. These approximate p-value equivalents describe the threshold for that comparison; they do not turn AIC or BIC into hypothesis tests.
Worked Example: Forward and Backward Selection With a Significance Threshold
Consider a hypothetical locus with three candidate variants and the same prespecified covariates throughout. Use a conditional likelihood-ratio test for each variant and a rule to retain or add a variant when its conditional p-value is below 0.05. Suppose the tests at the covariate-only starting model give \(G_1:p=0.012\), \(G_2:p=0.031\), and \(G_3:p=0.44\).
Forward path. The smallest p-value is for \(G_1\), so add \(G_1\). In the model now containing \(G_1\), suppose the conditional results for the remaining candidates are \(G_2:p=0.018\) and \(G_3:p=0.27\). Add \(G_2\). In the model with \(G_1\) and \(G_2\), suppose the conditional test for \(G_3\) gives \(p=0.12\). Since \(0.12\) is not below 0.05, stop. The resulting model contains \(G_1\) and \(G_2\).
Backward path. Start instead with all three variants. Suppose the conditional test for removing \(G_3\), given \(G_1\) and \(G_2\), gives \(p=0.12\). Under the stated removal rule, delete \(G_3\). Suppose that in the reduced model the conditional p-values for \(G_1\) and \(G_2\) are 0.009 and 0.018. Both are below 0.05, so neither is removed. This backward path also ends with \(G_1\) and \(G_2\).
The decisions follow the stipulated conditional tests at each step. The hypothetical results do not imply that forward and backward selection must always agree: their starting models and tested comparisons differ, and another locus can yield different paths or endpoints.
What Selection Does to Reported Evidence
A p-value from a test conducted after selection is not automatically calibrated as if that variant and model had been specified in advance. Selection tends to retain variants with unusually strong observed associations, including those whose apparent evidence is partly due to random variation. Likewise, effect estimates for selected variants can be exaggerated in magnitude on average, especially when many candidates are screened and only the strongest are reported. These are consequences of choosing based on the same data used for estimation and testing.
A simple calculation shows why repeated screening matters. Suppose, only for illustration, that 20 independent tests are performed when all their null hypotheses are true, and each test is retained if its p-value is below 0.05. Each test has probability 0.95 of not crossing the threshold. Under independence, the probability that none crosses is \(0.95^{20}\). Therefore:
Thus the chance of at least one nominally significant result is about 64.15% in this illustrative setting, rather than 5%. Variants in a locus are often correlated through linkage disequilibrium, so the independence calculation is not an exact description of a real variant set. It nevertheless shows why a nominal p-value selected from many candidates should not be interpreted as though there had been just one prespecified test.
Worked Example: Interpreting a Selected Association
Suppose a hypothetical forward-selection analysis considers 40 variants and retains \(G_7\), whose final conditional test has \(p=0.008\). The correct immediate description is that, in the selected joint model and conditional on its included covariates and variants, the test statistic is at least as incompatible with its null as observed with a tail probability of 0.008 under the test’s null model.
That statement does not mean there is a 0.8% probability that the null hypothesis is true. Nor does it guarantee that the p-value would remain 0.008 if the analysis accounted for searching across 40 candidates and choosing a model based on these same data. The selection process should be disclosed, and the strength of evidence should be interpreted in light of the candidate search and the dependence among variants. A selected coefficient is an estimated conditional association in this model, not proof that \(G_7\) causes the phenotype.
Choosing and Reporting a Strategy
No selection rule is best for every objective. AIC and BIC compare fit and complexity according to different penalties. A significance threshold gives a sequence of hypothesis-test decisions whose outcome depends on the threshold and the path. The number of candidate variants, their correlation, sample size, and whether the goal is prediction, compact description, or conditional association all affect which rule is useful. These goals should be stated rather than treated as interchangeable.
When reporting a selected locus model, state the candidate set, the prespecified covariates, the selection direction and rule, the threshold or criterion, and whether the reported tests and estimates are conditional on the selected model. Report the final joint estimates with their units and uncertainty, and avoid presenting the selected variants as a complete or uniquely determined set of biological signals. If a model-selection procedure was data-driven, that fact belongs alongside the reported evidence.
Common Mistakes
- Treating the smallest criterion as proof. AIC or BIC ranks the models compared under its penalty; it does not show that the winning model is true.
- Calling an AIC or BIC decision a significance test. The criterion has a penalty rule, not a null-hypothesis interpretation. Its implied likelihood-ratio threshold is not generally the chosen significance level.
- Interpreting a post-selection p-value as prespecified. A p-value from the selected model does not by itself account for searching across candidates and choosing that model using the same data.
- Assuming stepwise paths must agree. Forward and backward procedures start differently and make different conditional comparisons; they can select different sets.
- Equating a retained variant with a causal variant. Selection establishes neither causality nor a uniquely identified biological signal.
Check Your Understanding
Use the model definitions and selection rules above to answer these questions.
- In a Gaussian locus regression with an intercept, two covariate slopes, three variant coefficients, and a residual variance, what is \(k\) for the fitted model?
- For a nested model that adds two variant coefficients, write the condition under which BIC decreases after adding them.
- Why can a forward path and a backward path end with different variant sets even when they use the same significance threshold?
- In the independent-test illustration, why is the probability of at least one p-value below 0.05 greater than 0.05 when 20 tests are performed?
- What does a selected variant’s conditional association establish, and what does it not establish?