Covariate Selection Without Data-Driven Overfitting

Why Covariate Selection Needs a Plan

A genetic association estimate depends on the model used to obtain it. Adding or removing covariates can change the estimated genotype effect, its standard error, and even the question the model answers. A convenient-looking rule, such as retaining only covariates whose outcome-association p-values are below 0.05, does not by itself identify a sound adjustment set.

The goal is not to include every available variable or to maximize the number of small p-values. It is to define the association being estimated, decide which covariates are justified by the study design and scientific rationale, and then use model comparisons for a clearly stated purpose. This is especially important in genome-wide association analyses, where a model-selection rule applied repeatedly can create systematic inferential problems.

Prespecification does not mean that one universal set is correct for every study. Age, sex, study site, and ancestry-related variables may be justified by the design, the target population, or the association question. The decision should be made in light of the estimand and plausible sources of bias, rather than by a threshold applied mechanically to each covariate’s p-value. As established in the earlier material on covariate roles, adjustment can also change the interpretation of an effect; adding a variable is not automatically beneficial.

Write Down the Model Before Comparing It

For a binary phenotype, a common single-variant model uses logistic regression. Let \(Y_i\) indicate case status for person \(i\), let \(G_i\) be genotype dosage coded as 0, 1, or 2 copies of the effect allele, and let \(Z_i\) denote the chosen covariates. A main-effects model can be written as:

Here, \(\beta_G\) is the log odds ratio per additional effect-allele copy, conditional on the covariates in that model; \(\exp(\beta_G)\) is the corresponding conditional odds ratio. The model assumes the observations are independent for the likelihood used, the log odds are correctly represented by the included terms, and the parameters can be estimated without complete or quasi-complete separation. The usual Wald inference additionally relies on an adequate large-sample approximation. If genotype effects vary across covariate levels, a model without interaction terms may not describe that variation.

These assumptions do not dictate a covariate set. They clarify what a fitted model means and what conditions support its inference. When candidate models are compared, the sample and outcome definition should be held fixed. In particular, a change in the set of complete cases can make a fit comparison reflect both different covariates and different data.

Nested-Model Tests: A Targeted Comparison

A nested-model test compares a restricted model with a larger model that contains all the restricted model’s terms plus additional parameters. For logistic regression, the likelihood-ratio test uses the maximized log likelihoods from the two models, fitted to the same observations. If the restricted model has log likelihood \(\ell_R\), the larger model has log likelihood \(\ell_F\), and the larger model adds \(q\) parameters, the test statistic is:

Under the null hypothesis that all \(q\) added coefficients are zero, and under the regularity conditions for the likelihood-ratio approximation, \(G^2\) is approximately distributed as a chi-square random variable with \(q\) degrees of freedom. The comparison is useful when the added terms form a prespecified block—for example, testing whether a categorical variable’s set of indicator coefficients is jointly zero.

A likelihood-ratio test is not a general-purpose instruction to keep every covariate with a significant p-value. If many candidate blocks are tested and only “significant” ones retained, the final model has been selected using the data. The ordinary p-values from that selected model do not account for the selection process. Moreover, a covariate’s individual test of association with the phenotype is not the same question as whether controlling for it is important for the genetic association estimate.

Information Criteria: Balancing Fit and Complexity

Information criteria summarize a trade-off between model fit and the number of estimated parameters. For a fitted maximum-likelihood model with maximized log likelihood \(\ell\), parameter count \(k\), and sample size \(n\), the Akaike information criterion and Bayesian information criterion are:

The parameter count includes all estimated parameters in the likelihood, including the intercept and genotype coefficient. Among the candidate models being compared, a smaller AIC or BIC is preferred by that criterion. AIC is motivated by estimating relative predictive information loss; BIC applies a larger complexity penalty as sample size grows and, under regularity conditions when the true model is among the candidates, can favor selection of that model. Neither criterion is a test of whether a covariate is a confounder, and neither guarantees adequate control of population structure or other bias.

How P-Value Screening Can Change an Effect Estimate

A covariate can matter for a genotype association even if its own phenotype-association p-value does not pass a chosen threshold. In a simple linear model, this can be seen directly. Suppose a quantitative phenotype satisfies \(Y=\beta_G G+\gamma C+\varepsilon\), with an intercept understood, and the error has mean zero conditional on \(G\) and \(C\). If \(C\) is omitted, the population slope from regressing \(Y\) on \(G\) alone is:

This expression follows from the one-predictor regression slope, \(\operatorname{Cov}(G,Y)/\operatorname{Var}(G)\). Substituting the model for \(Y\) gives \(\operatorname{Cov}(G,Y)=\beta_G\operatorname{Var}(G)+\gamma\operatorname{Cov}(G,C)\), since the conditional-mean-zero error contributes zero covariance. Dividing by \(\operatorname{Var}(G)\) yields the result. The discrepancy is zero if \(\gamma=0\) or if \(G\) and \(C\) are uncorrelated; otherwise, omission changes the slope. This linear derivation illustrates a general concern, not an exact omitted-variable formula for logistic regression.

There is a second issue: selection itself uses random data. If analysts try several covariate sets and report the one that gives the preferred genotype estimate or the smallest genotype p-value, the reported estimate and its ordinary standard error are conditional on an unreported search. Conventional confidence intervals and p-values are calculated as if the reported model had been fixed in advance; they generally do not account for this selection. The resulting uncertainty can be understated, and effect estimates can be exaggerated by choosing a model that happens to produce a large association in that sample.

This does not mean that every use of model comparison is invalid. A prespecified likelihood-ratio test answers a defined nested-model question. AIC or BIC can provide a transparent comparison for a stated candidate set and objective. The concern is treating a data-dependent selection step as though it never happened, or allowing a fit criterion to replace knowledge of the study design and the target association.

A Practical Covariate-Selection Workflow

Common Mistake

• Wrong reasoning: “This covariate has p=0.12, so it is unrelated to the phenotype and should be removed; the resulting genotype p-value is the valid primary result.”
• Correction: A covariate p-value does not determine whether adjustment is appropriate. A covariate can be imprecisely associated with the phenotype yet alter the genotype estimate, and selecting variables using the same data can make standard inference ignore the selection step. Base the primary set on the question and design; use comparisons as supporting, explicitly qualified evidence.