Prediction and Variable Selection as Different Goals

Two goals that answer different questions

A genomic prediction model can estimate a phenotype accurately without identifying which genes truly contribute to its conditional mean. Conversely, a procedure can return a plausible set of active genes without producing the smallest prediction error on new samples. These are different statistical goals: one concerns the performance of predictions, and the other concerns whether the selected variables match a specified underlying set.

This distinction matters in a \(p\)-much-larger-than-\(n\) setting, where many coefficient vectors may fit the observed data similarly, and correlated genes can make their individual contributions difficult to separate. Penalization controls fitted coefficients, but the resulting fit should be judged against the goal of the analysis rather than treated as an automatic answer to both questions.

For clarity, assume predictors are centered or that an intercept is included and handled separately. Assume the linear conditional-mean model is appropriate, and write \(Y=X^\mathsf{T}\beta^*+\varepsilon\), with \(E(\varepsilon\mid X)=0\) and \(E(\varepsilon^2\mid X)=\sigma^2\). A new observation \((X_{\mathrm{new}},Y_{\mathrm{new}})\) is drawn from the target population, independently of the training data. These assumptions let us separate error in predicting the conditional mean from irreducible outcome noise. The central distinction remains useful when a linear model is an approximation, but then the prediction target and the interpretation of \(\beta^*\) must be stated carefully.

Prediction risk and its connection to coefficient error

The population predictive risk of a fitted vector, conditional on the training data, is the expected squared error on a new observation. Its excess prediction risk is the amount above the irreducible noise variance. With \(\Sigma=E(X_{\mathrm{new}}X_{\mathrm{new}}^\mathsf{T})\), the following identity holds under the model assumptions just stated.

To justify it, let \(d=\hat{\beta}-\beta^*\). The prediction error is \(\varepsilon_{\mathrm{new}}-X_{\mathrm{new}}^\mathsf{T}d\). Squaring and taking the expectation gives the noise term, the quadratic coefficient-error term, and a cross term. The cross term is zero because \(E(\varepsilon_{\mathrm{new}}\mid X_{\mathrm{new}})=0\); the quadratic term is \(d^\mathsf{T}\Sigma d\). Thus prediction risk measures coefficient error in directions represented by the predictor distribution. If \(\Sigma\) has small eigenvalues, large coefficient errors in its low-variance directions can have little impact on predictions.

By contrast, support recovery asks whether every zero and nonzero coordinate has been classified correctly. It does not weight errors by how often a predictor varies in the population or how strongly it predicts the outcome. A tiny nonzero estimate for a truly inactive gene counts as a false positive under exact support recovery, even if it changes predictions negligibly.

Why correlation complicates selection

If two predictors are strongly correlated, they can carry nearly the same predictive information. A selected model may use one where the underlying data-generating description assigns a nonzero coefficient to the other. This can preserve predictions while changing the selected set. When predictors are exactly or nearly redundant, the data may provide little information for deciding which variable should receive the coefficient.

An important implication is that exact recovery needs more than good prediction. Under a positive-definite \(\Sigma\), excess risk is at least \(\lambda_{\min}(\Sigma)\|\hat{\beta}-\beta^*\|_2^2\), where \(\lambda_{\min}(\Sigma)\) is its smallest eigenvalue. When predictors are highly correlated, this eigenvalue can be small, so a small excess risk need not imply small coefficient error. Even small coefficient error is not by itself enough to guarantee the exact support of an unthresholded estimate: a truly zero coefficient can have a very small nonzero estimate.

Support recovery is more plausible when nonzero coefficients are sufficiently separated from zero, the design contains enough information to distinguish predictors, and the fitting and selection procedure meets conditions appropriate to the signal and design. A common signal-strength idea is the minimum nonzero magnitude, \(\beta_{\min}=\min_{j\in S^*}|\beta_j^*|\). If a procedure controls coordinatewise estimation error below \(\beta_{\min}/2\), thresholding estimates at \(\beta_{\min}/2\) separates active from inactive coordinates, provided the error bound applies to every coordinate. This is a sufficient reasoning pattern, not a guarantee that a particular penalized fit will satisfy the bound.

Choosing a goal and diagnosing a mismatch

The aim of an analysis determines which evidence is relevant. If the primary aim is prediction for new samples from a defined population, assess predictive error on observations not used to fit the model, using a loss appropriate to the outcome. If the aim is exact support recovery, predictive error alone is insufficient: a procedure can predict well with false positives, false negatives, or substitutions among correlated genes. In practice, the true support is usually unknown, so exact recovery cannot be verified merely by inspecting a fitted model.

The following comparison summarizes the different roles of common penalized fits. It is not a ranking: which approach is useful depends on the scientific objective, signal structure, and design.

Useful diagnostics depend on the goal. For prediction, check performance on held-out data representative of the intended use, compare it with a relevant baseline, and examine whether results are sensitive to distribution shifts. For selection, report the selection rule and threshold, inspect whether genes are strongly correlated or nearly redundant, and assess whether scientific conclusions change under plausible modeling choices. A sparse output is not itself evidence of exact recovery, and a non-sparse output is not automatically a prediction failure.

Common warning signs include treating training error as evidence of generalization, interpreting a lasso-selected gene as uniquely identified despite strong correlation, or claiming that an unselected weak gene has no effect. Penalized estimates are affected by shrinkage and the chosen penalty; the selected set is a model output, not proof of a biological mechanism. Prediction and selection also do not, by themselves, establish a causal effect.