When Sparse Models Miss Many Weak Predictors

Why a Sparse Fit Can Miss a Useful Signal

A sparse model is attractive when genomic measurements greatly outnumber samples: it can produce a short list of predictors that is easier to inspect than thousands of coefficients. But sparsity is a modeling preference, not evidence that the underlying biology is sparse. If many features each contribute a small amount, a procedure that removes features one at a time may discard much of the signal even when the features are jointly informative.

The relevant distinction is between feature recovery and prediction. Feature recovery asks whether the selected variables correspond to the nonzero population coefficients. Prediction asks how close the fitted values are to outcomes in new samples. A selected set can be small and interpretable yet omit true weak effects; conversely, a model can predict well while selecting only one of several highly correlated substitutes. These are different goals, so a sparse selection alone cannot establish that a signal is truly sparse.

To make the mechanism precise, consider a centered linear model \(Y=X\beta+\varepsilon\). Here \(Y\) is the outcome vector, \(X\) is the design matrix, \(\beta\) is the coefficient vector, and \(\varepsilon\) is a mean-zero error vector. For the prediction-risk calculations below, assume a new observation satisfies \(Y_{\mathrm{new}}=X_{\mathrm{new}}^\mathsf{T}\beta+\varepsilon_{\mathrm{new}}\), with independent mean-zero error of variance \(\sigma^2\), and is independent of the training data. Predictors are centered and scaled as stated in each example. These assumptions describe a benchmark, not a guarantee that a genomic data set follows a linear model.

Why Lasso Thresholding Omits Weak Coefficients

The lasso minimizes a squared-error loss plus an \(L_1\) penalty. In a simple orthogonal benchmark, suppose the predictors are centered and scaled so \(X^\mathsf{T}X/n=I\). Let \(z_j=X_j^\mathsf{T}Y/n\) be the unpenalized least-squares coefficient for feature \(j\). With no intercept in the displayed objective, the lasso minimizes

Under orthogonality, the part depending on coordinate \(b_j\), up to an additive constant, is \(\frac{1}{2}(b_j-z_j)^2+\lambda_1|b_j|\). Thus each coefficient can be minimized separately. If \(z_j>\lambda_1\), the derivative for a positive coefficient is \(b_j-z_j+\lambda_1\), which is zero at \(b_j=z_j-\lambda_1\). If \(z_j<-\lambda_1\), the corresponding negative solution is \(b_j=z_j+\lambda_1\). For \(|z_j|\leq\lambda_1\), the minimum is at zero: the one-sided slopes there do not favor moving away from zero. Combining the cases gives the soft-thresholding rule.

This is the key mechanism: a coefficient is set exactly to zero if its observed score does not exceed the penalty threshold. Many small scores can therefore produce many omissions, even if the corresponding true effects are all nonzero. The orthogonal formula isolates the thresholding behavior; when \(p\) exceeds \(n\), predictors cannot all be mutually orthogonal in the sample. Correlation and rank deficiency alter the joint solution, so the formula is a teaching benchmark rather than a universal coordinate-by-coordinate description.

Comparing Prediction Under a Dense Signal

For ridge regression, the \(L_1\) penalty is replaced with a squared-\(L_2\) penalty; elastic net uses both. In the orthogonal benchmark, if ridge minimizes the squared-error loss plus \(\lambda_2\sum_j b_j^2/2\), its coordinate solution is \(z_j/(1+\lambda_2)\). With an elastic-net penalty \(\lambda_1\sum_j|b_j|+\lambda_2\sum_j b_j^2/2\), the solution is the soft-thresholded value divided by \(1+\lambda_2\). Ridge shrinks coefficients but does not threshold them to zero; elastic net can do both.

For a fixed fitted coefficient vector \(\hat\beta\), if new predictors have mean zero and covariance matrix \(\Sigma\), the expected squared prediction error for a new outcome is

The first term is irreducible outcome noise. The second is the prediction cost of coefficient error, weighted by the covariance of new predictors. If predictors are uncorrelated with variance one, \(\Sigma=I\), and that cost is the sum of squared coefficient errors. With correlated predictors, the cost depends on their joint covariance, not just on how many coefficients were omitted.

Correlation Can Change the Cost of Omission

In genomic data, correlated features are common. A lasso fit may retain one feature and omit another because one can partly stand in for the other. That can destabilize which feature is named without greatly changing predictions. In contrast, if many weak predictors contribute distinct, approximately independent components of the outcome, omitting all of them can accumulate substantial prediction error. The covariance-weighted error formula distinguishes these situations.

Diagnostics, Alternatives, and Limitations

A useful diagnostic is to compare predictive performance across penalty methods using validation data that were not used to fit the coefficients. Cross-validation for selecting penalty strength is useful for estimating out-of-sample prediction, but a small difference between methods may be within its uncertainty. Inspect the full prediction-error curve, not just the chosen penalty, and remember that the one-standard-error rule deliberately favors a simpler fit when its error is sufficiently close to the minimum. Selection stability is another, distinct question: repeated subsampling can reveal whether the same features recur, but stable selection does not prove that the model captures every weak contributor.

For a dense-signal concern, examine whether conclusions change when the penalty permits less sparsity, and compare lasso with ridge or elastic net on the same validation splits and prediction scale. Ridge is a natural alternative when many predictors may contribute small effects; elastic net allows shrinkage and some selection while often grouping correlated predictors more readily than lasso. None is automatically superior. Strong correlations, nonlinear relationships, omitted covariates, batch effects, and distribution shifts can invalidate the simple linear benchmark or make validation performance misleading.

• Do not infer a zero population effect from a zero lasso coefficient. A zero estimate may result from thresholding and finite-sample noise.
• Do not count omissions as a direct measure of prediction failure. Correlated features may substitute for one another; evaluate prediction using the relevant covariance and held-out outcomes.
• Do not compare methods at arbitrary, unequal tuning quality and call the result universal. Tune within training data and assess the comparison on held-out data.
• Do not treat a better prediction score as proof of correct gene identification. Prediction accuracy does not identify causal effects.

The main limitation of the worked calculations is their deliberately simple covariance and coefficient structure. In \(p\gg n\) data, the sample design is rank deficient, predictors may be correlated, and scores are noisy. The equations explain why sparse models can miss dense weak signals, but they do not predict how much a particular genomic analysis will lose. That requires data-appropriate validation and careful interpretation of selection uncertainty.