Sparse and Dense Signal Patterns

Signal Pattern and Penalty Choice

A predictive signal may be sparse: a small number of features carry most of the useful information, while many others contribute little. Or it may be distributed: predictive information is spread across a larger set of features, perhaps because several related variables each contain part of the signal. These descriptions concern how the signal is represented by the available predictors; they do not establish which variables cause an outcome.

Lasso and elastic net can produce different coefficient patterns even when applied to the same data. Lasso uses an absolute-value penalty that can set coefficients exactly to zero. Elastic net combines that penalty with a squared-coefficient penalty. The absolute-value component still permits zero coefficients, while the squared component can make sharing signal across related features less costly than selecting just one. Neither method guarantees recovery of the underlying pattern. The result also depends on the features, their scaling, and the penalty settings.

Objective and Orthogonal-Predictor Formula

Assume the response and predictors have been centered, the intercept is handled separately and is not penalized, and the columns of the design matrix \(X\) are standardized. For a squared-error model, write the elastic net objective as

Here \(\beta\) is the vector of coefficients, \(n\) is the number of observations, and \(\|\beta\|_1=\sum_j|\beta_j|\), while \(\|\beta\|_2^2=\sum_j\beta_j^2\). At \(\alpha=1\), the objective is lasso; at \(\alpha=0\), it is ridge; intermediate values combine both penalties. This tutorial uses the stated objective and scaling throughout. Changing the loss normalization or penalty convention changes the numerical interpretation of \(\lambda\).

A useful exact comparison is available when predictors are orthogonal under this scaling, so that \(X^\mathsf{T}X/n=I\). Define \(z_j=x_j^\mathsf{T}y/n\), where \(x_j\) is column \(j\) of \(X\). With orthogonal predictors, each coefficient can be minimized separately:

For lasso, \(\alpha=1\), so the threshold is \(\lambda\) and the denominator is 1. For elastic net, the threshold is \(\lambda\alpha\); when \(\alpha<1\), the denominator exceeds 1 if \(\lambda>0\), adding shrinkage to coefficients that survive the threshold. This formula relies on the orthogonality and standardization assumptions above. With correlated predictors, the coordinates do not minimize independently, so this coordinate formula cannot be applied as if the predictors were orthogonal.

Why the Coordinate Formula Holds

When \(X^\mathsf{T}X/n=I\), the squared-error part of the objective, up to a term that does not depend on \(\beta\), is \(\frac12\sum_j(\beta_j-z_j)^2\). Thus the part involving one coefficient is

If \(z_j>0\) and a minimizing coefficient is positive, differentiating on the positive side gives

This positive solution is valid only when \(z_j>\lambda\alpha\). If \(z_j\leq\lambda\alpha\), the minimum is at zero: the slope from the right at zero is \(-z_j+\lambda\alpha\), which is nonnegative, so moving to a positive coefficient cannot reduce the objective. Applying the same argument to a negative coefficient gives the corresponding result for \(z_j<0\), yielding the signed threshold formula. The denominator follows from the derivative of the quadratic penalty; it is not part of the threshold.

More Distributed Patterns Under Orthogonality

If several orthogonal predictors have moderate associations with the response, lasso may keep only the ones whose \(|z_j|\) exceed \(\lambda\). Elastic net at the same \(\lambda\) and an intermediate \(\alpha\) has a lower threshold, \(\lambda\alpha\), but also shrinks the retained coefficients through its denominator. The number of nonzero coefficients can therefore differ, while the sizes of retained coefficients differ as well. A lower threshold does not mean that every elastic net coefficient must be larger: its extra quadratic shrinkage acts in the opposite direction.

Correlated Features and Representation

The orthogonal examples clarify thresholding, but related features are often correlated. When two or more features carry similar information, lasso can favor a representation that uses just one of them. Which feature is selected can depend on the data and the details of the design. Elastic net's squared-coefficient penalty can make a shared representation more attractive, particularly when predictors are very similar. This is a tendency, not a rule that guarantees equal coefficients or selection of every related feature.

The effect can be seen exactly for duplicate predictors. Suppose \(p\) standardized columns are identical, their common fitted contribution is determined by the coefficient sum \(s=\sum_{j=1}^p\beta_j\), and the response is aligned so that the squared-error term, up to a constant, is \(\frac12(c-s)^2\), with \(c>0\). Consider nonnegative coefficients. For a fixed sum \(s\), the lasso penalty depends only on \(s\), while the elastic net quadratic penalty depends on how that sum is allocated.

Interpreting the Trade-Off

The examples show why there is no universal rule that one method is better for “sparse” or “dense” data. When a few features carry most of the signal and a compact representation is desirable, lasso's exact zeros can be useful. When many features contribute or related predictors offer overlapping information, elastic net may represent the signal across more coefficients. But a denser fitted coefficient vector is not proof that the underlying signal is truly distributed, and a sparse vector is not proof that only those selected features matter.

Coefficient patterns also depend on how predictors are represented. Scaling affects the penalty applied to a coefficient, which is why comparisons require an appropriate common scaling, as in the examples. Categorical encodings, interactions, and groups of related variables can change what “one feature” means. A penalty acts on the supplied columns; it does not automatically recognize a real-world construct as a single indivisible variable.

At fixed \(\lambda\), \(\alpha\), and design, the two methods need not have the same number of nonzero coefficients or the same total fitted contribution. Nor should a comparison be reduced to counting selected predictors: the coefficient magnitudes and the way correlated features share a representation matter. The useful practice is to state the assumed signal pattern, inspect the feature structure, and explain the representation each penalty produces without interpreting selection as causation.