How Lasso Performs Variable Selection

What Lasso Selection Means

Variable selection is one reason to use the lasso: unlike ordinary least squares, it can fit a model in which some predictor coefficients are exactly zero. This can produce a more compact description of the fitted relationship. But selection is a property of the fitted model under a particular penalty and data representation; a zero estimate does not establish that the predictor is irrelevant in every population or model.

The results below use the squared-error lasso objective. For its usual regression interpretation, assume observations follow a linear model with fixed predictors and errors having mean zero and constant variance; independent errors are also commonly assumed. Normal errors are not required to compute the lasso estimate. The lasso minimizes its stated objective whether or not the errors are normally distributed.

Definition: For responses $y_i$, predictors $x_{ij}$, and an intercept $\beta_0$, the lasso estimate minimizes $$ \frac{1}{2n}\sum_{i=1}^{n}\left(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2 +\lambda\sum_{j=1}^{p}|\beta_j|, \qquad \lambda\geq 0. $$ The intercept is not penalized. A predictor is selected by this fitted model when its estimated coefficient $\hat{\beta}_j$ is nonzero. The penalty parameter $\lambda$ controls the tradeoff between squared error and the total absolute size of the coefficients.

The factor $1/(2n)$ is a convention that makes the threshold formulas below take a convenient form. Changing the scaling of the squared-error term changes the numerical value corresponding to a given penalty. Predictor units also matter: a predictor measured in dollars and the same predictor measured in thousands of dollars have different coefficient scales. Standardizing predictors before penalization is therefore common when their units or ranges differ. The interpretation of selection must state which scaling was used.

Why the Absolute Penalty Can Produce Zeros

The mechanism is especially clear when predictors are centered, each has mean square $1$, and distinct predictor columns are orthogonal, so $x_j^\mathsf{T}x_k/n=0$ for $j\ne k$. Suppose the response is centered as well. Define $z_j=x_j^\mathsf{T}y/n$, the ordinary least-squares coefficient for predictor $j$ in this orthogonal setup. Apart from a term that does not depend on the coefficients, the objective separates into one-dimensional pieces:

\frac{1}{2}(\beta_j-z_j)^2+\lambda|\beta_j|.

For a positive coefficient, differentiating gives $\beta_j-z_j+\lambda=0$, so a positive minimizer is $z_j-\lambda$, provided $z_j>\lambda$. For a negative coefficient, the derivative is $\beta_j-z_j-\lambda=0$, so a negative minimizer is $z_j+\lambda$, provided $z_j<-\lambda$. At zero, the absolute-value penalty has subgradients from $[-1,1]$; zero minimizes the piece precisely when $|z_j|\leq\lambda$. Thus:

\hat{\beta}_j= \begin{cases} z_j-\lambda, & z_j>\lambda,\\ 0, & |z_j|\leq\lambda,\\ z_j+\lambda, & z_j<-\lambda. \end{cases} \qquad \hat{\beta}_j=\operatorname{sign}(z_j)(|z_j|-\lambda)_+.

Here $(a)_+=\max(a,0)$. The penalty does not merely shrink a small coefficient toward zero: once its unpenalized coefficient is within the threshold, the optimum is exactly zero. This is the soft-thresholding rule. It applies directly in the stated orthogonal, standardized setup; with general predictors, coefficients cannot usually be obtained by thresholding each ordinary least-squares coefficient separately.

Optimality check: Let $r=y-\hat{\beta}_0\mathbf{1}-\sum_{k=1}^{p}x_k\hat{\beta}_k$ be the fitted residual vector. For the objective above, a nonzero coefficient must satisfy $x_j^\mathsf{T}r/n=\lambda\operatorname{sign}(\hat{\beta}_j)$. A zero coefficient must satisfy $|x_j^\mathsf{T}r/n|\leq\lambda$. These conditions check whether the fitted coefficients minimize the objective. In the orthogonal setup, they reduce to the soft-thresholding rule.

Worked Selection Calculations

Worked Example: Selecting Predictors in an Orthogonal Model

A hypothetical insurer models a centered continuous response using four centered, standardized, mutually orthogonal predictors. The squared-error lasso objective uses $\lambda=0.8$, and the ordinary least-squares coefficients are $z=(1.5,\ 0.6,\ -1.1,\ 0.2)$. Find the lasso coefficients, identify the selected predictors, and check the result.

Method and assumptions. The predictor columns satisfy $x_j^\mathsf{T}x_j/n=1$ and $x_j^\mathsf{T}x_k/n=0$ for distinct $j,k$. The intercept is fitted separately and is not penalized. Apply soft thresholding to each $z_j$.

For predictor 1, $1.5>0.8$, so $\hat{\beta}_1=1.5-0.8=0.7$. For predictor 2, $|0.6|\leq0.8$, so $\hat{\beta}_2=0$. For predictor 3, $z_3=-1.1<-0.8$, so $\hat{\beta}_3=-1.1+0.8=-0.3$. For predictor 4, $|0.2|\leq0.8$, so $\hat{\beta}_4=0$.

The fitted coefficient vector is $(0.7,\ 0,\ -0.3,\ 0)$, so predictors 1 and 3 are selected. To check the optimality conditions, the orthogonal residual correlation for an active predictor is $z_j-\hat{\beta}_j$. These values are $1.5-0.7=0.8$ and $-1.1-(-0.3)=-0.8$, equal to $\lambda$ times the corresponding coefficient signs. For the zero coefficients, the residual correlations are $0.6$ and $0.2$, whose absolute values are both at most $0.8$. The conditions hold. The common mistake avoided here is to treat every nonzero ordinary least-squares coefficient as selected.

Worked Example: A Zero Estimate Is Not Proof of Irrelevance

In a second hypothetical orthogonal, standardized model, three ordinary least-squares coefficients are $z=(0.45,\ 0.52,\ -1.40)$, and the selected penalty is $\lambda=0.50$. Determine the lasso estimates and examine what the zero estimate for the first predictor does and does not establish.

For the first predictor, $|0.45|<0.50$, giving $\hat{\beta}_1=0$. For the second, $0.52>0.50$, giving $\hat{\beta}_2=0.52-0.50=0.02$. For the third, $-1.40<-0.50$, giving $\hat{\beta}_3=-1.40+0.50=-0.90$. Thus the fitted vector is $(0,\ 0.02,\ -0.90)$; predictors 2 and 3 are selected at this penalty.

The first predictor’s ordinary least-squares coefficient is $0.45$, not zero. Its lasso coefficient is zero because the penalty threshold is $0.50$. The coordinate objective at zero is $\tfrac12(0-0.45)^2+0.50|0|=0.10125$. At its unpenalized value, it is $\tfrac12(0.45-0.45)^2+0.50|0.45|=0.225$. The penalized objective therefore prefers zero in this coordinate. This calculation explains the fitted decision; it is not a test that the predictor has no association or predictive value in the population.

The common mistake avoided is to report “predictor 1 is irrelevant” based only on its zero lasso estimate. A precise conclusion is that it was not selected by this fit, using this data, scaling, model, and penalty. Selection is not a p-value, a causal conclusion, or a universal property of the predictor.

Worked Example: Checking a Selected Model and Making a Prediction

A hypothetical pricing analysis uses three centered, standardized orthogonal predictors and an unpenalized intercept of $10$ response units. The ordinary least-squares coefficients are $z=(1.6,\ 0.7,\ -1.2)$, and $\lambda=0.6$. Find the selected coefficients, predict for a profile with standardized predictor values $(1,\ 2,\ -1)$, and check the fit conditions.

Apply the threshold one predictor at a time. The estimates are $1.6-0.6=1.0$, $0.7-0.6=0.1$, and $-1.2+0.6=-0.6$. All three are nonzero, so all three predictors are selected in this fit.

The prediction is the intercept plus the coefficient-weighted predictor values: $10+(1.0)(1)+(0.1)(2)+(-0.6)(-1)=10+1.0+0.2+0.6=11.8$ response units.

For the optimality check, the three residual correlations are $1.6-1.0=0.6$, $0.7-0.1=0.6$, and $-1.2-(-0.6)=-0.6$. Each equals $0.6$ times the sign of its estimated coefficient. The active-coefficient conditions hold, and there are no zero coefficients to check. The common mistake avoided is to use unstandardized profile values with coefficients fitted to standardized predictors; the stated profile values are already on the standardized scale.

Interpreting Selection in Practice

The zero-versus-nonzero rule is exact for a particular fitted lasso model, but it does not by itself measure the importance of a predictor. A nonzero coefficient may be small, and a zero coefficient may reflect the penalty’s tradeoff rather than an absence of signal. The value of $\lambda$, the predictor scaling, the response, and the other predictors in the model all affect the fitted result. A selected coefficient should be interpreted in the units and model used to fit it.

For general, nonorthogonal predictors, the residual-correlation conditions provide the relevant check. An inactive predictor has zero coefficient when its residual correlation has absolute value no greater than $\lambda$. This is a condition within the fitted model, not evidence that the predictor would remain unselected under a different specification. In particular, a coefficient should not be translated into a causal claim merely because it is nonzero.

Selection can be useful when a compact predictive model is desired, but it also makes the reported model dependent on choices made during fitting. If a predictor is important for a business or regulatory reason, a zero estimate should prompt review of the modeling setup rather than automatic removal from every subsequent analysis. The lasso’s selection behavior is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus; exam questions may test both the threshold calculation and the limits of its interpretation.

Common Exam Traps

Using the wrong threshold. In the stated objective and orthogonal standardized case, compare $|z_j|$ with $\lambda$. Do not apply the rule without checking how the squared-error term and predictors were scaled.
Calling a zero coefficient proof of irrelevance. It means the variable is not selected in this particular penalized fit; it is not a hypothesis test or a universal conclusion.
Forgetting the intercept convention. The objective here does not penalize the intercept. Centering predictors and response makes the intercept separate from the slope calculations.
Thresholding ordinary least-squares coefficients in a general design. Coordinate-by-coordinate soft thresholding is exact for orthogonal standardized predictors. With general predictors, use the fitted residual correlations and the lasso optimality conditions.

Check Your Understanding

Assume centered, standardized, mutually orthogonal predictors, the squared-error objective with factor $1/(2n)$, and an unpenalized intercept. For each problem, identify the selected predictors when requested. Answers are shown for checking.

A fitted model has ordinary least-squares coefficients $z=(0.70,\ -0.90,\ 0.20)$ and $\lambda=0.70$. Find the lasso coefficients and identify selected predictors. (Answer: $(0,\ -0.20,\ 0)$; only predictor 2 is selected.)
For $z=(1.10,\ 0.35,\ -0.60)$ and $\lambda=0.40$, calculate the lasso coefficient vector. (Answer: $(0.70,\ 0,\ -0.20)$.)
At a proposed solution, a zero-coefficient predictor has residual correlation $0.25$, and $\lambda=0.25$. Does this satisfy the lasso condition for a zero coefficient? (Answer: Yes; $|0.25|\leq0.25$, including equality.)
For $z=(1.30,\ -0.55)$ and $\lambda=0.60$, find the coefficients and state why the second predictor is not selected. (Answer: $(0.70,\ 0)$; its ordinary least-squares coefficient has absolute value $0.55\leq0.60$, so thresholding sets it to zero.)
A model has $z=(0.80,\ 1.25,\ -0.40)$, $\lambda=0.50$, and intercept $4$. Find the lasso coefficients, count selected predictors, and predict at standardized predictor values $(2,\ -1,\ 3)$. (Answer: coefficients $(0.30,\ 0.75,\ 0)$, two selected predictors, prediction $4+(0.30)(2)+(0.75)(-1)+(0)(3)=3.85$.)

What You'll Learn