Deriving the Lasso Null-Model Threshold

Why the Null-Model Threshold Matters

The lasso penalty controls how strongly slope coefficients are shrunk toward zero. At sufficiently large penalty values, the fitted model contains no selected predictors: its slopes are all zero, and only an intercept may remain. The boundary of this all-zero region is useful for checking a lasso implementation and for setting the upper end of a penalty search.

The threshold is determined by the response and the predictors through their correlations with the residuals from the intercept-only model. It is not a universal constant, and its numerical value depends on predictor scaling and on how the least-squares objective is normalized.

Model, Objective, and Assumptions

Let \(y\) be the response vector for \(n\) observations, and let \(X\) be the \(n\) by \(p\) predictor matrix. Write \(\beta_0\) for the intercept and \(\beta=(\beta_1,\ldots,\beta_p)^T\) for the slope coefficients. We use the lasso objective

The intercept is not penalized. The penalty parameter \(\lambda\) is nonnegative. The derivation below concerns this squared-error objective; it does not require the response errors to have a particular probability distribution. The values of \(X\) and \(y\) are treated as the observed data.

For a fixed \(\beta\), the intercept that minimizes the squared-error part is the mean response minus the predictor means multiplied by their coefficients. In particular, when \(\beta=0\), the intercept is \(\bar{y}\). Define the centered response \(y_c=y-\bar{y}\mathbf{1}\), where \(\mathbf{1}\) is the vector of \(n\) ones, and let \(X_c\) be the matrix whose \(j\)th column is \(x_{cj}=x_j-\bar{x}_j\mathbf{1}\). After minimizing over the intercept, the objective for the slopes becomes

If the predictors and response have already been centered, then \(X_c=X\) and \(y_c=y\). Centering is essential when an intercept is included: the relevant correlations are between centered predictors and the centered response, not between the original columns and the original response.

Deriving the Threshold From the KKT Conditions

For the convex objective \(Q\), the necessary and sufficient optimality condition at a candidate \(\beta\) is that zero belongs to its subgradient. The subgradient of \(|\beta_j|\) is \(\operatorname{sign}(\beta_j)\) when \(\beta_j\ne0\), and is any value in \([-1,1]\) when \(\beta_j=0\). At the all-zero vector, the condition for each coordinate is therefore

All \(p\) inequalities hold precisely when \(\lambda\) is at least the largest absolute centered predictor-response correlation, using the \(1/n\) normalization. Thus

At \(\lambda=\lambda_{\max}\), the all-zero slope vector is a minimizer. For any \(\lambda>\lambda_{\max}\), it is the unique minimizer in the slopes. At equality, other slope vectors can also minimize the objective in degenerate or collinear designs, so the formula guarantees that the zero vector is optimal, not that it is necessarily the only optimal vector.

Why the KKT Condition Is the Right Boundary

The coordinatewise condition gives the threshold directly. A short objective comparison also shows why it marks the boundary. Set \(M=\max_j|x_{cj}^Ty_c/n|\). For any slope vector,

The inequality follows by bounding each centered correlation in absolute value by \(M\), so that \(\left|\beta^TX_c^Ty_c/n\right|\leq M\|\beta\|_1\). If \(\lambda\geq M\), the right-hand side is nonnegative, proving that \(\beta=0\) is optimal. If \(\lambda>M\), it is strictly positive for every nonzero \(\beta\), proving uniqueness of the zero slope vector.

If \(\lambda<M\), some predictor \(j\) has \(|x_{cj}^Ty_c/n|>\lambda\). Moving a small distance \(t>0\) from zero in that coordinate’s correlation direction changes the objective by

The linear term is negative and the quadratic term is of order \(t^2\). For sufficiently small positive \(t\), the total change is negative. Hence zero cannot minimize the objective below the threshold.

Checks, Failure Modes, and Alternatives

The threshold provides a direct diagnostic for a lasso fit. At a penalty larger than the calculated \(\lambda_{\max}\), all fitted slopes should be zero, up to numerical tolerance. At the threshold, zero slopes should satisfy the KKT inequalities. If a purported null-model fit has a substantial nonzero slope at a penalty well above the threshold, check whether the software uses a different normalization, whether it standardized the predictors internally, and whether the intercept was handled as assumed.

• Forgetting the intercept. With an unpenalized intercept, use centered predictors and response. Raw inner products generally give a different answer.
• Mixing objective conventions. For an objective with \(1/(2n)\) times the sum of squared errors, use the factor \(1/n\) in the threshold. A differently scaled loss requires a correspondingly different numerical penalty.
• Ignoring predictor units. Multiplying a predictor by a constant multiplies its absolute correlation, and can change \(\lambda_{\max}\). Confirm the scaling used in the actual fit.
• Misreading the threshold as an inferential result. It is an optimization boundary for the observed data, not a p-value, a test of association, or evidence that all biological effects are absent.

The lasso threshold differs from ridge regression’s behavior: an \(L_2\) penalty generally shrinks coefficients continuously but does not set them exactly to zero at a finite positive penalty. Soft thresholding for orthogonal predictors gives a coordinatewise version of the same boundary idea: a coefficient is zero when its predictor-response quantity is within the penalty. With correlated predictors, the KKT condition still gives the all-zero boundary, even though it does not provide a separate soft-threshold formula for every fitted coefficient.

The formula also has practical limits. It assumes squared-error loss and an \(L_1\) penalty on slopes, with the stated scaling and intercept convention. It does not apply unchanged to a different loss, such as a likelihood-based objective, or to a penalty that also acts on the intercept. Further, \(\lambda_{\max}\) depends on the observed response: it is a data-dependent starting value for optimization, not a fixed scientific constant.