Regularization Paths and Warm Starts

Why Fit a Sequence of Models?

Penalized GLM estimation usually asks for more than one fit. A practitioner may need to understand how the fitted coefficients change as the penalty increases or decreases, or to obtain candidate fits at a specified set of penalty values. Rather than treat each value as an unrelated optimization problem, we can organize the solutions into a regularization path.

A path is useful computationally and descriptively. Computationally, a fit at one penalty value is often a good starting point for the next nearby value. Descriptively, plotting coefficients against the penalty shows which coefficients become nonzero, how quickly they shrink, and whether they later return to zero. These patterns summarize the fitted models; they do not by themselves establish significance or causality.

Definition and Construction of a Path

Definition: Let $L(\beta_0,b)$ be a differentiable convex GLM loss, where $\beta_0$ is an unpenalized intercept and $b\in\mathbb{R}^p$ is the vector of penalized slopes. For a fixed mixing value $\alpha\in[0,1]$, the penalized objective at $\lambda\geq 0$ is $$ Q_\lambda(\beta_0,b) = L(\beta_0,b) + \lambda\left\{\alpha\lVert b\rVert_1+ \frac{1-\alpha}{2}\lVert b\rVert_2^2\right\}. $$ A regularization path is the collection of minimizers of $Q_\lambda$ over a specified sequence of penalty values, usually ordered from large to small.

The loss may be a negative log-likelihood, often scaled by sample size, and may include an offset. The intercept is excluded from the penalty. The lasso corresponds to $\alpha=1$, ridge to $\alpha=0$, and intermediate values to the elastic net. A path is a sequence of solutions to the chosen objective; its numerical values depend on the loss scaling and penalty convention, so those must be kept fixed throughout.

A typical computation begins with a decreasing grid $\lambda_1>\lambda_2>\cdots>\lambda_K$. At each value, the optimizer seeks a minimizer and checks the KKT conditions, as in the earlier material on penalized fitting. The fitted slopes at $\lambda_k$ are then used to initialize the optimization at $\lambda_{k+1}$. A grid may be denser where coefficient patterns change quickly, but the grid itself is a computational choice rather than a mathematical requirement.

Worked Example: The Lasso Entry Threshold and a Simple Path

Consider the convex loss $L(b)=\frac12\lVert y-Xb\rVert_2^2$, with no intercept, two columns satisfying $X^\mathsf{T}X=I$, and $X^\mathsf{T}y=(3,1.2)^\mathsf{T}$. Terms in the loss that do not depend on $b$ can be dropped, giving $L(b)=\frac12\lVert b\rVert_2^2-(3,1.2)b$, up to a constant. Use the lasso objective $L(b)+\lambda\lVert b\rVert_1$.

For each coordinate, minimizing $\frac12 b_j^2-c_jb_j+\lambda|b_j|$ gives the soft-thresholded solution $b_j=\operatorname{sign}(c_j)(|c_j|-\lambda)_+$, where $c=(3,1.2)$ and $x_+=\max(x,0)$. Thus the path is

b_1(\lambda)=(3-\lambda)_+,\qquad b_2(\lambda)=(1.2-\lambda)_+.

At $\lambda=3$, both coefficients are zero; for $\lambda>3$, they remain zero. The first coefficient becomes positive as $\lambda$ falls below 3, while the second enters only below 1.2. For example, at $\lambda=2$, $b=(1,0)$; at $\lambda=1$, $b=(2,0.2)$; and at $\lambda=0.5$, $b=(2.5,0.7)$. The calculation makes clear that decreasing the penalty allows coefficients to grow away from zero, although more complicated correlated designs need not have such simple paths.

When Are All Slopes Zero?

For a lasso-containing penalty, the intercept-only fit provides a natural starting point at the high-penalty end of the path. The relevant condition follows directly from the lasso KKT condition at zero slopes.

Result: Suppose $L(\beta_0,b)$ is differentiable and convex, and the intercept-only minimizer $\hat\beta_0$ of $L(\beta_0,0)$ is unique. For $\alpha>0$, define $$ \lambda_{\max} = \frac{\left\lVert\nabla_b L(\hat\beta_0,0)\right\rVert_\infty}{\alpha}. $$ Then $(\hat\beta_0,0)$ minimizes $Q_\lambda$ for every $\lambda\geq\lambda_{\max}$. Moreover, $\lambda_{\max}$ is the smallest penalty at which that intercept-only fit is optimal.

To prove the result, first note that differentiability and intercept optimality give $\partial L(\hat\beta_0,0)/\partial\beta_0=0$. At $b=0$, the ridge part of the penalty has derivative zero. The subgradient of $\lVert b\rVert_1$ at zero is the set of vectors whose coordinates lie in $[-1,1]$. Thus the slope KKT condition is

\nabla_b L(\hat\beta_0,0)+\lambda\alpha s=0 \quad\text{for some }s\text{ with }\lVert s\rVert_\infty\leq 1.

Such an $s$ exists exactly when $\lVert\nabla_b L(\hat\beta_0,0)\rVert_\infty\leq\lambda\alpha$. This is equivalent to $\lambda\geq\lambda_{\max}$. For a convex objective, the KKT conditions are sufficient for global optimality, so zero slopes are optimal for every such $\lambda$. If $\lambda<\lambda_{\max}$, the slope KKT condition fails at this intercept-only fit, so that fit is not optimal. This proves both claims. $\square$

The word threshold matters: $\lambda_{\max}$ is the smallest penalty at which the intercept-only fit is optimal, not a largest penalty. Zero slopes remain optimal for arbitrarily large penalties. When $\alpha=0$, there is no lasso threshold of this form; a ridge penalty generally shrinks slopes continuously rather than setting them exactly to zero.

Worked Example: Computing the Threshold From a Score

For a logistic GLM with an intercept and two slopes, suppose the intercept-only fitted probability is $0.4$ for each of five observations. Let the binary outcomes be $y=(1,0,1,0,0)$, and let the two predictor columns be $x_1=(1,0,1,0,0)$ and $x_2=(0,1,0,1,0)$. Use the average negative log-likelihood and the lasso penalty, so $\alpha=1$.

For this loss, the slope gradient at the intercept-only fit is the negative average of $x_j(y-0.4)$. For the first predictor, $x_1^\mathsf{T}(y-0.4)=0.6+0.6=1.2$, so its gradient is $-1.2/5=-0.24$. For the second, $x_2^\mathsf{T}(y-0.4)=-0.4-0.4=-0.8$, so its gradient is $0.8/5=0.16$. Therefore

\lambda_{\max} = \frac{\max(0.24,0.16)}{1} = 0.24.

At any $\lambda\geq0.24$, the intercept-only solution satisfies the slope KKT conditions and is optimal. At $\lambda=0.2$, it fails because the first score magnitude $0.24$ exceeds the available lasso threshold $0.2$.

Warm Starts and Changes Between Fits

A warm start initializes the optimization at a new penalty value using the preceding fit. For a decreasing sequence, the estimate at $\lambda_k$ is the initial coefficient vector at $\lambda_{k+1}$. This can save computation because a nearby penalty often has a nearby minimizer, so the optimizer begins closer to its target than it would from zero or an arbitrary starting point. It is an efficiency strategy, not a guarantee of a fixed iteration count or of convergence by itself; each fit still needs a suitable stopping check.

The following bound makes the proximity claim precise in a setting with strong convexity. For the bound, suppress the intercept or regard it as already included in a loss $L(b)$. Suppose $L$ is differentiable and $\mu_0$-strongly convex: for all $u,v$, $(\nabla L(u)-\nabla L(v))^\mathsf{T}(u-v)\geq\mu_0\lVert u-v\rVert_2^2$, with $\mu_0>0$. Let $b_\lambda$ minimize the elastic-net objective. Minimizers are assumed to exist.

Theorem: For $\lambda\geq\lambda'\geq0$, and $0\leq\alpha\leq1$, let $b_\lambda$ and $b_{\lambda'}$ minimize $L(b)+\lambda\{\alpha\lVert b\rVert_1+(1-\alpha)\lVert b\rVert_2^2/2\}$ at their respective penalties. Then $$ \lVert b_\lambda-b_{\lambda'}\rVert_2 \leq \frac{(\lambda-\lambda')\left\{\alpha\sqrt p+(1-\alpha)\lVert b_\lambda\rVert_2\right\}} {\mu_0+\lambda'(1-\alpha)}. $$ In the lasso case $\alpha=1$, this gives $\lVert b_\lambda-b_{\lambda'}\rVert_2\leq(\lambda-\lambda')\sqrt p/\mu_0$.

For the proof, write $d=b_\lambda-b_{\lambda'}$. Optimality gives subgradients $s_\lambda\in\partial\lVert b_\lambda\rVert_1$ and $s_{\lambda'}\in\partial\lVert b_{\lambda'}\rVert_1$ satisfying

\nabla L(b_\lambda)+\lambda\{\alpha s_\lambda+(1-\alpha)b_\lambda\}=0,\qquad \nabla L(b_{\lambda'})+\lambda'\{\alpha s_{\lambda'}+(1-\alpha)b_{\lambda'}\}=0.

Subtract the equations, take the inner product with $d$, and split each penalty term into a part at $\lambda'$ and a part at $\lambda-\lambda'$. Strong convexity bounds the loss contribution below by $\mu_0\lVert d\rVert_2^2$. Monotonicity of the lasso subgradient gives $(s_\lambda-s_{\lambda'})^\mathsf{T}d\geq0$; the ridge part at $\lambda'$ contributes $\lambda'(1-\alpha)\lVert d\rVert_2^2$. The remaining term is $(\lambda-\lambda')\{\alpha s_\lambda+(1-\alpha)b_\lambda\}^\mathsf{T}d$. Since every coordinate of $s_\lambda$ has magnitude at most 1, $\lVert s_\lambda\rVert_2\leq\sqrt p$; Cauchy–Schwarz bounds the absolute value of this remaining term by

(\lambda-\lambda')\{\alpha\sqrt p+(1-\alpha)\lVert b_\lambda\rVert_2\}\lVert d\rVert_2.

Combining the bounds yields the theorem after division by $\lVert d\rVert_2$ when $d\ne0$; when $d=0$, the result is immediate. Setting $\alpha=1$ gives the stated lasso-only bound. If the loss is not strongly convex, a positive ridge component can supply additional strong convexity at positive penalty values; the assumptions and denominator must then be adjusted accordingly. $\square$

Worked Example: Why a Nearby Fit Is a Useful Initialization

Return to the orthogonal two-predictor lasso example, with score vector $(3,1.2)$. The exact solution at $\lambda=1$ is $b_1=(2,0.2)$. At the nearby penalty $\lambda'=0.9$, the exact solution is $b_{0.9}=(2.1,0.3)$, so their Euclidean distance is $\sqrt{0.1^2+0.1^2}=\sqrt{0.02}\approx0.1414$.

If we use $b_1$ as the starting point at $\lambda'=0.9$, both coordinates are positive. The gradient of the smooth loss is $b-(3,1.2)$. Adding the lasso gradient $0.9(1,1)$ at this starting point gives the KKT residual $(2-3+0.9,\;0.2-1.2+0.9)=(-0.1,-0.1)$, whose Euclidean norm is about $0.1414$. Starting from $b=(0,0)$, the corresponding score magnitudes beyond the lasso threshold are $3-0.9=2.1$ and $1.2-0.9=0.3$, much larger. This example illustrates why a warm start can begin closer to satisfying optimality conditions. It does not imply that every GLM optimizer will take a particular number of iterations.

Reading Coefficient Paths

A coefficient path plot places $\lambda$, often on a logarithmic horizontal scale, on the horizontal axis and each fitted slope on the vertical axis. With a decreasing penalty, a lasso coefficient may enter from zero, grow in magnitude, or—in correlated or otherwise non-orthogonal designs—return to zero. Ridge coefficients are generally nonzero at finite penalties and tend to change continuously, so “entry” is most directly meaningful for lasso or elastic-net fits with a nonzero lasso component.

Path plots can reveal which predictors are affected early by penalization and how strongly estimates depend on the penalty. They should be read alongside the model specification and predictor scaling: different measurement scales can produce misleading comparisons unless predictors have been standardized appropriately. A path is not a ranking of causal importance, and a coefficient’s position on the path is not a p-value.

Worked Example: Correlation and Elastic-Net Shrinkage

Consider the two-slope convex quadratic loss $L(b)=\frac12b^\mathsf{T}Gb-c^\mathsf{T}b$, where $G=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}$ and $c=(1.25,0.8)^\mathsf{T}$. The matrix $G$ is positive definite because its determinant is $1-0.25=0.75>0$. Use the elastic-net objective with $\alpha=0.5$.

At $\lambda=0.5$, if both coefficients are positive, the KKT equations are $(G+0.25I)b=c-(0.25,0.25)^\mathsf{T}=(1,0.55)^\mathsf{T}$. The system has matrix $\begin{pmatrix}1.25&0.5\\0.5&1.25\end{pmatrix}$, with determinant $1.25^2-0.5^2=1.3125$. Solving gives

b_1=\frac{1.25(1)-0.5(0.55)}{1.3125} =\frac{0.975}{1.3125}\approx0.7429,\qquad b_2=\frac{1.25(0.55)-0.5(1)}{1.3125} =\frac{0.1875}{1.3125}\approx0.1429.

Both values are positive, so the assumed signs are consistent. At $\lambda=1$, the positive-sign system has matrix $\begin{pmatrix}1.5&0.5\\0.5&1.5\end{pmatrix}$ and right side $(0.75,0.3)^\mathsf{T}$, yielding $b=(0.4875,0.0375)^\mathsf{T}$. At $\lambda=1.5$, the candidate with the second coefficient zero has $b_1=(1.25-0.75)/(1+0.75)=2/7$. The inactive-coordinate score is $0.8-0.5(2/7)=0.6571$, which is less than the lasso threshold $\lambda\alpha=0.75$; hence zero is valid for the second coefficient. The path therefore shows the second coefficient shrinking to zero as the penalty grows. The two predictors’ correlation makes their joint path differ from the simple coordinatewise thresholding in the orthogonal example.

Common Mistakes and Practical Checks

Calling $\lambda_{\max}$ the largest zeroing penalty. The threshold is the smallest penalty at which the intercept-only fit is optimal; every larger penalty also keeps that fit optimal.
Assuming every path is monotone. Coefficient magnitudes need not change monotonically in correlated designs, and lasso coefficients can leave and re-enter.
Treating warm starts as a convergence guarantee. They provide an initialization. Each fit still requires optimization and a KKT-based stopping check.
Comparing path magnitudes without checking scaling. Penalty and loss conventions, as well as predictor scales, affect coefficient paths.
Interpreting entry as evidence of significance. A path records penalized estimates across tuning values; it is not a classical inferential procedure.

In practice, a useful path begins at a penalty known to yield zero slopes when a lasso component is present, then moves through a decreasing sequence using warm starts. Keep the objective definition fixed, verify convergence at each point, and interpret the plots as a description of how penalized estimates respond to changing regularization.

Check Your Understanding

Answer these questions using the objective and ideas developed above.

For a lasso GLM with a unique intercept-only fit, what score determines the threshold at which all slopes are zero?
Why does the lasso-only sensitivity bound require strong convexity of the loss as stated?
What is a warm start when fitting a decreasing sequence of penalty values, and what does it not guarantee?
Why can a coefficient enter or leave a lasso path differently when predictors are correlated?
Why should coefficient entry along a path not be interpreted as a test of statistical significance?