Why Initialize a Fit From the Previous Penalty?
A regularization path is a sequence of penalized regression fits on the same data, calculated at different penalty strengths. In genomic applications, the number of predictors can be much larger than the number of observations, so computing each fit from scratch may be expensive. A warm start uses the coefficient estimate at one penalty value as the initial coefficient vector for fitting a nearby value.
The motivation is computational: when two penalty values are close, their minimizers are often close too. Starting an iterative algorithm near the next minimizer can reduce the work needed to reach convergence. A warm start does not alter the objective, the target solution, or the convergence criterion. It is an initialization strategy, not a different estimator.
Set Up the Path and the Initialization
Suppose \(y\in\mathbb{R}^n\) is a response vector and \(X\in\mathbb{R}^{n\times p}\) is a predictor matrix. Assume the response and predictor columns have been centered, and that any intercept is fitted separately and left unpenalized. Use the same observations, response, predictor scaling, and penalty mixing parameter throughout the path. For \(\lambda>0\) and \(0\leq\alpha\leq1\), define the elastic-net estimate as a minimizer of
At \(\alpha=0\), this is ridge regression with penalty \(\lambda\|\beta\|_2^2/2\). At \(\alpha=1\), it is the lasso with penalty \(\lambda\|\beta\|_1\). Intermediate values give the elastic net. A path uses a descending sequence \(\lambda_1>\lambda_2>\cdots>\lambda_K>0\), with fixed \(\alpha\). The warm start at step \(k\) is \(\hat{\beta}(\lambda_{k-1},\alpha)\), used as the starting vector for the algorithm at \(\lambda_k\).
For a fixed \(\lambda>0\), the objective is convex. For ridge and elastic net with \(\alpha<1\), the positive quadratic penalty makes it strictly convex, so the coefficient minimizer is unique even if \(p>n\). The lasso objective is convex but need not have a unique coefficient minimizer when predictors are collinear. This distinction matters: warm starts are useful in either case, but for a nonunique lasso fit the path of coefficient vectors can depend on the algorithm's choices among minimizers.
Why Nearby Penalties Often Give Nearby Fits
A short sensitivity argument explains the warm-start intuition for ridge and elastic net. Write \(L(\beta)=\|y-X\beta\|_2^2/(2n)\), \(Q_\alpha(\beta)=\alpha\|\beta\|_1+(1-\alpha)\|\beta\|_2^2/2\), and \(G=X^\mathsf{T}X/n\). Let \(\beta_a\) and \(\beta_b\) minimize \(L(\beta)+\lambda_aQ_\alpha(\beta)\) and \(L(\beta)+\lambda_bQ_\alpha(\beta)\), respectively. Convex optimality gives
where \(g_a\) and \(g_b\) are subgradients of \(Q_\alpha\) at the corresponding estimates. Subtract the equations, set \(d=\beta_a-\beta_b\), and take the inner product with \(d\). The subgradient monotonicity of the convex function \(Q_\alpha\), together with its quadratic component, gives \(d^\mathsf{T}(g_a-g_b)\geq(1-\alpha)\|d\|_2^2\). Therefore,
If \(d\neq0\), dividing by \(\|d\|_2\) yields a bound on the distance between the fits:
For \(\alpha<1\), the denominator is positive when \(\lambda_a>0\), even if \(G\) is singular. This supports the expectation that sufficiently close penalties produce nearby estimates, although the bound may be loose and does not guarantee a particular reduction in computation. For the lasso, \(\alpha=1\), the quadratic penalty contributes no strong convexity; in a \(p>n\) problem, \(G\) is necessarily singular. Thus this global distance argument does not generally apply to lasso coefficients.
Worked Example: A Ridge Warm Start
Consider hypothetical centered data for which \(X^\mathsf{T}X/n=\operatorname{diag}(1,4)\) and \(X^\mathsf{T}y/n=(2,4)^\mathsf{T}\). For ridge regression, the minimizer satisfies \((G+\lambda I)\hat{\beta}=X^\mathsf{T}y/n\), so each coefficient is the corresponding entry of \(X^\mathsf{T}y/n\) divided by the diagonal entry of \(G+\lambda I\).
At \(\lambda=2\),
Use this as the initial vector at the nearby value \(\lambda=1\). The exact new fit is
The coefficient change is \((0.3333,0.1333)^\mathsf{T}\), with Euclidean length approximately \(0.3590\). This illustrates that the previous fit is a nearby, but not exact, starting value. In a ridge implementation that solves the system directly, there may be no iterative coefficient optimization to accelerate; the warm-start benefit applies to iterative solvers, while matrix factorizations or other reusable computations can provide separate efficiencies.
How Warm Starts Work in Practice
For coordinate descent, the previous coefficient vector supplies the initial coordinates and often a useful initial residual \(r=y-X\beta\). For proximal-gradient methods, it supplies the initial iterate before the gradient and penalty-proximal updates. In both cases, the algorithm continues until the stopping rule for the current \(\lambda\) is met. A path calculation should also update quantities tied to the current penalty; reusing an old residual or gradient without updating it for the current coefficient vector can make the implementation incorrect.
Keep the data, feature transformations, objective scaling, and \(\alpha\) unchanged across the path. Record the exact penalty sequence.
For a lasso path, a sufficiently large starting penalty can have the all-zero coefficient vector as a solution. Ridge coefficients generally are not exactly zero at any finite penalty, so compute a ridge starting fit by the chosen solver.
Initialize the solver at \(\hat{\beta}(\lambda_{k-1},\alpha)\), then optimize the objective using \(\lambda_k\), not \(\lambda_{k-1}\).
Apply the current fit's stopping criteria, verify the solution, and pass the converged estimate forward as the next warm start.
Worked Example: Lasso With Orthogonal Predictors
Suppose hypothetical centered and scaled predictors satisfy \(X^\mathsf{T}X/n=I\), and \(X^\mathsf{T}y/n=(2.4,1.1,-0.5)^\mathsf{T}\). With the lasso objective above, the coefficient for each orthogonal predictor is soft-thresholded: \(\hat{\beta}_j=\operatorname{sign}(z_j)\max(|z_j|-\lambda,0)\), where \(z=X^\mathsf{T}y/n\). Take the path from \(\lambda=1.5\) to \(\lambda=1.0\).
At \(\lambda=1.5\), the estimate is
Use that vector as the starting value at \(\lambda=1.0\). Applying the soft-thresholding formula at the new penalty gives
The second predictor enters the model as the penalty decreases. Under orthogonality, one coordinate update per coefficient reaches the exact solution, so this example verifies the path values but does not demonstrate fewer iterations than a cold start. With correlated predictors, coordinate updates interact, and a previous fit can provide a more useful initial point; the convergence check is still required.
Worked Example: An Elastic-Net Path Update
For hypothetical centered, scaled orthogonal predictors with \(X^\mathsf{T}X/n=I\) and \(X^\mathsf{T}y/n=(1.8,0.9)^\mathsf{T}\), take \(\alpha=0.5\). Under this design, the elastic-net solution is
At \(\lambda=1\), the threshold is \(1(0.5)=0.5\) and the denominator is \(1+1(0.5)=1.5\). Thus
At \(\lambda=0.8\), the threshold is \(0.8(0.5)=0.4\), and the denominator is \(1+0.8(0.5)=1.4\). The new fit is
The warm start \((0.8667,0.2667)^\mathsf{T}\) is close to the new fit, but it does not satisfy the new objective's optimality conditions exactly. The example also shows why the penalty value must be updated in both parts of the elastic-net penalty.
Diagnostics, Failure Modes, and Alternatives
A warm start is successful only if it reaches an adequately converged solution for the current penalty. Check the objective value and a suitable optimality measure at each path point. For coordinate descent, useful checks include the KKT conditions for the current objective: active coefficients should satisfy the appropriate stationarity equation, while zero coefficients must satisfy the lasso subgradient inequality. For proximal-gradient methods, check a stationarity or proximal-gradient residual, not just whether consecutive iterates look similar. Numerical convergence diagnostics are particularly important when the active set changes or coefficients are highly correlated.
Warm starts can fail to save time. A penalty grid with very large jumps may place the old fit far from the new solution. A path that uses an overly strict tolerance at every point can spend unnecessary computation, while a loose tolerance can pass inaccurate estimates forward. A warm-started solver may also appear stable because updates are tiny even when the KKT conditions are not satisfied. Compare objective and optimality diagnostics against the current penalty, and periodically compare selected fits with a cold-start solution.
| Approach | How it works | Strength and limitation |
|---|---|---|
| Warm-started path | Passes each converged fit to the next nearby penalty. | Often efficient for a dense, ordered path; its benefit depends on path spacing and solver behavior. |
| Cold starts | Begins each penalty fit from a fixed initial vector, often zero. | Useful as an implementation check or when fits are independent; may repeat work across nearby penalties. |
| Independent direct fits | Solves each penalty problem separately, possibly with a direct linear-algebra method. | Can be appropriate for ridge or a few isolated penalties; repeated computation may be wasteful for a long path. |
A warm-started path is not a substitute for a reliable solver or for validating the design matrix and scaling. Nor should one carry a coefficient vector uncritically between different response variables, different feature transformations, or different data sets: those are different optimization problems. When fitting separate training folds or resamples, each has its own response and design matrix; any warm start should be used only within the corresponding fixed problem. A lasso path can also have coefficient changes associated with predictors entering or leaving the active set, and collinearity can make individual coefficients unstable even when fitted values are similar.
Common Pitfall
Wrong reasoning: “The previous penalty's solution is close, so I can use it as the answer at the next penalty without checking convergence.” This confuses an initial point with an optimizer. Unless the old vector happens to satisfy the new objective's optimality conditions, it is not the new fit. Correction: initialize with the previous estimate, optimize at the new \(\lambda\), and verify convergence against the new objective. The purpose of a warm start is to reduce computation while preserving the target fit, not to replace that fit with an approximation chosen by convenience.
Check Your Understanding
Try each question, then compare your reasoning with the solutions below.
- For the objective in this tutorial, which value of \(\alpha\) gives ridge regression, and which gives the lasso?
- In a path from \(\lambda=2\) to \(\lambda=1\), what is the warm start for the second fit, and which penalty must the solver use while fitting it?
- For orthogonal lasso predictors with \(z=1.7\) and \(\lambda=0.6\), calculate the coefficient.
- Why does the strict-convexity argument for ridge and elastic net not generally guarantee a unique lasso coefficient vector when \(p>n\)?
- Name one diagnostic that should be checked at each path point and explain why checking only small changes between iterations is insufficient.
Solutions. (1) Ridge is \(\alpha=0\); lasso is \(\alpha=1\). (2) Initialize at the converged estimate for \(\lambda=2\), but optimize the objective with \(\lambda=1\). (3) Soft thresholding gives \(\operatorname{sign}(1.7)\max(1.7-0.6,0)=1.1\). (4) The lasso has no quadratic penalty term to ensure strict convexity, and when \(p>n\), \(X^\mathsf{T}X/n\) is singular; collinear predictors can therefore yield multiple coefficient minimizers. (5) Check a KKT or stationarity residual for the current penalty. Small changes between iterations can occur before the current objective's optimality conditions are satisfied.