Warm Starts Along Regularization Paths

Why Warm Starts Matter Along a Regularization Path

An elastic net fit is often computed for many values of the penalty parameter \(\lambda\), for example to compare candidate models during validation. Solving every fit from an unrelated starting point can repeat work: when two penalty values are close, their fitted coefficient vectors are often close as well. A warm start uses the solution at one value of \(\lambda\) as the initial coefficient vector for the next fit.

The purpose is computational, not statistical. Each fit must still minimize the objective for its own value of \(\lambda\). A warm start does not make the preceding coefficients the answer to the new optimization problem, and it does not change the penalty, training data, or model being fitted. It can help an iterative algorithm reach the new solution in fewer iterations.

Definition: For a sequence of elastic net fits, a warm start initializes the optimization at a new tuning value using the coefficient estimates from a previously solved fit. The solver then continues optimizing the new objective until its convergence criterion is met. A cold start instead uses a fixed or generic initial value, such as zero, for each fit.

The Elastic Net Objective and a Lambda Path

Consider a training sample with response vector \(y\), predictor matrix \(X\), and \(p\) slope coefficients. Assume the predictors and response have been centered, and that columns are scaled according to the chosen preprocessing convention. With the intercept handled separately and not penalized, one common elastic net objective is

Q_{\lambda,\alpha}(\beta) = \frac{1}{2n}\lVert y-X\beta\rVert_2^2 + \lambda\left\{ \alpha\lVert\beta\rVert_1 + \frac{1-\alpha}{2}\lVert\beta\rVert_2^2 \right\}, \qquad \lambda\geq 0,\quad 0\leq\alpha\leq 1.

Here \(\alpha\) determines the balance between the lasso penalty and the ridge penalty. At \(\alpha=1\), the objective has a lasso penalty; at \(\alpha=0\), it has a ridge penalty. A lambda path is a sequence of fits at a fixed \(\alpha\) and fixed training data, with \(\lambda\) varied across a grid. The grid is commonly ordered from larger \(\lambda\) to smaller \(\lambda\): a stronger penalty is fitted first, and successive fits relax it.

For \(\alpha>0\), if the predictors and response are centered, a value large enough to make all slope estimates zero is

\lambda_{\max} = \frac{\max_{1\leq j\leq p}|x_j^{\mathsf T}y|/n}{\alpha}.

At \(\beta=0\), the derivative of the squared-error part with respect to coefficient \(j\) is \(-x_j^{\mathsf T}y/n\). The lasso part of the penalty can balance that derivative at zero when its allowable subgradient interval, \([-\lambda\alpha,\lambda\alpha]\), contains \(x_j^{\mathsf T}y/n\). This condition must hold for every predictor, giving the stated maximum. The formula does not give a finite all-zero threshold for pure ridge (\(\alpha=0\)), because ridge coefficients generally approach zero as \(\lambda\) grows but are not exactly zero at a finite \(\lambda\).

Starting at or above \(\lambda_{\max}\) can therefore provide a simple initial fit for a lasso-containing path. In practice, software may calculate this starting value using the training data and its preprocessing. The exact scaling convention matters: changing the loss normalization or predictor scaling changes the numerical value of \(\lambda_{\max}\).

Why the Previous Fit Is a Useful Initial Value

For fixed data and \(\alpha\), changing \(\lambda\) changes the strength of the penalty but does not replace the observations. When adjacent grid values are close, the minimizer often changes moderately. The preceding solution is then a more informed initial point than zero or an arbitrary vector. This is particularly useful for iterative methods such as coordinate descent and proximal gradient methods, which update coefficients repeatedly.

The reason can be seen directly in a one-predictor case. Suppose the predictor has been centered and scaled so that \(x^{\mathsf T}x/n=1\), and let \(c=x^{\mathsf T}y/n\). The objective, up to a constant independent of \(\beta\), is

Q_{\lambda,\alpha}(\beta) = \frac{1}{2}(\beta-c)^2 + \lambda\left\{\alpha|\beta|+\frac{1-\alpha}{2}\beta^2\right\}.

If \(c>0\) and the minimizing coefficient is positive, differentiating the objective in that region gives

(\beta-c)+\lambda\alpha+\lambda(1-\alpha)\beta=0, \qquad \widehat{\beta}_{\lambda} = \frac{c-\lambda\alpha}{1+\lambda(1-\alpha)} \quad\text{when } c>\lambda\alpha.

This expression shows that the coefficient depends on \(\lambda\). For nearby values of \(\lambda\), the minimizer in this simple case is nearby too. The warm start uses that fact as a practical initialization strategy; it does not assume the new minimizer equals the old one. With multiple correlated predictors, coefficient paths can be more complicated, and a coefficient can enter or leave the model. The previous fit remains only a starting point.

Worked Example: Finding a Starting Point and Following an Orthogonal Path

Suppose a centered response and two centered, mutually orthogonal predictors satisfy \(x_1^{\mathsf T}x_1/n=x_2^{\mathsf T}x_2/n=1\), \(x_1^{\mathsf T}x_2/n=0\), and \(x_1^{\mathsf T}y/n=1.2\), \(x_2^{\mathsf T}y/n=-0.6\). Fit an elastic net with \(\alpha=0.5\). Find the all-zero threshold and the coefficients at \(\lambda=1.2\) and \(\lambda=0.4\). State how the first fit can initialize the second.

Method. Orthogonality makes the objective separate into one coefficient problem per predictor. For coefficient \(j\), the solution is zero if \(|c_j|\leq\lambda\alpha\); otherwise it is \(\operatorname{sign}(c_j)(|c_j|-\lambda\alpha)/[1+\lambda(1-\alpha)]\).

All-zero threshold. The largest absolute predictor-response cross-product per observation is \(\max(1.2,0.6)=1.2\). Thus \(\lambda_{\max}=1.2/0.5=2.4\). At \(\lambda=3\), which is above 2.4, both slope coefficients are zero.

At \(\lambda=1.2\). The lasso threshold is \(\lambda\alpha=1.2(0.5)=0.6\), and the denominator is \(1+\lambda(1-\alpha)=1+1.2(0.5)=1.6\). For predictor 1, \((1.2-0.6)/1.6=0.375\). For predictor 2, \(|-0.6|=0.6\), exactly the threshold, so its coefficient is zero. The fitted slope vector is \((0.375,0)\).

At \(\lambda=0.4\). The threshold is \(0.4(0.5)=0.2\), and the denominator is \(1+0.4(0.5)=1.2\). The coefficients are \((1.2-0.2)/1.2=0.8333\) and \(-\,(0.6-0.2)/1.2=-0.3333\). The slope vector is approximately \((0.8333,-0.3333)\).

Use \((0.375,0)\), the converged solution at \(\lambda=1.2\), as the initial vector for the \(\lambda=0.4\) optimization. The second solution is different, so the solver must continue updating until it meets its stopping criterion. The common mistake is to treat the earlier vector as the new answer rather than as an initialization.

Check. At \(\lambda=0.4\), both coefficient magnitudes exceed the threshold \(0.2\), and substituting into the positive- or negative-coefficient formula gives \(0.8333\) and \(-0.3333\), respectively.

Warm Starts in an Iterative Solver

A proximal gradient update illustrates how the initial vector affects computation while leaving the target solution unchanged. For the one-predictor objective above, split the smooth part as \(g(\beta)=\frac{1}{2}(\beta-c)^2+\lambda(1-\alpha)\beta^2/2\), and the nonsmooth part as \(\lambda\alpha|\beta|\). For step size \(\eta>0\), the update is

\beta^{(k+1)} = \operatorname{soft}\!\left( \beta^{(k)}-\eta g'(\beta^{(k)}),\, \eta\lambda\alpha \right), \qquad \operatorname{soft}(z,t)=\operatorname{sign}(z)\max(|z|-t,0).

The soft-thresholding operation applies the lasso penalty, while the gradient step accounts for squared error and the ridge penalty. Under a suitable step size, repeated updates approach the minimizer. A warm start changes \(\beta^{(0)}\), not the update rule or objective.

Worked Example: Comparing Warm and Cold Starts

Use the one-predictor objective with \(c=2\), \(\alpha=0.5\), and first fit \(\lambda=1\), then fit \(\lambda=0.8\). For the second fit, apply proximal gradient with \(\eta=0.5\). Compare starting from zero with starting from the converged first-fit coefficient. Count updates until the estimate is within 0.01 of the new solution.

First fit. Since \(2>1(0.5)\), the positive-coefficient formula applies. The solution is \((2-0.5)/[1+1(0.5)]=1.5/1.5=1\). This is the warm-start value.

New target. At \(\lambda=0.8\), the solution is \((2-0.8(0.5))/[1+0.8(0.5)]=1.6/1.4=8/7\), or approximately \(1.142857\). For positive iterates, \(g'(\beta)=1.4\beta-2\). The update becomes \(\operatorname{soft}(0.3\beta+1,0.2)=0.3\beta+0.8\), since the input is positive and exceeds 0.2.

Cold start from zero. The iterates are \(\beta^{(1)}=0.8\), \(\beta^{(2)}=1.04\), \(\beta^{(3)}=1.112\), and \(\beta^{(4)}=1.1336\). The fourth iterate differs from \(8/7\) by about \(0.0093\), within 0.01. The third differs by about \(0.0309\), so four updates are needed.

Warm start from one. The iterates are \(\beta^{(1)}=1.1\), \(\beta^{(2)}=1.13\), and \(\beta^{(3)}=1.139\). The third iterate differs from \(8/7\) by about \(0.0039\), while the second differs by about \(0.0129\). Thus three updates suffice.

Check. Both sequences target \(8/7\), and the update multiplies the error by 0.3. The warm start reduces the update count in this example from four to three; this small example illustrates the mechanism, not a guaranteed speed-up of a particular size.

When Warm Starts Are Appropriate

The clearest use is a sequence of nearby \(\lambda\) values with the same \(\alpha\), same training observations, same response, same predictors, and same preprocessing. Under these conditions, the objective changes in one controlled way, and the previous solution is a natural initial estimate. Solving from larger to smaller \(\lambda\) is common because the strongly penalized fit is often simple, including an all-zero slope fit when \(\alpha>0\) and \(\lambda\geq\lambda_{\max}\).

Warm starts can also be used when values of \(\lambda\) are visited in another order, but the starting point may be less useful if the new value is far away. A wide gap in the grid, a sharp change in active coefficients, or a difficult optimization problem may mean that the solver still needs many iterations. Always use convergence checks for the current objective; do not stop merely because the starting vector came from a converged fit.

Fix the data and preprocessing.
Construct the training design matrix and apply the chosen centering and scaling using only the data for this fit.

Choose the path.
Hold \(\alpha\) fixed and order the \(\lambda\) values, commonly from largest to smallest.

Initialize the next fit.
Use the preceding converged coefficient vector, with the intercept handled according to the software's convention.

Optimize and check convergence.
Run the solver for the new \(\lambda\) and verify its stopping rule, such as a sufficiently small change in objective or coefficients.

The same principle applies inside cross-validation, but each fold has its own training data and therefore its own sequence of objectives. Warm starts along the lambda path within one fold are appropriate. Reusing coefficients across folds is not a substitute for fitting on each fold's training observations. In particular, initializing a fold fit from coefficients estimated using its held-out observations risks allowing validation information into the fitting procedure. Keep fold-specific preprocessing and fitting separate, as required for a valid validation plan.

Worked Example: Keeping Warm Starts Separate Across Folds

Suppose a two-fold validation procedure fits one standardized predictor separately in each training fold. Use \(\alpha=0.5\), with \(c=x^{\mathsf T}y/n=1.4\) in fold A and \(c=0.9\) in fold B. Compute each fold's coefficients at \(\lambda=1\) and then at \(\lambda=0.5\). Identify the valid warm start for each second fit.

Method. In each fold, use \(\widehat{\beta}=(c-\lambda\alpha)/[1+\lambda(1-\alpha)]\) when \(c>\lambda\alpha\). The fold-specific value of \(c\) comes from that fold's training data.

Fold A. At \(\lambda=1\), the threshold is 0.5 and the denominator is 1.5, giving \((1.4-0.5)/1.5=0.6\). At \(\lambda=0.5\), the threshold is 0.25 and the denominator is 1.25, giving \((1.4-0.25)/1.25=0.92\). Initialize fold A's \(\lambda=0.5\) fit at 0.6.

Fold B. At \(\lambda=1\), the coefficient is \((0.9-0.5)/1.5=0.2667\). At \(\lambda=0.5\), it is \((0.9-0.25)/1.25=0.52\). Initialize fold B's \(\lambda=0.5\) fit at 0.2667.

Check and trap. The two second-fit targets differ because their training data differ. Using fold A's coefficient 0.6 as fold B's warm start would not change fold B's mathematical target if it were merely an initial vector and the solver fully converged, but it is not the natural fold-specific path initialization and can be misleading if fitting is stopped early or information from held-out data was used to obtain it. Each fold should use its own preceding solution and its own training-only preprocessing.

Common Mistakes and Practical Limits

Treating an initial value as a fitted answer. The old coefficients generally do not minimize the new-\(\lambda\) objective. Optimize again and check convergence.
Assuming the active predictors only accumulate. Coefficients may enter, leave, or change sign along a path, particularly with correlated predictors. A warm start does not imply nested selected-variable sets.
Comparing coefficients on different scales as if they were interchangeable. A change in centering, scaling, predictor order, or coding changes how coefficients correspond to the optimization variables. Align the parameterization before transferring an initial vector.
Carrying an initialization between different data sets without care. Each validation fold has a distinct training objective. Keep the path initialization fold-specific and prevent held-out observations from influencing fitting or preprocessing.
Expecting a speed-up in every case. A distant starting value, difficult optimization, or already inexpensive fit may offer little benefit. Warm starts help computation only when they reduce the work needed to meet the required accuracy.

For \(\alpha<1\) and \(\lambda>0\), the ridge component contributes strict convexity in the slope coefficients under this objective, so the slope minimizer is unique. At \(\alpha=1\), uniqueness can depend on the design matrix. Regardless, a solver's convergence criteria and the model's coefficient parameterization matter: a warm start is an algorithmic convenience, not a guarantee that every path fit has a unique solution or takes fewer iterations.

Key takeaway: Along a fixed-data, fixed-\(\alpha\) lambda path, initialize each fit with the preceding converged solution when that is computationally useful. The current objective must still be optimized and checked, and validation folds must retain separate training data, preprocessing, and fit paths.

Check Your Understanding

For each problem, distinguish the path initialization from the solution of the current objective.

A centered one-predictor fit has \(c=x^{\mathsf T}y/n=1.5\), \(\alpha=0.4\), and \(\lambda=1\). Find its positive elastic net coefficient. (Answer: \(0.6875\))
For centered predictors with \(\max_j|x_j^{\mathsf T}y|/n=1.1\) and \(\alpha=0.5\), find \(\lambda_{\max}\). What are the slope coefficients at \(\lambda=2.5\)? (Answer: \(\lambda_{\max}=2.2\); all slopes are zero at \(\lambda=2.5\))
For a one-predictor fit with \(c=1.8\), \(\alpha=0.5\), and \(\lambda=0.6\), use the solution at the preceding value as \(\beta^{(0)}=1.4\). With proximal-gradient step size \(\eta=0.5\), compute one update. (Answer: \(\beta^{(1)}=1.24\))
In a validation procedure, fold A's \(\lambda=1\) solution is 0.6 and fold B's is 0.2667. Which value should initialize each fold's next fit at \(\lambda=0.5\), assuming each fold follows its own path? (Answer: initialize fold A at 0.6 and fold B at 0.2667)
A solver is warm-started at the previous lambda solution and returns after one update, but its convergence criterion is not met. May that returned vector be reported as the converged fit at the new lambda? Explain. (Answer: No; continue optimization until the new objective's convergence criterion is met.)

What You'll Learn