Introduction
The elastic net combines an \(L_1\) penalty, which can set coefficients exactly to zero, with an \(L_2\) penalty, which shrinks coefficients toward zero. Its mixing parameter controls the balance. At one endpoint, the objective is lasso; at the other, it is ridge. Knowing which endpoint is which matters when interpreting a fitted model: the two penalties do not generally produce the same pattern of coefficients.
This tutorial uses a squared-error regression objective and an explicit mixing-parameter convention. The conclusions about the endpoints follow directly from that objective. They do not depend on interpreting a fitted coefficient as a causal effect.
What the Endpoints Mean
Substitute each endpoint into the objective rather than relying on the name of a software argument. When \(\rho=1\), the \(L_2\) term has coefficient zero and the objective becomes
which is the lasso objective in the stated scaling. When \(\rho=0\), the \(L_1\) term vanishes and the objective becomes
which is the ridge objective in the same scaling. Thus the endpoints concern the mixing parameter, not the penalty strength. At either endpoint, setting \(\lambda=0\) removes regularization and leaves ordinary least squares (when a least-squares solution exists).
The lasso’s \(L_1\) term can produce exact zeros. Ridge instead uses a smooth quadratic penalty, so it usually shrinks coefficients without setting them exactly to zero. “Usually” is important: ridge coefficients can be zero in special cases, such as when a predictor has no association with the response after accounting for the other predictors.
A Useful Formula for Orthogonal Predictors
A direct derivation makes the endpoint behaviors especially clear. Suppose the response and predictor columns are centered, and the columns are orthogonal and scaled so that \(x_j^\mathsf{T}x_k/n=1\) for \(j=k\) and \(0\) otherwise. Define \(z_j=x_j^\mathsf{T}y/n\). Up to terms that do not depend on \(\beta\), the squared-error part of the objective for coefficient \(j\) is \(\frac{1}{2}(\beta_j-z_j)^2\). Therefore the minimization separates into one problem per coefficient.
For a nonzero coefficient, differentiate the smooth terms and include the sign of the absolute-value penalty. The stationarity equation is
If \(z_j>\lambda\rho\), the solution is positive and rearranging gives \(\beta_j=(z_j-\lambda\rho)/(1+\lambda(1-\rho))\). If \(z_j<-\lambda\rho\), it is negative and the same argument gives \(\beta_j=(z_j+\lambda\rho)/(1+\lambda(1-\rho))\). At zero, the subgradient condition for the absolute value is satisfied exactly when \(|z_j|\leq\lambda\rho\). Combining these cases gives the soft-thresholding expression below, where \(S(z,t)=\operatorname{sign}(z)\max(|z|-t,0)\).
The derivation also explains the distinction. Lasso has a nonzero threshold \(\lambda\), so a coefficient is zero whenever \(|z_j|\leq\lambda\). Ridge has threshold zero in this formula: it scales \(z_j\) by \(1/(1+\lambda)\), rather than applying a positive cutoff. For a nonzero \(z_j\), that scaling leaves a nonzero coefficient.
Worked Example: Comparing Both Endpoints on One Orthogonal Design
Suppose three centered, standardized predictors are orthogonal, and their response summaries are \(z=(1.2,0.4,-0.8)\). Fit the elastic net with \(\lambda=0.5\) at each endpoint. Find the coefficient vectors and identify which endpoint selects a zero coefficient.
Method. Use the orthogonal-design endpoint formulas. The shared scaling assumption is essential; the summaries \(z_j\) alone would not justify these formulas for correlated predictors.
Lasso endpoint. Set \(\rho=1\). The threshold is \(\lambda=0.5\). Applying soft-thresholding component by component gives \(S(1.2,0.5)=1.2-0.5=0.7\), \(S(0.4,0.5)=0\) because \(0.4\leq0.5\), and \(S(-0.8,0.5)=-(0.8-0.5)=-0.3\). Thus \(\widehat{\beta}=(0.7,0,-0.3)\). The middle coefficient is zero because its absolute summary is below the threshold.
Ridge endpoint. Set \(\rho=0\). The denominator is \(1+\lambda=1.5\), so the coefficients are \(1.2/1.5=0.8\), \(0.4/1.5=0.2667\), and \(-0.8/1.5=-0.5333\). To four decimal places, \(\widehat{\beta}=(0.8000,0.2667,-0.5333)\).
Check. Lasso has one exact zero, while all three ridge coefficients are nonzero. Each ridge coefficient has the same sign as its corresponding \(z_j\) and has smaller absolute value. The common mistake is to apply the lasso threshold to ridge; ridge divides by \(1+\lambda\) and does not use that threshold.
Following a Coefficient as the Mix Changes
For a fixed positive \(\lambda\), changing \(\rho\) changes both the soft threshold and the denominator. The threshold is \(\lambda\rho\), while the denominator is \(1+\lambda(1-\rho)\). As \(\rho\) increases toward one, the threshold grows and the denominator decreases. Consequently, a coefficient’s size need not move in one simple direction throughout the whole path. The endpoint distinction is more reliable than a blanket claim that increasing \(\rho\) always increases or always decreases every coefficient.
Worked Example: Locating the Zero-Coefficient Boundary
For one orthogonal standardized predictor, let \(z=0.4\) and \(\lambda=0.5\). Calculate the coefficient at \(\rho=0\), \(\rho=0.5\), \(\rho=0.8\), and \(\rho=1\). State where it first becomes exactly zero as \(\rho\) increases.
Method. Apply \(\widehat{\beta}=S(z,\lambda\rho)/(1+\lambda(1-\rho))\). Since \(z>0\), the coefficient is positive when \(z>\lambda\rho\), and is zero when \(z\leq\lambda\rho\).
At \(\rho=0\), the threshold is \(0\), the denominator is \(1+0.5=1.5\), and \(\widehat{\beta}=0.4/1.5=0.2667\).
At \(\rho=0.5\), the threshold is \(0.5(0.5)=0.25\), and the denominator is \(1+0.5(0.5)=1.25\). Therefore \(\widehat{\beta}=(0.4-0.25)/1.25=0.15/1.25=0.12\).
At \(\rho=0.8\), the threshold is \(0.5(0.8)=0.4\). Since \(z\) equals the threshold, soft-thresholding gives zero; the denominator is \(1+0.5(0.2)=1.1\), so the coefficient remains zero. At \(\rho=1\), the threshold is \(0.5\), which exceeds \(z=0.4\), so the coefficient is also zero.
Check. The boundary is \(z=\lambda\rho\), or \(\rho=0.4/0.5=0.8\). The coefficient is positive below that boundary and exactly zero at and above it. Rounding is to four decimal places where needed. A common mistake is to decide the boundary using \(\lambda\) alone, which applies only at the lasso endpoint; here the threshold is \(\lambda\rho\).
Endpoint Calculations with Correlated Predictors
Orthogonality made the preceding calculations separable. With correlated predictors, the endpoint objectives remain lasso and ridge, but the orthogonal coefficient formula does not apply. At the ridge endpoint, differentiating the objective gives the normal equations. If \(X^\mathsf{T}X/n+\lambda I\) is invertible, the solution is
For \(\lambda>0\), the matrix is positive definite when the predictors are represented by the penalized coefficients, so the ridge coefficient solution is unique. The lasso endpoint instead has an absolute-value penalty; its solution must satisfy the corresponding sign and zero-coordinate optimality conditions.
Worked Example: Comparing Endpoints with Correlated Predictors
Suppose a centered, hypothetical two-predictor design has \(X^\mathsf{T}X/n=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}\) and \(X^\mathsf{T}y/n=(1,0.2)^\mathsf{T}\). With \(\lambda=0.5\), find the ridge estimate and verify whether \((0.5,0)^\mathsf{T}\) is a lasso solution.
Ridge method. Add \(\lambda I\) to the Gram matrix to obtain \(\begin{pmatrix}1.5&0.5\\0.5&1.5\end{pmatrix}\), whose determinant is \(1.5(1.5)-0.5(0.5)=2\). Its inverse is \(\frac{1}{2}\begin{pmatrix}1.5&-0.5\\-0.5&1.5\end{pmatrix}\). Multiplying by the response-summary vector gives \(\widehat{\beta}_{\mathrm{ridge}}=(0.5(1.5-0.1),\,0.5(-0.5+0.3))^\mathsf{T}=(0.7,-0.1)^\mathsf{T}\).
Lasso check. For the proposed coefficient vector, the fitted predictor summary is \((X^\mathsf{T}X/n)\beta=(0.5,0.25)^\mathsf{T}\). Subtract the response summary to get \((-0.5,0.05)^\mathsf{T}\). The first coefficient is positive, so its lasso stationarity condition adds \(+\lambda=0.5\): \(-0.5+0.5=0\). The second coefficient is zero; its condition is \(|0.05|\leq0.5\), which holds. Therefore \((0.5,0)^\mathsf{T}\) satisfies the lasso optimality conditions and is a lasso solution.
Check. Here lasso has an exact zero, whereas ridge has two nonzero coefficients. The mistake to avoid is applying the orthogonal soft-thresholding formula to this correlated design. The ridge result comes from the matrix normal equations, and the lasso result is checked using its active- and zero-coefficient conditions.
Common Endpoint Traps
- Reversing the endpoints. Under the definition used here, \(\rho=1\) is lasso and \(\rho=0\) is ridge. Always substitute into the objective. Software can use different parameter labels or conventions.
- Calling ridge sparse. Ridge can have zero coefficients in special data configurations, but its quadratic penalty does not impose the lasso’s positive threshold. It generally produces a dense coefficient vector.
- Confusing \(\rho=0\) with \(\lambda=0\). The first gives ridge when \(\lambda>0\); the second removes both penalty components for every \(\rho\).
- Ignoring objective scaling. A factor such as \(1/(2n)\) in the loss and \(1/2\) in the ridge penalty determines how the numerical value of \(\lambda\) enters the formulas. Compare objectives before comparing penalty values.
- Using orthogonal formulas on correlated predictors. The soft-thresholding expression above requires the stated design conditions. For correlated predictors, solve the appropriate endpoint optimization problem instead.
Check Your Understanding
For each problem, use the objective and conditions stated in the question. Assume the intercept is unpenalized.
- A centered orthogonal design has standardized predictors with \(z=(0.9,-0.3)\). At the lasso endpoint with \(\lambda=0.4\), calculate both coefficients and identify any selected-out predictor. (Answer: \((0.5,0)\); the second predictor is selected out.)
- For the same orthogonal summaries \(z=(0.9,-0.3)\), calculate the ridge coefficients when \(\lambda=0.5\). Give four decimal places. (Answer: \((0.6000,-0.2000)\).)
- For one orthogonal standardized predictor, \(z=0.6\), \(\lambda=0.5\), and \(\rho=0.4\), calculate the elastic net coefficient to four decimal places. (Answer: \(0.3077\).)
- A centered correlated design has \(X^\mathsf{T}X/n=\begin{pmatrix}1&0.25\\0.25&1\end{pmatrix}\), \(X^\mathsf{T}y/n=(0.75,0.5)^\mathsf{T}\), and ridge penalty strength \(\lambda=0.5\). Use the ridge normal equations to calculate both coefficients to four decimal places. (Answer: \((0.4571,0.2571)\).)
- For orthogonal standardized predictors with \(z=(0.8,0.35,-0.6)\), \(\lambda=0.4\), and \(\rho=0.75\), calculate the elastic net coefficient vector to four decimal places. (Answer: \((0.4545,0.0455,-0.2727)\).)