Choosing Between Ridge and Lasso Penalties

Why the Penalty Choice Matters

Ridge and lasso both fit regression models by balancing fit to the observed data against a penalty on coefficient size. Their penalties differ, however, in a consequential way: ridge generally shrinks coefficients without setting them exactly to zero, while lasso can set some coefficients to zero. With strongly correlated predictors, their coefficient patterns can also differ substantially. The appropriate choice depends on whether the main goal is stable prediction, a compact set of predictors, or a balance between the two.

The comparison requires a consistent convention for the tuning parameter. In the formulas below, the response and predictors are centered, the intercept is not penalized, and the objectives use a residual sum of squares divided by $2n$. Predictor scaling must also be considered: coefficient penalties depend on the units of the predictors. Standardizing predictors is common when their scales differ, but the exact normalization matters for calculations.

Definition: For centered response vector $y$, centered design matrix $X$, and coefficient vector $\beta$, the ridge and lasso estimates minimize, respectively, $$ \frac{1}{2n}\|y-X\beta\|_2^2+\frac{\lambda}{2}\|\beta\|_2^2 \qquad\text{and}\qquad \frac{1}{2n}\|y-X\beta\|_2^2+\lambda\|\beta\|_1, $$ where $\lambda\geq 0$, $\|\beta\|_2^2=\sum_j\beta_j^2$, and $\|\beta\|_1=\sum_j|\beta_j|$. The intercept is excluded from both penalties. With this convention, ridge adds $\lambda I$ to the scaled cross-product matrix, while the lasso threshold in an orthogonal, unit-norm design is $\lambda$.

When predictors have different units, a coefficient’s size depends partly on the unit of measurement. For example, recording a predictor in dollars rather than thousands of dollars changes its coefficient even if the underlying information is unchanged. Comparing or penalizing raw coefficient sizes can therefore produce an unintended result. A common approach is to center and scale predictors before fitting, while recording the means and scales so predictions can later be returned to the original units.

How the Coefficients Differ

A clean comparison is available when the predictor columns are mutually orthogonal. Define $q_j=x_j^\mathsf{T}x_j/n$ and $z_j=x_j^\mathsf{T}y/n$, where $x_j$ is column $j$ of $X$. Orthogonality means $x_j^\mathsf{T}x_k/n=0$ for $j\ne k$. If, in addition, $q_j=1$ for every column, the columns have the unit normalization required for the simple formulas below. Standardizing to sample variance one using divisor $n-1$, by itself, does not give $q_j=1$; it gives $q_j=(n-1)/n$.

For an orthogonal design, the objective separates into a sum of one-coefficient problems. For ridge, differentiating the $j$th term gives $q_j\hat{\beta}_j-z_j+\lambda\hat{\beta}_j=0$. For lasso, minimizing the absolute-value-penalized term gives the soft-thresholded estimate. Thus, with general positive $q_j$, the estimates are:

\hat{\beta}^{\text{ridge}}_j=\frac{z_j}{q_j+\lambda}, \qquad \hat{\beta}^{\text{lasso}}_j =\frac{\operatorname{sign}(z_j)(|z_j|-\lambda)_+}{q_j}, \qquad (a)_+=\max(a,0).

In particular, when every $q_j=1$, ridge divides $z_j$ by $1+\lambda$, and lasso sets the coefficient to zero when $|z_j|\leq\lambda$. This thresholding is the source of lasso’s sparsity in this special case. Ridge’s quadratic penalty is smooth and has no corresponding threshold that makes a coefficient exactly zero.

Why the lasso formula holds: For one orthogonal column, the part of the lasso objective that depends on its coefficient is $\frac{q}{2}\beta^2-z\beta+\lambda|\beta|$, where $q=x^\mathsf{T}x/n>0$ and $z=x^\mathsf{T}y/n$. If $\beta>0$, setting the derivative $q\beta-z+\lambda$ to zero gives $\beta=(z-\lambda)/q$, which is positive only if $z>\lambda$. If $\beta<0$, the derivative is $q\beta-z-\lambda$; its zero is $(z+\lambda)/q$, which is negative only if $z<-\lambda$. At $\beta=0$, the lasso condition is $|z|\leq\lambda$. Combining these cases gives the stated soft-threshold formula. This derivation relies on orthogonality to separate the coefficients; it is not a general coefficient formula for correlated predictors.

Worked Example: Comparing Coefficients in an Orthogonal Design

A hypothetical risk model uses three centered, mutually orthogonal predictors. Each column satisfies $x_j^\mathsf{T}x_j/n=1$, and the values $z_j=x_j^\mathsf{T}y/n$ are $2.4$, $0.7$, and $-1.2$. At $\lambda=0.8$, calculate the ridge and lasso coefficients. Identify which method sets any coefficient to zero.

Method. The columns are orthogonal and have the required unit normalization, so use the formulas with $q_j=1$.

For ridge, the denominator is $1+\lambda=1.8$ for each predictor:

\hat{\beta}^{\text{ridge}} =\left(\frac{2.4}{1.8},\frac{0.7}{1.8},\frac{-1.2}{1.8}\right) =(1.3333,\ 0.3889,\ -0.6667).

For lasso, threshold each $z_j$ by $0.8$, retaining its sign:

\hat{\beta}^{\text{lasso}} =\bigl((2.4-0.8)_+,\ (0.7-0.8)_+,\ -(|-1.2|-0.8)_+\bigr) =(1.6,\ 0,\ -0.4).

The second coefficient is zero because $|0.7|\leq0.8$. The first and third coefficients remain nonzero because their absolute $z_j$ values exceed the threshold. Ridge shrinks all three coefficients but leaves all three nonzero. The lasso values have been checked against the threshold condition and the sign of each original $z_j$. Common mistake avoided: treating the lasso as simply dividing every coefficient by a common ridge-style denominator would miss the exact zero.

Correlated Predictors: Shared Signal or Sparse Choice?

With correlated predictors, the separate one-coefficient formulas no longer apply. The penalties still give useful general tendencies. Ridge’s squared penalty makes it costly for any one coefficient to become very large. When predictors carry similar information, ridge often distributes coefficient weight across them. Lasso’s absolute-value penalty can produce a sparse solution, but with highly correlated predictors it may select one predictor and exclude another that carries similar information. Which member is selected can be sensitive to the data. These are tendencies, not guarantees for every dataset or value of $\lambda$.

The distinction is especially clear for two identical predictor columns. Their fitted values depend only on the sum of their coefficients. Ridge prefers an even split of a given sum because that split minimizes the sum of squared coefficients. Lasso has the same absolute penalty for every nonnegative split with the same sum, so it can have multiple solutions. The example below works through both consequences.

Worked Example: Two Identical Predictors

In a hypothetical insurance model, two centered predictors are identical. Each has $x^\mathsf{T}x/n=1$, and $x^\mathsf{T}y/n=2.5$. Use the stated objectives with $\lambda=0.5$. Find the ridge coefficients and the sum of the lasso coefficients; give one lasso solution.

Method. Since the columns are identical, the fitted values depend on $t=\beta_1+\beta_2$. For the part of the squared-error objective that varies with $t$, the contribution is $\frac12(t-2.5)^2$. Components of the response orthogonal to the shared predictor add a constant and do not affect the minimizing coefficients.

For ridge, symmetry and the squared penalty give an equal split, so write $\beta_1=\beta_2=t/2$. The part of the objective depending on $t$ is:

\frac12(t-2.5)^2+\frac{0.5}{2}\left[\left(\frac{t}{2}\right)^2+\left(\frac{t}{2}\right)^2\right] =\frac12(t-2.5)^2+\frac{0.5t^2}{4}.

Differentiating with respect to $t$ gives $t-2.5+0.25t=0$, so $1.25t=2.5$, and $t=2$. Therefore $\beta_1=\beta_2=1$. Directly, either ridge normal equation is $\beta_1+\beta_2+0.5\beta_j=2.5$; substituting $(1,1)$ gives $2+0.5=2.5$.

For lasso, a nonnegative split of $t$ has $|\beta_1|+|\beta_2|=t$. The minimizing sum is found from:

\frac12(t-2.5)^2+0.5t.

Its derivative is $t-2.5+0.5=t-2$, giving $t=2$. Thus the lasso coefficient sum is $2$, and any nonnegative pair summing to $2$ minimizes this part of the objective; for example, $(2,0)$ is a lasso solution. Both methods yield the same fitted contribution, but ridge shares it equally while lasso permits a sparse allocation. Common mistake avoided: $2.5/(1+0.5/2)=2$ is the ridge coefficient sum $t$, not either individual ridge coefficient. The two individual ridge coefficients are $t/2=1$.

Choosing a Penalty for a Modeling Goal

Penalty choice is not a contest in which one method is always superior. The decision should reflect the purpose of the model, the structure of the predictors, and evidence about predictive performance. If prediction is the priority and many predictors may contribute small amounts of information, ridge is a natural candidate: it retains predictors while shrinking their effects. If a compact model is important, lasso is a natural candidate because its fitted coefficient vector can contain exact zeros. When predictors are strongly correlated, ridge is often attractive if the goal is to retain their shared signal; lasso may be attractive if selecting a smaller set is required, but its selection among near-substitutes can be unstable.

A practical comparison fits each method over a range of penalty strengths and evaluates predictive error using the same cross-validation folds. The penalty strength must be tuned for each method; a numerical $\lambda$ does not impose the same amount of shrinkage in ridge and lasso because their penalty functions differ. If the lowest cross-validated error is the sole goal, compare those errors. If a sparse model is also required, state that requirement explicitly: a slightly higher estimated error may be acceptable in exchange for fewer selected predictors. The final choice should reflect the stated objective, not just the method’s label.

Worked Example: Prediction Versus a Sparse Model

A hypothetical pricing team compares ridge and lasso for predicting claim-related costs from 12 standardized rating variables, several of which are strongly correlated. After tuning each method, the team records mean squared prediction errors on the same three validation folds. Ridge gives $16$, $20$, and $18$; lasso gives $17$, $19$, and $21$. The fitted ridge model retains all 12 variables, while the lasso model retains 4. Choose a method for (a) prediction alone and (b) a requirement to use no more than 5 variables.

Method. Average the fold errors separately, then apply the stated modeling goal.

For ridge, the mean error is $(16+20+18)/3=54/3=18$. For lasso, it is $(17+19+21)/3=57/3=19$. Thus ridge has the lower observed mean validation error by $19-18=1$.

(a) If prediction alone is the goal, choose ridge on these results because its mean validation error is lower. Its retention of all 12 variables is not itself a disadvantage under this stated goal. (b) If no more than 5 variables may be used, choose lasso among these candidates: it retains 4, whereas ridge retains 12. The lasso model meets the variable-count requirement, although its mean validation error is higher. Common mistake avoided: choosing lasso just because it is sparse when the stated goal is prediction alone, or choosing ridge for its lower error while ignoring a binding sparsity requirement. These hypothetical fold results are evidence for this comparison, not a guarantee of future performance.

Common Traps and a Decision Checklist

A frequent calculation error is applying the unit-normalization formulas after standardizing predictors in a way that leaves $q_j\ne1$. For instance, suppose an orthogonal column has $q_j=0.8$, $z_j=1.2$, and $\lambda=0.3$. The lasso coefficient is:

\hat{\beta}_j^{\text{lasso}} =\frac{1.2-0.3}{0.8} =1.125.

Using $(|z_j|-\lambda)_+$ without dividing by $q_j$ gives $0.9$, which is incorrect for this normalization. The ridge coefficient is $1.2/(0.8+0.3)=1.0909$, not $1.2/(1+0.3)$. These formulas require the predictor’s actual $q_j$, not an assumption that its squared length equals $n$.

Other important checks are:

Compare like with like. Use the same data splits and prediction metric when comparing tuned methods.
Keep the objective convention straight. Changing the factor attached to the penalty changes how a stated numerical $\lambda$ maps to the formulas.
Do not infer a unique selected predictor from correlation alone. Lasso may select one of several correlated predictors, and the selected member can depend on the data.
Do not claim that ridge performs variable selection. With a positive finite penalty, ridge generally shrinks coefficients rather than setting them exactly to zero.
Match the choice to the goal. Prediction accuracy, a sparse representation, and stable treatment of correlated predictors are distinct considerations.

Key takeaway: Ridge uses a squared penalty and typically shares coefficient weight among correlated predictors; lasso uses an absolute-value penalty and can produce exact zeros. Choose between them by considering the model’s prediction goal, the need for sparsity, predictor correlation, and validation evidence.

Check Your Understanding

Use the objective convention and definitions in this tutorial. Each problem is hypothetical.

Two orthogonal columns have $q_1=q_2=1$, with $z_1=1.4$ and $z_2=-0.5$. At $\lambda=0.6$, calculate both lasso coefficients. (Answer: $(0.8,0)$)
An orthogonal predictor has $q=0.8$, $z=1.2$, and $\lambda=0.3$. Calculate its ridge and lasso coefficients. (Answer: Ridge $1.0909$; lasso $1.125$)
Two identical centered predictors each have $x^\mathsf{T}x/n=1$ and $x^\mathsf{T}y/n=1.8$. With $\lambda=0.6$, give the ridge coefficients and the lasso coefficient sum. (Answer: Ridge coefficients $(0.6923,0.6923)$; lasso coefficient sum $1.2$)
On the same three validation folds, a tuned ridge model has errors $12,15,13$, while a tuned lasso model has errors $11,16,14$. Which has the lower mean error, and what are the two means? (Answer: Ridge; ridge mean $13.3333$, lasso mean $13.6667$)
A team must deliver a model using at most 6 predictors. Ridge has the lower cross-validated error but retains 10 predictors; lasso has slightly higher cross-validated error and retains 5. Which meets the stated predictor-count constraint? (Answer: Lasso; it retains 5 predictors and meets the constraint.)

What You'll Learn