A Penalized Regression Problem-Solving Workshop

Solving Penalized Regression Problems Systematically

A ridge or lasso problem can look like a regression calculation, a scaling question, or a variable-selection question. The reliable first step is to identify the exact objective being minimized. The numerical value assigned to a penalty parameter depends on the scaling of that objective, so a coefficient cannot be calculated correctly from a value of $\lambda$ alone.

This workshop uses the centered linear model with response $y$, predictors $x_1,\ldots,x_p$, and an unpenalized intercept. The response and predictor columns are centered for the coefficient calculations; the intercept is restored afterward when needed. The objective conventions below are specified explicitly and are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: With centered response vector $y$, centered design matrix $X$, and coefficient vector $\beta$, the ridge estimate minimizes $$ \frac{1}{2n}\|y-X\beta\|_2^2+\frac{\lambda}{2}\sum_{j=1}^p\beta_j^2, $$ and the lasso estimate minimizes $$ \frac{1}{2n}\|y-X\beta\|_2^2+\lambda\sum_{j=1}^p|\beta_j|, $$ where $n$ is the number of observations and $\lambda\geq 0$. The intercept is not included in either penalty. These objectives use the factor $1/(2n)$ for the squared-error term; a different convention changes the numerical scaling of $\lambda$.

Start With the Objective and the Scale

The two penalties have different consequences. Ridge adds a squared-coefficient penalty, which generally makes estimates smaller without setting them exactly to zero. Lasso adds an absolute-value penalty, which can set estimates exactly to zero. A problem may state either objective directly or describe a software convention; translate that convention before calculating.

Predictors should be put on comparable scales before applying a shared penalty when their units or dispersions differ. For predictor $j$, let $\bar{x}_j$ be its sample mean and let $s_j>0$ be its stated scaling factor. Define $z_{ij}=(x_{ij}-\bar{x}_j)/s_j$. The penalty then acts on coefficients for these scaled predictors. If the fitted coefficient on $z_j$ is $\hat{\theta}_j$, the coefficient for the original predictor is $\hat{\beta}_j=\hat{\theta}_j/s_j$. With an unpenalized intercept, its fitted value on the original scale is $\hat{\beta}_0=\bar{y}-\sum_j\hat{\beta}_j\bar{x}_j$.

The precise definition of $s_j$ matters. Some calculations use the sample standard deviation, while others specify a root-mean-square centered scale or provide already-standardized columns. Use the value and convention given in the problem rather than silently switching between them. If the response is also standardized using scale $s_y$, the original response-scale slope is $s_y\hat{\theta}_j/s_j$, and the intercept must also be returned to the response scale.

Worked Example: Ridge and Lasso in an Orthogonal Design

There are $n=4$ observations and three centered, standardized predictor columns: $x_1=(1,1,-1,-1)$, $x_2=(1,-1,1,-1)$, and $x_3=(1,-1,-1,1)$. Each column has mean square 1, and the columns are mutually orthogonal. The response is $y=(3.1,0.9,-1.5,-2.5)$. Using $\lambda=0.5$ and the objective conventions above, calculate the ridge and lasso coefficients. State which estimate is sparse.

Method: calculate the predictor-response inner products, then use the orthogonal-design formulas. The response mean is $(3.1+0.9-1.5-2.5)/4=0$, consistent with the centered calculation. The normalized inner products are $$ \frac{x_1^\mathsf{T}y}{4}=\frac{3.1+0.9+1.5+2.5}{4}=2,\quad \frac{x_2^\mathsf{T}y}{4}=\frac{3.1-0.9-1.5+2.5}{4}=0.8, $$ $$ \frac{x_3^\mathsf{T}y}{4}=\frac{3.1-0.9+1.5-2.5}{4}=0.3. $$ Thus the unpenalized coefficients in this orthogonal design are $2,0.8,0.3$.

For ridge, each coefficient is divided by $1+\lambda=1.5$. The estimates are $2/1.5=1.3333$, $0.8/1.5=0.5333$, and $0.3/1.5=0.2000$. None is zero.

For lasso, apply soft thresholding at $\lambda=0.5$: subtract 0.5 from a positive coefficient if it exceeds 0.5, set a coefficient with absolute value at most 0.5 to zero, and apply the corresponding rule to negative values. The estimates are $2-0.5=1.5$, $0.8-0.5=0.3$, and $0$, respectively. The lasso estimate is sparse because its third coefficient is exactly zero; the ridge estimate is not. This comparison uses the same $\lambda$ but different penalty forms, so it is not a claim that the two objectives have equal fitted loss.

Check: The fitted lasso coefficients are obtained by reducing the first two positive correlations and dropping the third, whose magnitude $0.3$ is below the threshold $0.5$. The ridge coefficients are each smaller than their unpenalized counterparts, as expected.

Common mistake avoided: Applying the lasso threshold to ridge coefficients. Ridge divides by $1+\lambda$ here; it does not use the lasso zeroing rule.

Why the Orthogonal-Design Formulas Work

The preceding calculation follows by separating the objective into one coefficient at a time. Suppose the centered predictor columns satisfy $X^\mathsf{T}X/n=I$, the identity matrix, and define $c_j=x_j^\mathsf{T}y/n$. Expanding the squared-error term gives a constant independent of $\beta$, plus $\frac12\sum_j(\beta_j-c_j)^2$. The ridge penalty therefore adds a quadratic term for each coordinate, while the lasso penalty adds an absolute-value term for each coordinate.

\begin{aligned} Q_{\text{ridge},j}(\beta_j) &=\frac12(\beta_j-c_j)^2+\frac{\lambda}{2}\beta_j^2,\\ \frac{dQ_{\text{ridge},j}}{d\beta_j} &=(\beta_j-c_j)+\lambda\beta_j=0,\\ \hat{\beta}_{j,\text{ridge}}&=\frac{c_j}{1+\lambda}. \end{aligned}

For lasso, the objective for one coordinate is $\frac12(\beta_j-c_j)^2+\lambda|\beta_j|$. When $\beta_j>0$, its derivative is $\beta_j-c_j+\lambda$, so a positive solution is $\beta_j=c_j-\lambda$, valid when $c_j>\lambda$. When $\beta_j<0$, the derivative is $\beta_j-c_j-\lambda$, so a negative solution is $\beta_j=c_j+\lambda$, valid when $c_j<-\lambda$. At zero, the minimum occurs when $|c_j|\leq\lambda$. Together these cases give the soft-threshold rule:

\hat{\beta}_{j,\text{lasso}} =\operatorname{sign}(c_j)(|c_j|-\lambda)_+, \qquad (a)_+=\max(a,0).

These simple coordinate formulas require the stated orthogonality and normalization. For a nonorthogonal design, ridge coefficients are found from the full system $(X^\mathsf{T}X/n+\lambda I)\hat{\beta}=X^\mathsf{T}y/n$. Lasso estimates generally require solving the joint optimization problem; thresholding the ordinary least-squares coefficients one by one is not generally valid.

Standardize, Solve, and Return to Original Units

Scaling is part of the calculation, not a cosmetic reporting step. A shared penalty on unscaled coefficients can penalize a one-unit change in one predictor differently from a one-unit change in another, simply because the predictors use different units. After fitting, translating the coefficients back is essential for interpretation and prediction using original measurements.

Worked Example: Lasso After Predictor Scaling

A hypothetical pricing analysis has four observations. The predictors have means $\bar{x}_1=40$, $\bar{x}_2=5$ and specified centered root-mean-square scales $s_1=8$, $s_2=2$. Their standardized columns are $z_1=(1,1,-1,-1)$ and $z_2=(1,-1,1,-1)$. The response values are $y=(113,111,81,95)$, in dollars. Fit lasso with $\lambda=2$ using the stated standardized predictors, then recover the original-scale equation and predict at $x_1=48,x_2=4$.

Method: calculate centered correlations, threshold, and undo the predictor scaling. The response mean is $\bar{y}=(113+111+81+95)/4=100$. Centering gives $y-\bar y=(13,11,-19,-5)$. The standardized predictor correlations are $$ \frac{z_1^\mathsf{T}(y-\bar y)}{4} =\frac{13+11+19+5}{4}=12,\qquad \frac{z_2^\mathsf{T}(y-\bar y)}{4} =\frac{13-11-19+5}{4}=-3. $$

The columns are orthogonal and each has mean square 1. Soft thresholding at 2 gives $\hat{\theta}_1=12-2=10$ and $\hat{\theta}_2=-3+2=-1$. These are coefficients for $z_1,z_2$, not for the original predictors.

Return to the original predictor scales: $\hat{\beta}_1=10/8=1.25$ dollars per unit of $x_1$, and $\hat{\beta}_2=-1/2=-0.5$ dollars per unit of $x_2$. The intercept is $$ \hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}_1-\hat{\beta}_2\bar{x}_2 =100-(1.25)(40)-(-0.5)(5)=52.5. $$ The original-scale equation is $\hat{y}=52.5+1.25x_1-0.5x_2$. At $x_1=48,x_2=4$, the prediction is $52.5+1.25(48)-0.5(4)=52.5+60-2=110.5$ dollars.

Check: The same prediction follows on the standardized scale. At the requested values, $z_1=(48-40)/8=1$ and $z_2=(4-5)/2=-0.5$; hence $\hat{y}=100+10(1)-1(-0.5)=110.5$ dollars.

Common mistake avoided: Reporting $10$ and $-1$ as slopes per original predictor unit. They are slopes for scaled predictors; dividing by the stated scales and restoring the intercept gives the correct original-scale model.

Correlated Predictors: Solve Jointly and Check Conditions

With correlated predictors, the estimate for one variable depends on the others. Ridge uses the full correlation structure and generally distributes coefficient weight across correlated predictors. Lasso can set one or more coefficients to zero, so among strongly related predictors it may retain one while assigning little or no coefficient to another. Which predictor is retained can depend on the data and the precise relationships; a zero coefficient is not, by itself, proof that a predictor has no relationship with the response.

For lasso, a proposed solution should satisfy both the equations for nonzero coefficients and the conditions for coefficients set to zero. In the objective used here, if $\hat{\beta}_j\ne0$, the condition is $x_j^\mathsf{T}(y-X\hat{\beta})/n=\lambda\operatorname{sign}(\hat{\beta}_j)$. If $\hat{\beta}_j=0$, the condition is $|x_j^\mathsf{T}(y-X\hat{\beta})/n|\leq\lambda$. These conditions guard against selecting a coefficient as zero when its remaining correlation exceeds the penalty threshold.

Worked Example: Comparing Ridge and Lasso With Correlated Predictors

Two centered, standardized predictors have correlation $0.8$, so their normalized Gram matrix is $G=\begin{pmatrix}1&0.8\\0.8&1\end{pmatrix}$. Their normalized correlations with the centered response are $c=(1,0.7)^\mathsf{T}$. For $\lambda=0.2$, calculate the ridge coefficients and determine whether lasso can have $(0.8,0)$ as its solution.

Method: solve the ridge normal equations and check both lasso conditions. For ridge, $G+\lambda I=\begin{pmatrix}1.2&0.8\\0.8&1.2\end{pmatrix}$. Its determinant is $1.2(1.2)-0.8(0.8)=1.44-0.64=0.8$. Thus $$ \hat{\beta}_{\text{ridge}} =\frac{1}{0.8} \begin{pmatrix}1.2&-0.8\\-0.8&1.2\end{pmatrix} \begin{pmatrix}1\\0.7\end{pmatrix} =\frac{1}{0.8} \begin{pmatrix}1.2-0.56\\-0.8+0.84\end{pmatrix} =\begin{pmatrix}0.8\\0.05\end{pmatrix}. $$ Multiplication checks the result: $1.2(0.8)+0.8(0.05)=1$, and $0.8(0.8)+1.2(0.05)=0.7$.

For the proposed lasso solution $(0.8,0)$, the first coefficient is positive. Its active-variable condition is $c_1-(G\hat{\beta})_1=1-0.8=0.2=\lambda$, so that condition holds. The residual correlation for the second predictor is $c_2-(G\hat{\beta})_2=0.7-0.8(0.8)=0.06$. Since $|0.06|\leq0.2$, the zero-coefficient condition also holds. Therefore $(0.8,0)$ satisfies the lasso conditions.

Check: Ridge retains a small second coefficient, $0.05$; lasso sets that coefficient to zero while keeping the first at $0.8$. The correlated design explains why the methods need not make the same allocation of coefficients.

Common mistake avoided: Checking only the equation for lasso’s nonzero coefficient. A proposed zero coefficient also needs its residual correlation to be no greater than $\lambda$ in absolute value.

A Practical Exam Workflow and Common Traps

A disciplined sequence reduces errors when a question combines model interpretation, arithmetic, and scaling.

Write the objective.
Identify the squared-error multiplier, penalty type, value of $\lambda$, and whether the intercept is penalized. Do not assume a penalty convention from memory.

Check centering and scaling.
Record which predictors are standardized, the means and scales used, and whether the response is also standardized. Use the problem’s convention consistently.

Choose the calculation that applies.
Use coordinate formulas for an orthogonal normalized design. For correlated predictors, use the ridge system or solve and verify the lasso conditions; do not apply orthogonal formulas without their assumptions.

Interpret and verify.
Check which coefficients are zero, whether the estimated magnitudes are consistent with shrinkage, and whether the final coefficients and intercept are on the requested scale.

A frequent exam trap is to treat every $\lambda$ as if it had the same numerical meaning. For example, if an objective is written as $\|y-X\beta\|_2^2+\lambda\|\beta\|_2^2$, its penalty parameter is not on the same scale as the $\lambda$ in $\|y-X\beta\|_2^2/(2n)+\lambda\|\beta\|_2^2/2$. Multiplying the second objective by $2n$ gives the first form with penalty coefficient $n\lambda$. Use the objective as written before calculating.

Another trap is to infer that ridge performs variable selection because its coefficients are small. Under a finite penalty, ridge generally shrinks but does not set coefficients exactly to zero. Conversely, lasso sparsity is a feature of its estimate, not a guarantee that the selected variables are uniquely or causally important. Correlation can make coefficient allocation and selection sensitive to the observed data.

Key takeaway: Identify the precise objective, respect the predictor scaling, and use formulas only under their stated design assumptions. Ridge shrinks through a quadratic penalty; lasso can produce exact zeros through an absolute-value penalty. With correlated predictors, solve jointly and check the lasso conditions for both active and zero coefficients.

Check Your Understanding

For each problem, state the calculation that supports your answer and check that the coefficient scale and penalty convention match the information given.

Two orthogonal standardized predictors have normalized response correlations $1.4$ and $0.5$. Under the objectives defined above and $\lambda=0.4$, find the ridge and lasso coefficients. (Answer: Ridge $(1,0.3571)$; lasso $(1,0.1)$.)
A fitted model uses a standardized predictor $z=(x-20)/4$, an unpenalized intercept, response mean $50$, and estimated coefficient $6$ on $z$. Find the original-scale intercept and slope, then predict at $x=24$. (Answer: Intercept $20$, slope $1.5$, prediction $56$.)
Two standardized predictors have correlation $0.5$, normalized response correlations $c=(0.9,0.6)^\mathsf{T}$, and lasso penalty $\lambda=0.2$. Find the solution assuming both coefficients are positive, and check whether setting the second coefficient to zero would satisfy its condition. (Answer: $(2/3,1/15)$; the one-active-coefficient candidate $(0.7,0)$ fails because the second residual correlation is $0.25>0.2$.)
For ridge, let $G=\begin{pmatrix}1&-0.6\\-0.6&1\end{pmatrix}$, $c=(0.8,-0.2)^\mathsf{T}$, and $\lambda=0.4$. Solve $(G+\lambda I)\hat{\beta}=c$. (Answer: $(0.625,0.125)$.)

What You'll Learn