A Worked Ridge Regression Calculation

From the Ridge Objective to a Numerical Solution

Ridge regression is most useful to understand by calculating it: form the predictor cross-product matrix, add the penalty to its diagonal, and solve the resulting linear system. The penalty changes the coefficient estimates and usually changes the fitted values as well. A reliable calculation therefore states the exact objective being minimized and compares its results with ordinary least squares.

Definition: For a response vector $y$, a design matrix $X$ whose columns are centered predictors, and a penalty parameter $\lambda \geq 0$, the ridge coefficient estimate minimizes $$ \operatorname{RSS}(\boldsymbol{\beta})+\lambda\boldsymbol{\beta}^{T}\boldsymbol{\beta} =(y-X\boldsymbol{\beta})^{T}(y-X\boldsymbol{\beta})+\lambda\boldsymbol{\beta}^{T}\boldsymbol{\beta}. $$ The intercept is not included in the penalty. With centered predictors and a centered response, the intercept is the response mean and can be treated separately. This tutorial uses the displayed objective; other conventions, such as dividing the RSS by the sample size, require a correspondingly different interpretation of $\lambda$.

The ridge normal equations and solution are

(X^{T}X+\lambda I)\widehat{\boldsymbol{\beta}}_{\lambda}=X^{T}y, \qquad \widehat{\boldsymbol{\beta}}_{\lambda} =(X^{T}X+\lambda I)^{-1}X^{T}y.

For $\lambda>0$, the matrix $X^{T}X+\lambda I$ is invertible, including when the predictor columns are linearly dependent. At $\lambda=0$, the equations reduce to the least-squares normal equations; an inverse formula applies only when $X^{T}X$ is invertible. The arithmetic below uses the normal equations directly, which is often clearer for a small problem.

Why Adding the Penalty Changes the Equations

The formula follows by expanding the objective and differentiating with respect to the coefficient vector. Since $X^{T}X$ is symmetric, the derivative of the quadratic least-squares term is $-2X^{T}y+2X^{T}X\boldsymbol{\beta}$. The derivative of $\lambda\boldsymbol{\beta}^{T}\boldsymbol{\beta}$ is $2\lambda\boldsymbol{\beta}$. At a minimum, the gradient is zero:

-2X^{T}y+2X^{T}X\boldsymbol{\beta}+2\lambda\boldsymbol{\beta}=0.

Dividing by 2 and collecting the terms involving $\boldsymbol{\beta}$ gives

(X^{T}X+\lambda I)\boldsymbol{\beta}=X^{T}y.

When $\lambda>0$, for any nonzero vector $\mathbf{a}$, $ \mathbf{a}^{T}(X^{T}X+\lambda I)\mathbf{a} =\|X\mathbf{a}\|^{2}+\lambda\|\mathbf{a}\|^{2}>0 $. Thus the matrix is positive definite and has a unique solution.

Worked Example: Two Correlated Predictors at Two Penalties

A hypothetical insurer models a centered claim-cost measure using two centered rating variables. The four observations are shown below. Calculate the least-squares coefficients and the ridge coefficients for $\lambda=4$ and $\lambda=12$, then compare fitted values and residual sums of squares.

Observation	$x_1$	$x_2$	$y$
1	-1	-2	-3
2	-1	0	-1
3	1	0	1
4	1	2	3

Step 1: Calculate the cross-products. Both predictors and the response sum to zero, so they are centered. The entries of $X^{T}X$ are $x_1^{T}x_1=1+1+1+1=4$, $x_1^{T}x_2=2+0+0+2=4$, and $x_2^{T}x_2=4+0+0+4=8$. Also, $x_1^{T}y=3+1+1+3=8$ and $x_2^{T}y=6+0+0+6=12$. Therefore,

X^{T}X= \begin{pmatrix}4&4\\4&8\end{pmatrix}, \qquad X^{T}y= \begin{pmatrix}8\\12\end{pmatrix}.

Step 2: Solve for least squares. At $\lambda=0$, the equations are $4\beta_1+4\beta_2=8$ and $4\beta_1+8\beta_2=12$. Subtracting the first equation from the second gives $4\beta_2=4$, so $\widehat{\beta}_2=1$ and $\widehat{\beta}_1=1$.

Step 3: Solve at $\lambda=4$. Add 4 to each diagonal entry, giving $ \begin{pmatrix}8&4\\4&12\end{pmatrix}\boldsymbol{\beta} =\begin{pmatrix}8\\12\end{pmatrix} $. The determinant is $8(12)-4(4)=80$, so

\widehat{\boldsymbol{\beta}}_{4} =\frac{1}{80} \begin{pmatrix}12&-4\\-4&8\end{pmatrix} \begin{pmatrix}8\\12\end{pmatrix} = \begin{pmatrix}(96-48)/80\\(-32+96)/80\end{pmatrix} = \begin{pmatrix}0.6\\0.8\end{pmatrix}.

Step 4: Solve at $\lambda=12$. The penalized matrix is $ \begin{pmatrix}16&4\\4&20\end{pmatrix} $, with determinant $16(20)-4(4)=304$. Hence

\widehat{\boldsymbol{\beta}}_{12} =\frac{1}{304} \begin{pmatrix}20&-4\\-4&16\end{pmatrix} \begin{pmatrix}8\\12\end{pmatrix} = \begin{pmatrix}192/304\\64/304\end{pmatrix} = \begin{pmatrix}12/19\\4/19\end{pmatrix}.

Step 5: Compare fitted values and check the calculations. The least-squares fitted values are $x_1+x_2=(-3,-1,1,3)$, exactly the observed response, so its RSS is 0. At $\lambda=4$, the fitted values are $(-2.2,-1.4,1.4,2.2)$; the residuals are $(-0.8,0.4,-0.4,0.8)$, and the RSS is $0.64+0.16+0.16+0.64=1.60$. At $\lambda=12$, the fitted values are $(-20/19,-12/19,12/19,20/19)$. The residuals are $(-37/19,-7/19,7/19,37/19)$, giving RSS $2(37^2+7^2)/19^2=2836/361$, approximately $7.856$. The increased RSS is consistent with the fit moving away from the exact least-squares fit.

Method and mistake avoided. The method is to add $\lambda$ to the diagonal of $X^{T}X$, not to its off-diagonal entries. Notice also that $\widehat{\beta}_1$ rises from $0.6$ at $\lambda=4$ to $12/19$, approximately $0.632$, at $\lambda=12$. Ridge penalizes the coefficient vector as a whole; an individual coefficient need not decrease monotonically as $\lambda$ increases.

Orthogonal Predictors: A Direct Check

When centered predictor columns are orthogonal, $X^{T}X$ is diagonal. The ridge equations then separate into one calculation per predictor: if $x_j^{T}x_j=c_j$, then $ \widehat{\beta}_{j,\lambda}=x_j^{T}y/(c_j+\lambda) $. This is a convenient check on the matrix method, not a different definition of ridge regression.

Worked Example: Calculate Coefficients and Fitted Values with Orthogonal Columns

A hypothetical pricing analysis uses the centered predictor columns $x_1=(1,-1,0)^{T}$ and $x_2=(1,1,-2)^{T}$, with centered response $y=(1,-3,2)^{T}$. Calculate the coefficients for $\lambda=0$ and $\lambda=2$, and compare the fitted values.

Step 1: Verify the cross-products. The columns are centered because each sums to zero. Their inner product is $1(1)+(-1)(1)+0(-2)=0$, and their squared lengths are $2$ and $6$. The response cross-products are $x_1^{T}y=1+3=4$ and $x_2^{T}y=1-3-4=-6$. Thus $X^{T}X=\begin{pmatrix}2&0\\0&6\end{pmatrix}$ and $X^{T}y=(4,-6)^{T}$.

Step 2: Calculate both coefficient vectors. At $\lambda=0$, the coefficients are $4/2=2$ and $-6/6=-1$. At $\lambda=2$, they are $4/(2+2)=1$ and $-6/(6+2)=-3/4$.

Step 3: Calculate fitted values and RSS. Least squares gives $2x_1-x_2=(1,-3,2)^{T}=y$, so RSS is 0. Ridge gives $x_1-(3/4)x_2=(1/4,-7/4,3/2)^{T}$. Subtracting these fitted values from $y$ gives residuals $(3/4,-5/4,1/2)^{T}$. The RSS is $9/16+25/16+1/4=38/16=19/8=2.375$. The fitted values and RSS follow directly from the stated data, providing a check on the coefficient calculation.

Method and mistake avoided. Orthogonality makes each coefficient calculation independent, but the denominator is the predictor’s squared length plus $\lambda$. Using the same shrinkage denominator for both predictors would be incorrect because their squared lengths, 2 and 6, differ.

Keeping an Unpenalized Intercept

When predictors are centered but the response is not, the intercept is the response mean. The ridge equations apply to the centered response $y_i-\bar y$, while the intercept remains $ \widehat{\beta}_0=\bar y $. For a single predictor recorded on its original scale, the equivalent calculation uses $ \widehat{\beta}_1=x_c^{T}y_c/(x_c^{T}x_c+\lambda) $, where $x_c$ and $y_c$ are the centered predictor and response. The intercept is then recovered as $ \widehat{\beta}_0=\bar y-\widehat{\beta}_1\bar x $.

Worked Example: Recover the Intercept After Centering

For a hypothetical portfolio study, five records have predictor values $x=(1,2,3,4,5)$ and response values $y=(3,5,7,9,11)$. Use the ridge objective with $\lambda=5$, penalizing the slope but not the intercept. Calculate the fitted line and compare its slope with least squares.

Step 1: Find the means and centered cross-products. The means are $\bar x=15/5=3$ and $\bar y=35/5=7$. The centered predictor is $(-2,-1,0,1,2)$, and the centered response is $(-4,-2,0,2,4)$. Therefore $x_c^{T}x_c=4+1+0+1+4=10$, and $x_c^{T}y_c=8+2+0+2+8=20$.

Step 2: Calculate the ridge slope and intercept. The ridge slope is $20/(10+5)=4/3$. Since the intercept is not penalized, it is $\widehat{\beta}_0=7-(4/3)(3)=3$. The fitted line is $\widehat y=3+(4/3)x_c$, or equivalently $\widehat y=3+(4/3)(x-3)$.

Step 3: Compare with least squares and check the fit. At $\lambda=0$, the slope is $20/10=2$ and the intercept is $7-2(3)=1$, giving fitted values $(3,5,7,9,11)$, exactly the response. At $\lambda=5$, the fitted values are $(1/3,5/3,3,13/3,17/3)$. The residuals are $(8/3,10/3,4,14/3,16/3)$, or, equivalently, $(-4/3,-2/3,0,2/3,4/3)$ when calculated as observed minus fitted. Their squared sum is $(16+4+0+4+16)/9=40/9$. The mean fitted value is $3$, matching $\bar y$, as expected with an unpenalized intercept.

Method and mistake avoided. Center the variables to calculate the penalized slope, then recover the intercept from the means. Penalizing the intercept would not produce the stated ridge model and would generally fail the check that the fitted mean equals the response mean.

Calculation Checks and Common Traps

A numerical ridge answer should be checked against the objective and the data, not just against a matrix inverse. The following checks catch frequent errors:

State the penalty convention. For this tutorial, the objective is RSS plus $\lambda\sum_j\beta_j^2$. If the objective instead uses RSS divided by $n$, the numerical value of the penalty must be interpreted under that convention.
Penalize slopes, not the intercept. With centered predictors, keep the intercept at $\bar y$. With uncentered predictors, center first or explicitly use the unpenalized-intercept equations.
Add the penalty to the diagonal. The off-diagonal cross-products remain unchanged; the diagonal entries each increase by $\lambda$.
Check the limiting case. Setting $\lambda=0$ must recover the least-squares normal equations. For positive $\lambda$, compare fitted values and RSS with the data to catch arithmetic or sign errors.
Do not infer componentwise monotonicity. The penalty acts on the full coefficient vector. Correlation among predictors can make one coefficient rise even as the overall penalty increases.

Key takeaway: A ridge calculation is determined by the stated objective, the cross-products $X^{T}X$ and $X^{T}y$, and the treatment of the intercept. Add $\lambda I$ to the predictor cross-product matrix, solve the system, and verify the resulting fit against the original observations.

Check Your Understanding

For each problem, use the objective RSS plus $\lambda$ times the sum of squared slopes, with an intercept unpenalized where relevant.

A centered two-predictor design has $X^{T}X=\begin{pmatrix}4&0\\0&9\end{pmatrix}$ and $X^{T}y=(8,-9)^{T}$. Find the ridge coefficients for $\lambda=3$. (Answer: $(8/7,-3/4)^{T}$.)
A centered design has $X^{T}X=\begin{pmatrix}5&2\\2&5\end{pmatrix}$, $X^{T}y=(9,6)^{T}$, and $\lambda=1$. Solve the two ridge normal equations. (Answer: $(21/16,9/16)^{T}$.)
A centered single predictor has $x^{T}x=12$, $x^{T}y=18$, and the uncentered response mean is 4. For $\lambda=6$, give the slope, intercept, and fitted value at centered predictor value 2. (Answer: slope $1$, intercept $4$, fitted value $6$.)
A centered design has $X^{T}X=\begin{pmatrix}4&2\\2&4\end{pmatrix}$ and $X^{T}y=(8,8)^{T}$. Find the coefficient vector at $\lambda=0$ and at $\lambda=2$. (Answer: at $\lambda=0$, $(4/3,4/3)^{T}$; at $\lambda=2$, $(1,1)^{T}$.)
Let $x_1=(1,-1,0)^{T}$, $x_2=(1,1,-2)^{T}$, and $y=(1,-3,2)^{T}$. For $\lambda=2$, find the fitted values and RSS. (Answer: fitted values $(1/4,-7/4,3/2)^{T}$; RSS $19/8=2.375$.)
A one-predictor study has $\bar x=3$, $\bar y=7$, centered cross-product $x_c^{T}y_c=20$, and $x_c^{T}x_c=10$. With $\lambda=5$, find the slope and intercept, without penalizing the intercept. (Answer: slope $4/3$, intercept $3$.)

Observation	\(x_1\)	\(x_2\)	\(y\)
1	-1	-2	-3
2	-1	0	-1
3	1	0	1
4	1	2	3

What You'll Learn