Ridge-Penalized Estimation and Its Closed-Form Solution

Why Penalize a Least-Squares Fit?

Least squares chooses coefficients to make fitted values close to the observed response. When columns of the design matrix are linearly dependent, however, distinct coefficient vectors can produce exactly the same fitted values. The least-squares objective then cannot distinguish among those coefficient vectors, so a unique estimate need not exist. Ridge estimation adds a penalty on the size of the coefficient vector. This changes the optimization problem in a simple way: it favors smaller coefficients and, for a positive penalty parameter, makes the minimizer unique even when predictors are collinear.

The focus here is the mathematical objective and its solution. Let $n$ be the number of observations and $p$ the number of coefficients. The response vector $y$ is in $\mathbb{R}^n$, the design matrix $X$ is in $\mathbb{R}^{n\times p}$, and the coefficient vector $\beta$ is in $\mathbb{R}^p$. The penalty parameter $\lambda$ is a scalar.

Definition: For $\lambda\geq 0$, the ridge-penalized least-squares objective is $$ Q_\lambda(\beta)=\lVert y-X\beta\rVert^2+\lambda\lVert\beta\rVert^2. $$ Here $\lVert v\rVert^2=v^\mathsf{T}v$ is the squared Euclidean norm. The ridge estimator is a value of $\beta$ that minimizes $Q_\lambda(\beta)$. When $\lambda>0$, this minimizer exists and is unique.

In this definition, all $p$ coefficients are penalized. If an application uses an unpenalized intercept, its objective must be written with a penalty that excludes that coefficient; the formula and solution below apply to the stated objective, in which every component of $\beta$ is penalized.

Differentiate the Objective

To differentiate, first expand the squared norms using $X\beta\in\mathbb{R}^n$ and $\beta\in\mathbb{R}^p$:

Q_\lambda(\beta) =(y-X\beta)^\mathsf{T}(y-X\beta)+\lambda\beta^\mathsf{T}\beta =y^\mathsf{T}y-2\beta^\mathsf{T}X^\mathsf{T}y +\beta^\mathsf{T}X^\mathsf{T}X\beta+\lambda\beta^\mathsf{T}\beta.

The matrix $X^\mathsf{T}X$ is symmetric, and the differential of a quadratic form $\beta^\mathsf{T}A\beta$, for symmetric $A$, is $2\beta^\mathsf{T}A\,d\beta$. Applying that rule gives a differential that is a row vector multiplying $d\beta$; transposing its coefficient gives the gradient as a column vector.

dQ_\lambda =\left[-2(X^\mathsf{T}y)^\mathsf{T} +2\beta^\mathsf{T}X^\mathsf{T}X +2\lambda\beta^\mathsf{T}\right]d\beta,

\nabla_\beta Q_\lambda(\beta) =2X^\mathsf{T}X\beta-2X^\mathsf{T}y+2\lambda\beta =2\left[(X^\mathsf{T}X+\lambda I_p)\beta-X^\mathsf{T}y\right].

Each term in the gradient is a $p$-vector: $X^\mathsf{T}X$ is $p\times p$, $X^\mathsf{T}y$ is $p\times1$, and $I_p$ is the $p\times p$ identity matrix. The Hessian is constant in $\beta$:

\nabla_\beta^2 Q_\lambda(\beta)=2(X^\mathsf{T}X+\lambda I_p).

Worked Example: One Coefficient

Let $X=(1,1,1)^\mathsf{T}$, $y=(1,2,3)^\mathsf{T}$, and $\lambda=1$. There is a single coefficient $\beta$. The required products are $X^\mathsf{T}X=3$ and $X^\mathsf{T}y=6$, so the gradient is $2(4\beta-6)$. Setting it to zero gives $\hat\beta=6/4=1.5$.

The fitted response is $X\hat\beta=(1.5,1.5,1.5)^\mathsf{T}$, so the residual is $(-0.5,0.5,1.5)^\mathsf{T}$. The objective value is $0.25+0.25+2.25+1(1.5^2)=2.75+2.25=5$. The positive curvature, given here by the Hessian $2(3+1)=8$, confirms that this stationary point is the unique minimum.

Closed-Form Solution and Uniqueness

A stationary point satisfies $\nabla_\beta Q_\lambda(\beta)=0$. Rearranging this first-order condition yields a linear system. The central issue is whether its coefficient matrix can be inverted.

Theorem: Suppose $X\in\mathbb{R}^{n\times p}$, $y\in\mathbb{R}^n$, and $\lambda>0$. Then $Q_\lambda$ has a unique global minimizer, given by $$ \hat\beta_{\mathrm{ridge}} =(X^\mathsf{T}X+\lambda I_p)^{-1}X^\mathsf{T}y. $$ No full-column-rank assumption on $X$ is required.

Proof. For any nonzero $v\in\mathbb{R}^p$,

v^\mathsf{T}(X^\mathsf{T}X+\lambda I_p)v =(Xv)^\mathsf{T}(Xv)+\lambda v^\mathsf{T}v =\lVert Xv\rVert^2+\lambda\lVert v\rVert^2>0.

The first term is nonnegative, while the second is strictly positive because $\lambda>0$ and $v\ne0$. Thus $X^\mathsf{T}X+\lambda I_p$ is positive definite and therefore invertible. The first-order condition has exactly one solution, namely the displayed formula.

To verify that this solution is the global minimizer, write $A=X^\mathsf{T}X+\lambda I_p$ and $c=X^\mathsf{T}y$. The objective is $y^\mathsf{T}y-2\beta^\mathsf{T}c+\beta^\mathsf{T}A\beta$, and the solution satisfies $A\hat\beta_{\mathrm{ridge}}=c$. For any $h\in\mathbb{R}^p$, substitute $\beta=\hat\beta_{\mathrm{ridge}}+h$ and expand. The linear terms in $h$ cancel because $A\hat\beta_{\mathrm{ridge}}=c$, leaving

Q_\lambda(\hat\beta_{\mathrm{ridge}}+h) =Q_\lambda(\hat\beta_{\mathrm{ridge}})+h^\mathsf{T}Ah.

If $h\ne0$, positive definiteness gives $h^\mathsf{T}Ah>0$. Therefore every other coefficient vector has a strictly larger objective value. This proves both global optimality and uniqueness. $\square$

Worked Example: Duplicate Predictors

Consider $X=\begin{pmatrix}1&1\\0&0\\1&1\end{pmatrix}$, $y=(2,0,2)^\mathsf{T}$, and $\lambda=2$. The two columns of $X$ are identical, so they are linearly dependent. Direct multiplication gives $X^\mathsf{T}X=\begin{pmatrix}2&2\\2&2\end{pmatrix}$ and $X^\mathsf{T}y=(4,4)^\mathsf{T}$.

Adding $2I_2$ gives the ridge system $\begin{pmatrix}4&2\\2&4\end{pmatrix}\hat\beta=(4,4)^\mathsf{T}$. The solution is $\hat\beta=(2/3,2/3)^\mathsf{T}$, since each row of the left-hand matrix gives $4(2/3)+2(2/3)=4$. The matrix is invertible despite the duplicated columns: its determinant is $16-4=12$, which is positive.

The fitted values are $X\hat\beta=(4/3,0,4/3)^\mathsf{T}$. The residual sum of squares is $(2/3)^2+0^2+(2/3)^2=8/9$, and the penalty contribution is $2[(2/3)^2+(2/3)^2]=16/9$. Thus $Q_2(\hat\beta)=24/9=8/3$. The penalty selects one coefficient vector and makes it the unique minimizer; the unpenalized fit could not distinguish how to divide the total coefficient between these identical columns.

What Changes When the Penalty Is Zero?

When $\lambda=0$, the objective is ordinary least squares and its Hessian is $2X^\mathsf{T}X$. For any $v$, $v^\mathsf{T}X^\mathsf{T}Xv=\lVert Xv\rVert^2$. Consequently, $X^\mathsf{T}X$ is positive definite exactly when $Xv\ne0$ for every nonzero $v$, which is equivalent to $X$ having full column rank. If $X$ is rank deficient, there is a nonzero $v$ with $Xv=0$. Whenever $\beta$ is a least-squares minimizer, $\beta+tv$ gives the same fitted values for every scalar $t$, so the minimizer is not unique. A positive $\lambda$ removes this flat direction from the objective by adding $\lambda\lVert v\rVert^2$.

Worked Example: Correlated, but Not Identical, Predictors

Take $X=\begin{pmatrix}1&0\\1&1\\0&1\end{pmatrix}$, $y=(3,0,0)^\mathsf{T}$, and $\lambda=1$. The columns overlap in the second row, and $X^\mathsf{T}X=\begin{pmatrix}2&1\\1&2\end{pmatrix}$, $X^\mathsf{T}y=(3,0)^\mathsf{T}$. Therefore the ridge system is

\begin{pmatrix}3&1\\1&3\end{pmatrix}\hat\beta =\begin{pmatrix}3\\0\end{pmatrix}.

The determinant is $9-1=8$, and the inverse is $\frac18\begin{pmatrix}3&-1\\-1&3\end{pmatrix}$. Hence $\hat\beta=(9/8,-3/8)^\mathsf{T}$. To check the calculation, the first row of the system gives $27/8-3/8=3$, and the second gives $9/8-9/8=0$.

The fitted vector is $(9/8,3/4,-3/8)^\mathsf{T}$, so the residual sum of squares is $({-15}/8)^2+(3/4)^2+(-3/8)^2=225/64+36/64+9/64=135/32$. The penalty contribution is $(9/8)^2+(-3/8)^2=90/64=45/32$. Thus $Q_1(\hat\beta)=135/32+45/32=180/32=45/8$.

Interpretation and Common Pitfalls

The added term $\lambda I_p$ changes the curvature of the objective in every coefficient direction. This is why a positive penalty makes the solution unique, including directions in which the original design matrix provides no information. The amount of penalization depends on the numerical scale of the objective and of the predictors, so the same numerical $\lambda$ need not have the same effect under a rescaling of the data. The formula itself does not prescribe how to choose $\lambda$.

Assuming collinearity makes the ridge inverse impossible. It can make $X^\mathsf{T}X$ singular, but for $\lambda>0$, $X^\mathsf{T}X+\lambda I_p$ is positive definite and invertible.
Using the wrong sign in the gradient. The residual term contributes $-2X^\mathsf{T}y$, while the penalty contributes $+2\lambda\beta$. Setting the gradient to zero gives $(X^\mathsf{T}X+\lambda I_p)\beta=X^\mathsf{T}y$.
Claiming uniqueness for $\lambda=0$ without checking rank. The unpenalized solution is unique only when $X$ has full column rank. A positive penalty supplies the missing positive curvature.
Thinking the penalty changes the number of rows or observations. It changes the coefficient-space objective and its $p\times p$ system; $y$ remains an $n$-vector and $X$ remains $n\times p$.

Key takeaway: The ridge gradient is $2[(X^\mathsf{T}X+\lambda I_p)\beta-X^\mathsf{T}y]$. For $\lambda>0$, the matrix in the normal equation is positive definite, so the closed-form solution exists and is unique whether or not the predictors are linearly independent.

Check Your Understanding

Use the objective and conditions stated above to answer these questions.

What are the dimensions of $X^\mathsf{T}X$, $X^\mathsf{T}y$, and the identity matrix used in the ridge system?
Derive the gradient of $Q_\lambda(\beta)$ from its expanded quadratic form.
Why does $v^\mathsf{T}(X^\mathsf{T}X+\lambda I_p)v>0$ hold for every nonzero $v$ when $\lambda>0$?
If two columns of $X$ are identical, what feature of the ridge objective ensures its minimizer is still unique for positive $\lambda$?
For $\lambda=0$, what condition on $X$ guarantees a unique least-squares coefficient vector?