The Ridge Regression Solution

Why Ridge Has a Closed-Form Solution

In multiple regression, highly correlated predictors can make the ordinary least-squares coefficient estimates unstable or even nonunique. Ridge regression changes the estimation problem by adding a penalty on the squared coefficients. That penalty does more than discourage large estimates: when its tuning parameter is positive, it makes the matrix in the coefficient equations invertible, including when the predictors are linearly dependent.

The derivation below concerns the ridge coefficients, not a method for choosing the penalty parameter. We use a common convention in which the objective is residual sum of squares plus the penalty. Other conventions can multiply the residual sum of squares by a constant, which changes the numerical scale assigned to the tuning parameter.

Definition: Let $y$ be an $n$-vector of responses and let $X$ be an $n\times p$ matrix of predictor values. For centered responses and predictor columns, the ridge estimator minimizes $$ Q(\boldsymbol{\beta})=(y-X\boldsymbol{\beta})'(y-X\boldsymbol{\beta})+\lambda\boldsymbol{\beta}'\boldsymbol{\beta}, $$ where $\boldsymbol{\beta}$ is the $p$-vector of coefficients and $\lambda\geq 0$ is the ridge penalty parameter. The intercept is not included in $\boldsymbol{\beta}$ and is not penalized.

Centering means subtracting each variable’s sample mean from its observed values. With an intercept in the original regression, this separates the intercept from the slope calculation: after fitting the ridge slopes to centered data, the intercept is the response mean minus the predictor means multiplied by their estimated slopes. In particular, if all predictors have been centered, the intercept estimate is the response mean. This is consistent with treating the intercept separately rather than adding it to the penalized coefficient vector.

The algebraic estimator requires no probability distribution for the errors. A sampling model, such as independent errors with mean zero and constant variance, can be used for statistical interpretation, but it is not needed to derive the minimizer of this objective. The examples are hypothetical regression calculations; no interest rate, mortality assumption, or error distribution is involved.

Deriving the Ridge Normal Equations

Expand the squared residual term and differentiate with respect to the coefficient vector. The identity matrix appears because the penalty is the sum of the squared coefficients, $\boldsymbol{\beta}'\boldsymbol{\beta}$.

Expand the objective.
$Q(\boldsymbol{\beta})=y'y-2\boldsymbol{\beta}'X'y+\boldsymbol{\beta}'X'X\boldsymbol{\beta}+\lambda\boldsymbol{\beta}'\boldsymbol{\beta}$.

Differentiate.
The gradient is $-2X'y+2X'X\boldsymbol{\beta}+2\lambda\boldsymbol{\beta}$. The last term is $2\lambda I_p\boldsymbol{\beta}$, since $I_p\boldsymbol{\beta}=\boldsymbol{\beta}$.

Set the gradient equal to zero.
$(X'X+\lambda I_p)\boldsymbol{\beta}=X'y$.

Solve when the matrix is invertible.
For $\lambda>0$, the solution is $\hat{\boldsymbol{\beta}}_{\mathrm{ridge}}=(X'X+\lambda I_p)^{-1}X'y$.

Why the identity matrix matters: $X'X$ and $I_p$ are both $p\times p$, so they can be added. The identity matrix adds $\lambda$ to each diagonal entry of $X'X$ and leaves its off-diagonal entries unchanged. Thus the penalty contributes to every coefficient’s squared magnitude, not to pairwise products between different coefficients.

The displayed inverse formula is valid for $\lambda>0$. At $\lambda=0$, it reduces to the ordinary least-squares normal equations, and an inverse exists only if $X$ has full column rank. If there are more predictors than independent directions in the data, ordinary least squares may have no unique coefficient vector. A positive ridge penalty avoids that requirement.

Why a Positive Penalty Guarantees a Unique Estimate

The matrix $X'X$ is positive semidefinite: for any $p$-vector $v$, $v'X'Xv=(Xv)'(Xv)$, which is a squared length and therefore cannot be negative. Adding $\lambda I_p$ with $\lambda>0$ makes the resulting matrix positive definite.

v'(X'X+\lambda I_p)v =(Xv)'(Xv)+\lambda v'v =\lVert Xv\rVert^2+\lambda\lVert v\rVert^2>0 \quad\text{for every }v\ne 0.

The strict inequality holds because $v\ne 0$ implies $\lVert v\rVert^2>0$, while $\lambda>0$; the first term is nonnegative. A positive-definite matrix has no nonzero vector in its null space, so it is invertible. The objective is also strictly convex: its Hessian is $2(X'X+\lambda I_p)$, which is positive definite. Therefore the solution is not merely a stationary point; it is the unique global minimizer.

This argument does not require the columns of $X$ to be independent. If two predictors are identical, for example, the corresponding columns of $X$ are linearly dependent and $X'X$ is singular. But the positive penalty still makes $X'X+\lambda I_p$ positive definite. If $\lambda=0$, this guarantee disappears.

Worked Ridge Calculations

Worked Example: Perfectly Collinear Predictors

A hypothetical insurer models a centered claim-cost response using two centered rating variables with identical observed values. The cross-products are $X'X=\begin{pmatrix}4&4\\4&4\end{pmatrix}$ and $X'y=\begin{pmatrix}12\\12\end{pmatrix}$. Use $\lambda=2$ to find the ridge coefficients and check the normal equations. No interest rate, mortality assumption, or error distribution is involved.

Method. Add $\lambda I_2$ to $X'X$, solve the resulting two equations, and verify that the solution satisfies them. The columns of $X$ are identical, so $X'X$ is singular before the penalty is added.

First, $$ X'X+2I_2= \begin{pmatrix}4&4\\4&4\end{pmatrix} +\begin{pmatrix}2&0\\0&2\end{pmatrix} =\begin{pmatrix}6&4\\4&6\end{pmatrix}. $$ The normal equations are $6\hat\beta_1+4\hat\beta_2=12$ and $4\hat\beta_1+6\hat\beta_2=12$. Subtracting the second equation from the first gives $2\hat\beta_1-2\hat\beta_2=0$, so $\hat\beta_1=\hat\beta_2$. Substitution gives $10\hat\beta_1=12$, hence $\hat\beta_1=\hat\beta_2=1.2$.

Check. Each row of the penalized matrix multiplied by $(1.2,1.2)'$ gives $6(1.2)+4(1.2)=12$. The ridge estimate is therefore $(1.2,1.2)'$, and it satisfies both normal equations.

Trap avoided. With $\lambda=0$, the ordinary least-squares equations only require $\beta_1+\beta_2=3$, giving infinitely many coefficient pairs. Treating $X'X$ as invertible and applying the ordinary least-squares inverse formula would be invalid. The positive penalty produces a unique pair.

Worked Example: Using Cross-Products to Compute Coefficients

For a hypothetical centered model of monthly portfolio losses, suppose the observed cross-products are $X'X=\begin{pmatrix}5&1\\1&3\end{pmatrix}$ and $X'y=\begin{pmatrix}7\\5\end{pmatrix}$. With $\lambda=1$, compute the ridge coefficients and check the objective value using $y'y=18$. No interest rate, mortality assumption, or error distribution is involved.

Method. Form the penalized matrix, solve its two-by-two system, then use the cross-product expansion of the objective to verify the calculation.

The penalized matrix and equations are $$ X'X+I_2=\begin{pmatrix}6&1\\1&4\end{pmatrix}, \qquad 6\hat\beta_1+\hat\beta_2=7,\quad \hat\beta_1+4\hat\beta_2=5. $$ The determinant is $6(4)-1(1)=23$, so the matrix is invertible. Its inverse is $\frac{1}{23}\begin{pmatrix}4&-1\\-1&6\end{pmatrix}$. Therefore $$ \hat{\boldsymbol{\beta}}_{\mathrm{ridge}} =\frac{1}{23} \begin{pmatrix}4&-1\\-1&6\end{pmatrix} \begin{pmatrix}7\\5\end{pmatrix} =\frac{1}{23}\begin{pmatrix}28-5\\-7+30\end{pmatrix} =\begin{pmatrix}1\\1\end{pmatrix}. $$

Check. Multiplication by the penalized matrix gives $(6+1,1+4)'=(7,5)'=X'y$, as required. The residual sum of squares is $y'y-2\hat{\boldsymbol{\beta}}'X'y+\hat{\boldsymbol{\beta}}'X'X\hat{\boldsymbol{\beta}}=18-2(12)+10=4$. The penalty is $1(1^2+1^2)=2$, so the objective is $4+2=6$.

Trap avoided. The inverse must be applied to $X'X+\lambda I_2$, not to $X'X$ alone. The computed coefficients also satisfy the penalized equations, providing an independent check of the solution.

Worked Example: Keeping the Intercept Out of the Penalty

A hypothetical model predicts a centered risk response from one centered rating variable. There are $n=4$ observations, the response mean is $10$, and the centered cross-products are $X'X=4$ and $X'y=8$. With $\lambda=2$, find the slope and intercept, then calculate the prediction at centered predictor value $x=1$. No interest rate, mortality assumption, or error distribution is involved.

Method. Estimate the slope from centered data using the ridge formula, and estimate the intercept separately from the means. Because the predictor is centered, its sample mean is zero.

The slope is $$ \hat\beta_{\mathrm{ridge}}=\frac{X'y}{X'X+\lambda} =\frac{8}{4+2}=\frac{4}{3}. $$ The unpenalized intercept is the response mean, so $\hat\alpha=10$. The prediction at $x=1$ is $\hat y=10+(4/3)(1)=34/3$, or approximately $11.33$.

Check. With centered predictor values, the intercept equation for an unpenalized intercept is $\sum_i(y_i-\hat\alpha-\hat\beta x_i)=0$. Since $\sum_i x_i=0$, it gives $\hat\alpha=\bar y=10$. The slope equation is $(4+2)(4/3)=8=X'y$.

Trap avoided. If the intercept were incorrectly included in the penalty, its equation would be $(n+\lambda)\hat\alpha=\sum_i y_i$, giving $\hat\alpha=40/6=20/3$, not $10$. That is a different objective and does not implement the usual convention of leaving the intercept unpenalized.

Common Errors and Practical Checks

The ridge formula is compact, but several details determine whether it is being applied correctly. Before calculating, identify which coefficients are penalized, the precise scaling of the objective, and whether the penalty parameter is positive. The following checks catch frequent algebraic errors.

Using the wrong matrix dimension. If there are $p$ penalized predictors, the added identity matrix must be $I_p$, not an identity matrix sized to the number of observations.
Adding the penalty off the diagonal. The term $\lambda I_p$ adds $\lambda$ only to the diagonal entries. It does not add $\lambda$ to every entry of $X'X$.
Penalizing the intercept by accident. Under the stated convention, the intercept is estimated separately. Centering makes this separation explicit.
Assuming the ordinary least-squares inverse always exists. When predictors are collinear, $X'X$ may be singular. For $\lambda>0$, use the inverse of $X'X+\lambda I_p$, which is positive definite.
Changing the objective without changing the formula. If the objective is $\frac{1}{n}\mathrm{RSS}+\lambda\boldsymbol{\beta}'\boldsymbol{\beta}$, its normal equations contain $X'X+n\lambda I_p$. The formula must match the objective’s scaling.

A final verification is to multiply the estimated coefficient vector by $X'X+\lambda I_p$ and confirm that the result equals $X'y$. If the objective includes an unpenalized intercept, also check that its residuals sum to zero. These checks confirm the equations used; they do not determine whether a particular value of $\lambda$ is appropriate.

Key takeaway: Under the objective $\mathrm{RSS}+\lambda\boldsymbol{\beta}'\boldsymbol{\beta}$, the ridge coefficient estimate is $(X'X+\lambda I_p)^{-1}X'y$ for $\lambda>0$. The identity matrix adds a positive amount to each penalized diagonal entry, making the objective strictly convex and the solution unique even when predictors are collinear.

Check Your Understanding

For each problem, use the stated objective convention and identify which coefficients are penalized.

A centered two-predictor model has $X'X=\begin{pmatrix}3&0\\0&5\end{pmatrix}$, $X'y=(6,10)'$, and $\lambda=1$. Find both ridge coefficients. (Answer: $(1.5000,\ 1.6667)'$.)
A centered model has $X'X=\begin{pmatrix}4&2\\2&4\end{pmatrix}$, $X'y=(6,6)'$, and $\lambda=2$. Solve the penalized normal equations. (Answer: $(0.7500,\ 0.7500)'$.)
Two predictors are perfectly collinear, with $X'X=\begin{pmatrix}9&9\\9&9\end{pmatrix}$ and $X'y=(18,18)'$. For $\lambda=3$, compute the ridge estimate and state why the inverse exists. (Answer: $(6/7,\ 6/7)'$; $X'X+3I_2$ is positive definite.)
A centered one-predictor model has $n=5$, response mean $12$, $X'X=8$, $X'y=16$, and $\lambda=2$. The intercept is unpenalized. Find the slope, intercept, and fitted response at centered $x=2$. (Answer: slope $1.6$, intercept $12$, fitted response $15.2$.)
For the objective $\frac{1}{n}\mathrm{RSS}+\lambda\boldsymbol{\beta}'\boldsymbol{\beta}$, write the penalized normal equations for a centered design with $p$ predictors. (Answer: $(X'X+n\lambda I_p)\hat{\boldsymbol{\beta}}=X'y$.)

What You'll Learn