Why a Positive Penalty Stabilizes Estimation

Why Adding a Penalty Changes the Linear System

When predictors are redundant, the data may not contain enough information to determine every regression coefficient separately. In ordinary least squares, this can make the normal-equation matrix singular, so there is no unique coefficient vector. Ridge regression adds a positive penalty to the objective. The key algebraic consequence is that, when every coefficient is penalized, the matrix in the ridge normal equations is invertible—even if the design matrix is rank deficient.

This result explains one precise sense in which the penalty stabilizes estimation: it guarantees a unique solution to the stated optimization problem. It does not guarantee that the model is appropriate, that the predictions are accurate, or that the result is numerically well-conditioned for every very small penalty.

Definition: Let $X$ be a fixed $n \times p$ real design matrix, $y$ an $n$-vector of responses, and $\beta$ a $p$-vector of coefficients. For a penalty parameter $\lambda>0$, the fully penalized ridge estimate minimizes $$ Q(\beta)=\lVert y-X\beta\rVert^2+\lambda\lVert\beta\rVert^2. $$ Here $\lVert z\rVert^2=z^\mathsf{T}z$. The ridge normal equations are $(X^\mathsf{T}X+\lambda I_p)\hat{\beta}=X^\mathsf{T}y$, where $I_p$ is the $p \times p$ identity matrix. This tutorial assumes that all $p$ coefficients in $\beta$ are penalized.

The assumption about penalizing every coefficient matters. If a fitting procedure leaves some coefficient unpenalized, the argument below does not automatically establish invertibility for the entire system. The result here is specifically about adding $\lambda I_p$ to the full matrix $X^\mathsf{T}X$.

The Positive-Definiteness Argument

A square matrix is positive definite if its quadratic form is positive for every nonzero vector. Positive definiteness implies invertibility: if a positive-definite matrix $A$ had a nonzero vector $z$ with $Az=0$, then $z^\mathsf{T}Az=0$, contradicting positive definiteness.

For any $z\in\mathbb{R}^p$, calculate the quadratic form of the ridge matrix:

z^\mathsf{T}(X^\mathsf{T}X+\lambda I_p)z =(Xz)^\mathsf{T}(Xz)+\lambda z^\mathsf{T}z =\lVert Xz\rVert^2+\lambda\lVert z\rVert^2.

Both squared norms are nonnegative. If $z\ne 0$, then $\lVert z\rVert^2>0$; because $\lambda>0$, it follows that $\lambda\lVert z\rVert^2>0$. Therefore the entire expression is strictly positive, whether or not $Xz=0$. This proves that $X^\mathsf{T}X+\lambda I_p$ is positive definite and hence invertible.

The same reasoning shows why rank deficiency is not an obstacle to this result. Rank deficiency means there is some nonzero $z$ for which $Xz=0$. The data term in the quadratic form then contributes zero, but the penalty term contributes $\lambda\lVert z\rVert^2>0$. That positive contribution removes the zero direction from the penalized system.

Start with an arbitrary nonzero vector.
To test positive definiteness, take any $z\ne 0$ in the coefficient space.

Expand the quadratic form.
The data contribution is $\lVert Xz\rVert^2$, and the penalty contribution is $\lambda\lVert z\rVert^2$.

Use the strict inequality.
The first contribution is nonnegative and the second is strictly positive because $\lambda>0$ and $z\ne0$.

Conclude invertibility.
The matrix is positive definite, so the ridge normal equations have exactly one solution.

When Ordinary Least Squares Has No Unique Solution

Ordinary least squares minimizes $\lVert y-X\beta\rVert^2$, without the penalty term. Its normal equations are $X^\mathsf{T}X\hat{\beta}=X^\mathsf{T}y$. If the columns of $X$ are linearly dependent, then there is a nonzero vector $z$ such that $Xz=0$. Whenever $\hat{\beta}$ is a least-squares solution, adding any multiple of $z$ leaves the fitted values unchanged:

X(\hat{\beta}+cz)=X\hat{\beta}+cXz=X\hat{\beta} \qquad\text{for every real }c.

Thus the coefficients need not be unique even though the fitted values are. Ridge adds a cost for moving in coefficient directions such as $z$, selecting a unique minimizer under the assumptions stated above.

Worked Example: Two Identical Predictors

A hypothetical portfolio study uses two predictors that happen to record the same quantity. The design matrix and response are $X=\begin{bmatrix}1&1\\2&2\end{bmatrix}$ and $y=\begin{bmatrix}2\\4\end{bmatrix}$. First determine whether ordinary least squares has a unique coefficient vector. Then solve the fully penalized ridge problem with $\lambda=1$. Use the normal equations and check the result against the objective.

Method and ordinary least squares. The columns of $X$ are identical, so predictions depend only on $\beta_1+\beta_2$. Exact fits require $\beta_1+\beta_2=2$: the first row requires this sum to equal 2, and the second row then predicts $2(2)=4$. For example, both $(2,0)$ and $(0,2)$ fit exactly. Ordinary least squares therefore has infinitely many solutions.

Form the ridge system. Direct multiplication gives $X^\mathsf{T}X=\begin{bmatrix}5&5\\5&5\end{bmatrix}$ and $X^\mathsf{T}y=\begin{bmatrix}10\\10\end{bmatrix}$. Adding $I_2$, since $\lambda=1$, gives

\begin{bmatrix}6&5\\5&6\end{bmatrix} \begin{bmatrix}\hat{\beta}_1\\\hat{\beta}_2\end{bmatrix} = \begin{bmatrix}10\\10\end{bmatrix}.

Subtracting the second equation from the first gives $\hat{\beta}_1-\hat{\beta}_2=0$. Substitution into either equation gives $11\hat{\beta}_1=10$, so $\hat{\beta}_1=\hat{\beta}_2=10/11$.

Check the objective. The fitted values are $20/11$ and $40/11$, so the residuals are $2/11$ and $4/11$. The residual sum of squares is $4/121+16/121=20/121$. The penalty is $(10/11)^2+(10/11)^2=200/121$. Thus $Q=220/121=20/11$. This confirms the calculation from the stated objective. The common mistake this avoids is trying to invert the singular ordinary least-squares matrix instead of adding $\lambda$ to each diagonal entry.

The Penalty Selects Among Redundant Coefficients

When predictors are exact multiples of one another, ordinary least squares can identify the effect of a combination of coefficients without identifying each coefficient separately. Ridge still uses the observed fit, but its penalty also affects which coefficient vector is chosen. The next calculation shows both features in a design with a redundant column and an observation that cannot be fitted.

Worked Example: Predictors That Are Scalar Multiples

For a hypothetical severity model, let $X=\begin{bmatrix}1&2\\2&4\\0&0\end{bmatrix}$, $y=\begin{bmatrix}3\\6\\1\end{bmatrix}$, and $\lambda=2$. Find the ridge estimate, establish why ordinary least squares is not unique, and verify the ridge objective value.

Identify the redundancy. The second column is twice the first. For any coefficients, the first two fitted values depend on $s=\beta_1+2\beta_2$, and the last fitted value is zero. Ordinary least squares is minimized when $s=3$, which fits the first two responses exactly; the third response leaves a residual of 1 for every coefficient vector. There are infinitely many pairs satisfying $\beta_1+2\beta_2=3$, so ordinary least squares is not unique.

Build and solve the ridge system. Matrix multiplication gives $X^\mathsf{T}X=\begin{bmatrix}5&10\\10&20\end{bmatrix}$ and $X^\mathsf{T}y=\begin{bmatrix}15\\30\end{bmatrix}$. Adding $2I_2$ gives

\begin{bmatrix}7&10\\10&22\end{bmatrix} \begin{bmatrix}\hat{\beta}_1\\\hat{\beta}_2\end{bmatrix} = \begin{bmatrix}15\\30\end{bmatrix}.

The determinant of the coefficient matrix is $7(22)-10(10)=54$, which is positive. Its inverse is $\frac{1}{54}\begin{bmatrix}22&-10\\-10&7\end{bmatrix}$. Therefore

\hat{\beta} =\frac{1}{54} \begin{bmatrix}22&-10\\-10&7\end{bmatrix} \begin{bmatrix}15\\30\end{bmatrix} =\frac{1}{54}\begin{bmatrix}30\\60\end{bmatrix} =\begin{bmatrix}5/9\\10/9\end{bmatrix}.

Check the objective. The first fitted value is $5/9+20/9=25/9$, and the second is $50/9$; the third is 0. The residuals are $2/9$, $4/9$, and 1. Hence the residual sum of squares is $4/81+16/81+1=101/81$. The penalty is $2[(5/9)^2+(10/9)^2]=2(125/81)=250/81$. The total is $351/81=13/3$. Ridge has chosen a unique pair, but it does not force the fitted combination $s$ to remain exactly 3 because the penalty trades off fit against coefficient size. The common mistake avoided here is assuming that ridge must preserve the exact ordinary least-squares fit.

Invertibility and Numerical Conditioning Are Related but Different

A matrix can be invertible yet still be difficult to solve with numerical precision. Positive $\lambda$ guarantees mathematical invertibility for the fully penalized system, but a very small $\lambda$ may leave the matrix poorly conditioned. The following example compares the diagonal entries before and after the penalty; it does not determine how $\lambda$ should be selected.

Worked Example: A Small Diagonal Entry

Suppose a two-predictor design has $X=\begin{bmatrix}0.1&0\\0&2\end{bmatrix}$, response $y=\begin{bmatrix}1\\4\end{bmatrix}$, and penalty parameter $\lambda=0.99$. Compute the ordinary least-squares and ridge coefficients, then compare the condition numbers of their normal-equation matrices.

Ordinary least squares. Here $X^\mathsf{T}X=\begin{bmatrix}0.01&0\\0&4\end{bmatrix}$ and $X^\mathsf{T}y=\begin{bmatrix}0.1\\8\end{bmatrix}$. Both diagonal entries are positive, so the unpenalized system is invertible. Solving coordinate by coordinate gives $\hat{\beta}_{OLS}=(0.1/0.01,\;8/4)=(10,\;2)$.

Ridge estimate. Adding $0.99I_2$ gives the diagonal matrix $\begin{bmatrix}1&0\\0&4.99\end{bmatrix}$. Thus $\hat{\beta}_1=0.1/1=0.1$ and $\hat{\beta}_2=8/4.99=800/499$, which is approximately 1.6032. The check is direct: $1(0.1)=0.1$ and $4.99(800/499)=8$, as required by the normal equations.

Compare conditioning. For a positive diagonal matrix, its condition number is the ratio of its largest diagonal entry to its smallest. Before the penalty, this ratio is $4/0.01=400$. After the penalty, it is $4.99/1=4.99$. The penalty has increased the small diagonal entry and reduced this condition number. The common mistake is to conclude that invertibility alone guarantees numerical stability; here the condition-number calculation provides additional information beyond the positive-definiteness proof.

Common Traps and Practical Meaning

Using a zero penalty. The proof requires $\lambda>0$. If $\lambda=0$, the ridge matrix is just $X^\mathsf{T}X$, which can be singular.
Adding the penalty to every matrix entry. Adding $\lambda I_p$ adds $\lambda$ to each diagonal entry and zero to off-diagonal entries. For example, $\begin{bmatrix}5&5\\5&5\end{bmatrix}+I_2=\begin{bmatrix}6&5\\5&6\end{bmatrix}$, not a matrix with 1 added to all four entries.
Confusing unique coefficients with a perfect fit. Ridge gives a unique minimizer under the stated assumptions, but the minimizer need not have zero residuals or match the ordinary least-squares coefficients.
Equating invertibility with a well-chosen penalty. Every positive $\lambda$ gives the algebraic guarantee here; that fact alone does not show that a particular value is suitable for prediction.

The result is useful whenever predictors are redundant or nearly redundant: the ridge system remains uniquely solvable even when ordinary least squares does not. The important qualification is that this is a statement about a fully penalized coefficient vector and the matrix $X^\mathsf{T}X+\lambda I_p$. It is not a universal claim about every regression fitting procedure.

Key takeaway: For every nonzero $z$, $z^\mathsf{T}(X^\mathsf{T}X+\lambda I_p)z=\lVert Xz\rVert^2+\lambda\lVert z\rVert^2>0$ when $\lambda>0$. Therefore the fully penalized ridge matrix is positive definite and invertible, including when $X$ is rank deficient.

Check Your Understanding

Work through each problem, then compare your result with the answer at the end of the item.

A design has $X^\mathsf{T}X=\begin{bmatrix}4&4\\4&4\end{bmatrix}$. With $\lambda=3$, write the ridge matrix and calculate its determinant. (Answer: $\begin{bmatrix}7&4\\4&7\end{bmatrix}$, determinant $33$.)
A design has $X^\mathsf{T}X=\begin{bmatrix}0&0\\0&9\end{bmatrix}$. For $\lambda=0.5$, state the eigenvalues of the ridge matrix and its condition number. (Answer: eigenvalues $0.5$ and $9.5$; condition number $19$.)
Suppose $X^\mathsf{T}X=\begin{bmatrix}2&2\\2&2\end{bmatrix}$, $X^\mathsf{T}y=\begin{bmatrix}6\\6\end{bmatrix}$, and $\lambda=1$. Solve the ridge normal equations. (Answer: $\hat{\beta}=(6/5,\;6/5)^\mathsf{T}$.)
If $X$ is rank deficient, choose a nonzero $z$ with $Xz=0$. Explain why $z^\mathsf{T}(X^\mathsf{T}X+0.25I)z$ is positive, and state what that implies. (Answer: it equals $0.25\lVert z\rVert^2>0$, so the ridge matrix is positive definite and invertible.)
Let $X=\begin{bmatrix}1&0\\0&0\end{bmatrix}$, $y=\begin{bmatrix}4\\2\end{bmatrix}$, and $\lambda=2$. Describe the ordinary least-squares solutions and calculate the ridge estimate. (Answer: ordinary least squares has $\beta_1=4$ with any $\beta_2$; ridge gives $\hat{\beta}=(4/3,\;0)^\mathsf{T}$.)

What You'll Learn