Ridge Regression From Singular Value Decomposition

Why Derive Ridge With the SVD?

When the number of predictors exceeds the number of observations, or when predictors are linearly dependent, the design matrix does not have full column rank. In that situation, ordinary least squares cannot uniquely determine every coefficient: multiple coefficient vectors can give the same fitted values. Ridge regression adds a positive penalty to the least-squares objective. The singular value decomposition (SVD) makes explicit how this changes the coefficient calculation and why the resulting estimate is unique.

The derivation below concerns the coefficient vector for a fixed positive penalty parameter. It does not choose that parameter. As in the earlier material on Predictor Scaling Before Penalization, the predictor scale matters: the penalty acts on the coefficients as represented in the design matrix. For clarity, treat any intercept separately and do not penalize it; equivalently, the response and predictors can first be adjusted for the intercept. The matrix \(X\) below is the design matrix for the coefficients being penalized.

The assumptions needed for the algebra are modest: \(X\) and \(y\) are finite real-valued matrices/vectors of compatible dimensions, and \(\lambda\) is strictly positive. No full-rank assumption on \(X\) is required. A statistical model such as \(y=X\beta+\varepsilon\) can be used to motivate the fit, but the existence and uniqueness result below follows directly from the objective and does not require normally distributed errors.

From the Penalized Normal Equations to the SVD

Differentiate \(Q(\beta)\) with respect to \(\beta\). Setting its gradient to zero gives the penalized normal equations:

For ordinary least squares, \(X^\mathsf{T}X\) is singular when \(X\) is rank deficient, so it cannot be inverted to obtain a unique coefficient vector. Ridge adds \(\lambda I_p\). For any nonzero \(z\in\mathbb{R}^p\),

The inequality holds because \(\|Xz\|_2^2\geq 0\), \(\|z\|_2^2>0\), and \(\lambda>0\). Thus \(X^\mathsf{T}X+\lambda I_p\) is positive definite and invertible. The objective is strictly convex, so its minimizer is unique, regardless of the rank of \(X\).

To write that minimizer in SVD form, let \(r=\operatorname{rank}(X)\) and use the compact SVD \(X=U_rD_rV_r^\mathsf{T}\). The columns of \(U_r\in\mathbb{R}^{n\times r}\) and \(V_r\in\mathbb{R}^{p\times r}\) are orthonormal, and \(D_r=\operatorname{diag}(d_1,\ldots,d_r)\), where every singular value \(d_j\) is positive. Substitution into the penalized normal equations gives the expression below.

One way to verify the last line is to substitute it into the normal equations. Multiplication by \(X^\mathsf{T}X+\lambda I_p\) applies \(d_j^2+\lambda\) to the coefficient along each column of \(V_r\), canceling its denominator and leaving \(V_rD_rU_r^\mathsf{T}y=X^\mathsf{T}y\). Components in the null space of \(X\), which are orthogonal to the columns of \(V_r\), are zero in this solution. The positive penalty makes any nonzero such component increase the objective without changing the fitted values.

Worked SVD Calculations

Assumptions, Diagnostics, and Alternatives

The unique-solution result requires only \(\lambda>0\) and a finite design matrix. It does not assert that ridge coefficients are unbiased estimates of an underlying parameter, nor that the chosen penalty is suitable for prediction. Those are separate questions. The formula also depends on the stated objective: using a different scaling of the residual term changes the numerical meaning of \(\lambda\), unless the penalty is rescaled accordingly.

A practical calculation should check that the fit used the intended \(\lambda\), applied the penalty to the intended coefficients, and treated the intercept consistently. In numerical software, singular values very close to zero can be sensitive to floating-point precision; a reported numerical rank depends on the tolerance used. This does not invalidate the mathematical formula, but it can affect computed SVD components. Confirm that the singular values and design dimensions are plausible, and compare the SVD solution with a direct solve of \((X^\mathsf{T}X+\lambda I_p)\hat{\beta}=X^\mathsf{T}y\) on a small problem.

The ordinary least-squares solution is the main contrast. When \(X\) has full column rank, it is uniquely given by \((X^\mathsf{T}X)^{-1}X^\mathsf{T}y\); when \(X\) is rank deficient, least-squares fitted values remain defined but the coefficient vector is not unique. The Moore–Penrose pseudoinverse selects the least-squares coefficient vector of smallest Euclidean length. Ridge instead solves a penalized objective for a positive \(\lambda\), giving a unique coefficient vector. A truncated SVD is another possible approach: it discards selected small-singular-value components, whereas ridge retains the nonzero SVD components with modified factors. These approaches answer related but not identical estimation questions.

A frequent error is to say, “\(X^\mathsf{T}X\) is singular, so ridge cannot be computed.” That reasoning overlooks the penalty term. The matrix that must be inverted is \(X^\mathsf{T}X+\lambda I_p\), not \(X^\mathsf{T}X\) alone, and for every \(\lambda>0\) it is positive definite. A different error is to set \(\lambda=0\) while still claiming ridge has produced a unique solution. At \(\lambda=0\), the objective is ordinary least squares and rank deficiency again allows multiple coefficient vectors. As \(\lambda\) decreases toward zero, the ridge solution approaches the minimum-length least-squares solution, but that limiting statement does not make the \(\lambda=0\) coefficient unique in general.