The Geometric View of Ridge Regression

Why Ridge Has a Geometric Interpretation

Ridge regression is often introduced by adding a penalty to residual sum of squares. The same fit can be viewed as minimizing residual sum of squares subject to a bound on the size of the coefficient vector. This second view explains why ridge estimates are pulled toward zero and why, unlike a variable-selection method, they are generally not exactly zero.

For clarity, consider a linear regression with \(p\) predictors whose columns and response have been centered. The intercept is therefore handled separately and is not included in the coefficient penalty. Let \(X\) be the \(n\times p\) design matrix, \(y\) the centered response vector, and \(\beta\) the vector of slope coefficients. Assume \(X\) has full column rank. The residual sum of squares is

The full-rank assumption makes the unpenalized least-squares estimate unique and gives the RSS contours their familiar elliptical shape. The intercept is not constrained; if predictors are not centered, the corresponding treatment of an unpenalized intercept must be accounted for before using the slope-only geometry below.

The ball is spherical in the coordinates of the coefficients being penalized. Consequently, predictor scaling matters: changing the units of a predictor changes the numerical scale of its coefficient and can change the geometry. Standardizing predictors before fitting is one way to make coefficient magnitudes more comparable, but it is a modeling choice rather than a requirement of the constrained formulation.

RSS Contours and the Constrained Solution

Write \(H=X^\mathsf{T}X\) and \(z=X^\mathsf{T}y\). Expanding the squared residual gives

For a fixed RSS value, this equation describes a contour in coefficient space. Under the full-rank assumption, \(H\) is positive definite, so contours around the ordinary least-squares estimate are ellipses. The ordinary least-squares estimate is \(\hat{\beta}_{\mathrm{OLS}}=H^{-1}z\), the center of these ellipses, and it gives the smallest RSS without a coefficient constraint.

To find the constrained solution, imagine expanding an RSS contour outward from its minimum until it first touches the feasible ball. If the ordinary least-squares estimate lies inside the ball, the unconstrained minimum is already feasible. If it lies outside, the first contact occurs on the ball’s boundary. At a smooth point of contact, the contour and the sphere have parallel normal directions. Algebraically, their gradients are proportional.

Why the Constrained and Penalized Forms Match

For a positive radius, introduce a nonnegative Lagrange multiplier \(\lambda\) for the constraint \(\beta^\mathsf{T}\beta\le t\). The Lagrangian is

Differentiating with respect to \(\beta\) and setting the gradient to zero gives the stationarity equation. Because the gradient of RSS is \(2(H\beta-z)\), this equation is

This is the ridge normal equation for the objective \(\operatorname{RSS}(\beta)+\lambda\beta^\mathsf{T}\beta\). The multiplier is positive when the constraint is active and its positive radius is smaller than the ordinary least-squares coefficient norm. The stationarity equation alone does not determine the radius: the multiplier must also be chosen so that the solution satisfies the active constraint.

The existence statement follows by examining how the norm changes with \(\lambda\). Let \(d_j>0\) be the eigenvalues of \(H\), and express \(z\) in the corresponding orthonormal eigenvector coordinates as \(q_1,\ldots,q_p\). Then the squared norm of the ridge solution is

If \(z\ne0\), at least one \(q_j\) is nonzero. Each nonzero term decreases as \(\lambda\) increases, so the sum decreases strictly and continuously. At \(\lambda=0\), it equals the ordinary least-squares squared norm; as \(\lambda\) tends to infinity, it approaches zero. Therefore, every strictly intermediate positive squared radius is attained at one positive finite multiplier. The multiplier is unique because the norm decreases strictly.

This also explains the usual nonzero-coefficient description of ridge. For finite \(\lambda\), \(H+\lambda I\) is invertible, and its solution is generally not the zero vector when \(z\ne0\). As the radius approaches zero, the corresponding multiplier grows without bound; that limiting behavior does not mean a finite multiplier produces the exact zero solution.

Ellipses, Directions, and Shrinkage

When \(H\) is not a multiple of the identity matrix, RSS contours are ellipses rather than circles. The ridge solution still lies where an RSS contour first meets the coefficient ball, but the ellipse’s orientation and curvature affect the point of contact. Thus the coefficients need not all shrink by the same proportion in the original predictor coordinates.

The Zero-Radius Endpoint and an Important Trap

A positive but small sphere and a zero-radius sphere are not interchangeable in the multiplier equation. When \(t=0\), feasibility itself forces \(\beta=0\). At this point the boundary constraint has zero gradient, so the usual smooth-boundary stationarity argument does not supply a finite multiplier for every problem. In particular, if \(z\ne0\), substituting \(\beta=0\) into \((H+\lambda I)\beta=z\) gives \(0=z\), which is impossible for any finite \(\lambda\).

What the Geometry Does—and Does Not—Say

The geometric view is useful because it separates two questions: which coefficient vectors are allowed, and how well each allowed vector fits the data. Tightening the ball cannot lower the minimum RSS, since it removes candidate coefficient vectors. Increasing the penalty in the penalized formulation likewise pushes the solution toward smaller coefficient norm.

The geometry does not say that ridge sets coefficients exactly to zero, nor that the coefficient sphere is invariant to predictor units. For a nonzero \(z\) and a positive finite multiplier, the ridge estimate is generally nonzero. Also, a sphere in unstandardized coefficients can represent very different practical changes along different predictors. Interpret a ridge fit in the coordinates and scaling actually used to define its penalty.

A final exam trap is to confuse the radius with its square. If the constraint is written \(\beta^\mathsf{T}\beta\le t\), then \(t\) is the squared radius, not the radius itself. The corresponding boundary condition is \(\|\beta\|=\sqrt{t}\). Keeping that distinction explicit avoids incorrect multiplier calculations.