From a Penalty to a Geometric Constraint
Ridge regression is often introduced by adding a penalty to the least-squares objective. The same idea can be understood geometrically: instead of adding a cost for large coefficients, restrict the coefficients to a region around zero and find the best-fitting model inside it. This view makes clear why ridge coefficients are generally not obtained by simply shortening the ordinary least-squares coefficient vector.
Consider a linear model with response vector \(y\), design matrix \(X\), and coefficient vector \(\beta\). For this geometric discussion, the coefficients in \(\beta\) are the ones whose squared norm is being constrained. Assume \(X\) has full column rank so that the ordinary least-squares estimate is unique. The residual sum of squares at \(\beta\) is
The ridge constrained problem is to minimize this residual sum of squares subject to a bound \(t\) on the squared coefficient norm. The bound must satisfy \(t\geq 0\). With two coefficients, the feasible set is a disk and its boundary is a circle; with more coefficients, it is a ball bounded by a sphere.
The circle is centered at the origin of coefficient space, not at the ordinary least-squares estimate. The least-squares estimate determines the center of the residual contours, while the constraint determines which points are allowed. The ridge solution is the feasible point with the smallest residual sum of squares.
Why Residual Contours Are Centered at Least Squares
Write \(\hat{\beta}_{\mathrm{LS}}\) for the ordinary least-squares estimate. Since \(X\) has full column rank, it solves the normal equations \(X'X\hat{\beta}_{\mathrm{LS}}=X'y\). Expanding the residual sum of squares around this estimate gives
To see why, set \(h=\beta-\hat{\beta}_{\mathrm{LS}}\). Then \(y-X\beta=(y-X\hat{\beta}_{\mathrm{LS}})-Xh\). Squaring and expanding produces a cross-product involving \(X'(y-X\hat{\beta}_{\mathrm{LS}})\). That vector is zero by the least-squares normal equations, leaving the displayed expression.
The matrix \(X'X\) is positive definite under the full-rank assumption. Therefore, equal-RSS contours around \(\hat{\beta}_{\mathrm{LS}}\) are ellipses in two coefficient dimensions. Their shape depends on the predictors: when the predictors are correlated, the contours can be tilted relative to the coefficient axes. The constrained solution is where the lowest attainable RSS contour meets the feasible disk. If the unconstrained minimum is already inside the disk, no boundary is needed.
Connecting the Constraint to the Ridge Penalty
The constraint form and penalty form describe the same trade-off, with a parameter correspondence rather than an identity between their parameter values. For a fixed bound, introduce a nonnegative Lagrange multiplier \(\lambda\). The Lagrangian is
At an optimum, the coefficient gradient is zero and the complementary-slackness condition holds. Differentiating with respect to \(\beta\) gives
The remaining conditions are \(\beta'\beta\leq t\) and \(\lambda(\beta'\beta-t)=0\). Thus, if the bound is active, \(\beta'\beta=t\) and the multiplier is positive; if the least-squares estimate satisfies the bound, the constraint can be inactive and \(\lambda=0\). For \(\lambda>0\), \(X'X+\lambda I\) is positive definite, so the penalized solution is unique. The numerical value of \(\lambda\) depends on the chosen bound and on the scale of the residual-sum-of-squares objective.
This condition also gives a geometric interpretation: at an active solution, the gradient of RSS and the gradient of the squared norm are aligned. In two dimensions, the RSS contour just touching the circle is tangent to it. The ridge estimate is not chosen by moving directly toward the origin in every design; it is chosen by finding this contact point.
Worked Examples
Worked Example: Projecting onto a Circular Bound
A hypothetical calibration study has two predictors with \(X'X=I\), \(X'y=(3,4)'\), and \(y'y=25\). Find the constrained least-squares estimate when \(\beta'\beta\leq 4\), and check its fit.
Method. Because \(X'X=I\), the normal equations give \(\hat{\beta}_{\mathrm{LS}}=X'y=(3,4)'\). Its squared norm is \(3^2+4^2=25\), so it lies outside the feasible circle of radius \(\sqrt{4}=2\). In this special case, the RSS contours are circles centered at \((3,4)\). The closest feasible point to that center is on the same ray from the origin.
The least-squares coefficient vector has length \(\sqrt{25}=5\). The required scaling factor is the feasible radius divided by that length, \(2/5\). Therefore,
The squared norm is \(1.2^2+1.6^2=1.44+2.56=4\), so the bound is met exactly. Since \(X'X=I\), the RSS increase from the least-squares minimum is the squared distance between the coefficient vectors:
The least-squares RSS is \(y'y-(X'y)'\hat{\beta}_{\mathrm{LS}}=25-25=0\), so the constrained RSS is \(9\). This agrees with the direct residual calculation: the constrained residual is \(X((3,4)'-(1.2,1.6)')\), whose squared length is \(1.8^2+2.4^2=9\). The result is checked both against the norm bound and against the residual calculation. The radial projection works here because the RSS contours are circular. The common mistake this example avoids is applying radial scaling without checking the predictor geometry.
Worked Example: Correlated Predictors Change the Direction
In a hypothetical two-predictor model, suppose \(X'X=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), \(X'y=(4,2)'\), and \(y'y=8\). Find the constrained estimate for \(\beta'\beta\leq 13/8\). Then compare it with the radial projection of the least-squares estimate.
Method. First solve the ordinary least-squares normal equations. The vector \((2,0)'\) gives \(X'X(2,0)'=(4,2)'\), so \(\hat{\beta}_{\mathrm{LS}}=(2,0)'\). Its squared norm is \(4\), which exceeds the bound \(13/8\); the constraint is active.
Use the Lagrange equations and test \(\lambda=1\). The coefficient matrix is \(\begin{pmatrix}3&1\\1&3\end{pmatrix}\), with determinant \(9-1=8\). Solving \(\begin{pmatrix}3&1\\1&3\end{pmatrix}\beta=(4,2)'\) gives \(\beta=(10/8,2/8)'=(5/4,1/4)'\). Its squared norm is \(25/16+1/16=26/16=13/8\), so this solution satisfies the boundary condition. It also satisfies the stationarity equation: \(X'X\beta=(11/4,7/4)'\), and adding \(\lambda\beta=(5/4,1/4)'\) gives \((4,2)'\).
The least-squares RSS is \(8-(2,0)(4,2)'=0\). The constrained RSS can be checked from the quadratic expression:
A radial projection of \((2,0)'\) onto the circle would be \((\sqrt{13/8},0)'\). That point is not the constrained optimum: the actual solution is \((5/4,1/4)'\), with a nonzero second coefficient. The off-diagonal entries in \(X'X\) make the RSS contours tilted, so the point of tangency does not generally lie on the ray from the origin through the least-squares estimate. The norm, stationarity equation, and RSS have all been checked. The common mistake avoided is assuming that a circular constraint always produces a radial coefficient rescaling.
Worked Example: A Bound That Does Not Restrict Least Squares
A separate hypothetical model has \(X'X=\begin{pmatrix}3&1\\1&2\end{pmatrix}\), \(X'y=(5,5)'\), and \(y'y=18\). Determine the constrained solution when \(\beta'\beta\leq 6\). Does the finite bound necessarily cause shrinkage?
Method. The normal equations are satisfied by \(\hat{\beta}_{\mathrm{LS}}=(1,2)'\), because \(X'X(1,2)'=(5,5)'\). Its squared norm is \(1^2+2^2=5\), which is less than the bound \(6\). Thus the least-squares estimate is feasible. Since it already minimizes RSS over all coefficient vectors, it also minimizes RSS over the smaller feasible set.
For a numerical check, \(\hat{\beta}_{\mathrm{LS}}'X'y=1\cdot5+2\cdot5=15\), and \(\hat{\beta}_{\mathrm{LS}}'X'X\hat{\beta}_{\mathrm{LS}}=15\). Therefore the least-squares RSS is \(18-2(15)+15=3\). No feasible coefficient vector can have a smaller RSS, and the least-squares estimate itself is feasible. The constrained solution is consequently \((1,2)'\), with RSS \(3\) and norm squared \(5\). The multiplier is \(\lambda=0\), because the constraint is inactive. This example avoids the mistake of assuming that every finite norm bound changes the fit: only a bound that excludes the least-squares solution forces a different estimate.
Reading the Geometry Carefully
The circle or sphere describes which coefficient vectors are allowed; it does not describe the shape of the RSS contours. Those contours are determined by \(X'X\). If \(X'X\) is a multiple of the identity, the contours are circles and radial shrinkage is appropriate. With a general positive-definite \(X'X\), they are ellipses, and the constrained solution is found at their contact with the boundary.
A useful exam check is to compare the least-squares norm with the proposed bound before doing any constrained optimization:
- If \(\hat{\beta}_{\mathrm{LS}}'\hat{\beta}_{\mathrm{LS}}\leq t\), the least-squares estimate is feasible and remains optimal.
- If \(\hat{\beta}_{\mathrm{LS}}'\hat{\beta}_{\mathrm{LS}}>t\), the constraint is active, and the constrained solution must lie on the boundary under the stated full-rank setup.
- For an active constraint, verify both \(\beta'\beta=t\) and the stationarity equation \((X'X+\lambda I)\beta=X'y\), with \(\lambda\geq0\).
The constrained and penalized formulations provide complementary ways to describe the trade-off: one sets a maximum coefficient norm, while the other selects a penalty strength. This geometric perspective is aligned with the SOA Exam PA (Predictive Analytics) syllabus because it connects the ridge objective to the structure of its solution and helps diagnose common calculation errors.
Check Your Understanding
For each problem, identify whether the constraint binds where requested, show the relevant calculation, and compare your result with the answer.
- Suppose \(X'X=I\), \(X'y=(6,8)'\), and the bound is \(\beta'\beta\leq25\). Find the constrained estimate and state whether the bound is active. (Answer: \((3,4)'\); the bound is active because the least-squares squared norm is \(100\), and the constrained squared norm is \(25\).)
- A model has least-squares estimate \((1,2)'\) and bound \(t=9\). Is the constrained solution different from least squares? Give its squared norm. (Answer: No; \((1,2)'\) remains optimal because its squared norm is \(5\leq9\).)
- Let \(X'X=\begin{pmatrix}2&1\\1&2\end{pmatrix}\), \(X'y=(4,2)'\), and use \(\lambda=1\) in the penalized normal equations. Find the coefficient vector and its squared norm. (Answer: \((5/4,1/4)'\), with squared norm \(13/8\).)
- With \(X'X=I\) and \(X'y=(3,4)'\), constrain \(\beta'\beta\leq9\). Find the constrained estimate and the Lagrange multiplier under the objective \(\operatorname{RSS}+\lambda\beta'\beta\). (Answer: \((9/5,12/5)'\), with \(\lambda=2/3\).)
- In two dimensions, an analyst says the constrained ridge solution must be the radial projection of the least-squares estimate onto the coefficient circle, even when predictors are correlated. Is this generally correct? (Answer: No. Radial projection is justified when the RSS contours are circular, such as when \(X'X\) is proportional to the identity; otherwise the contours are generally ellipses and their tangency point need not be radial.)