Why More Predictors Than Cases Creates a Problem
In a regression data set, let \(n\) be the number of observations and \(p\) the number of predictor columns. When \(p>n\), a model has more slope coefficients than observed responses. Ordinary least squares can then fit the same data with multiple coefficient vectors: there are directions in coefficient space that do not change the fitted values. Ridge regression adds a positive penalty to the coefficient sizes. This makes the coefficient estimate unique, even though the predictors still outnumber the cases.
The issue is the matrix being inverted, not a special property of the response. If \(X\) is the \(n\times p\) design matrix, then the ordinary least-squares normal equations involve \(X^\top X\), a \(p\times p\) matrix. Its rank cannot exceed \(n\). When \(p>n\), it is therefore singular and has no inverse. Ridge replaces that matrix with one that is invertible for every positive penalty value.
The first term measures squared residual error, while the second penalizes large slope coefficients. The definition requires \(\lambda>0\) for the uniqueness result. At \(\lambda=0\), the objective is ordinary least squares, and with \(p>n\) the coefficient vector is not unique. The algebra below does not require normally distributed errors. It is a property of the stated optimization problem.
For clarity, the worked calculations use designs without an intercept. In a model with an unpenalized intercept, one can center the response and predictor columns, apply these slope formulas to the centered data, and then recover the intercept as the response mean minus the predictor means multiplied by the estimated slopes.
The Ridge Matrix and Its Uniqueness
Differentiating the objective with respect to \(\beta\) gives the ridge normal equations. Setting the gradient equal to zero yields
where \(I_p\) is the \(p\times p\) identity matrix. For \(\lambda>0\), the coefficient estimate is
This expression is valid whether or not \(X\) has full column rank. In particular, when \(p>n\), \(X^\top X\) alone is singular, but the penalized matrix is not.
To justify the theorem, take any nonzero vector \(z\in\mathbb{R}^p\). Then
The first term is nonnegative, and the second is strictly positive because \(z\ne0\) and \(\lambda>0\). This is the definition of positive definiteness, which implies invertibility. It also shows why the objective is strictly convex: its Hessian is \(X^\top X+\lambda I_p\). A strictly convex quadratic has exactly one minimizer.
By contrast, when \(p>n\), the null space of \(X\) contains a nonzero vector \(z\), since the rank of \(X\) is at most \(n<p\). If \(\beta\) is any coefficient vector, then \(X(\beta+z)=X\beta\). Ordinary least squares therefore gives the same fitted values for these different coefficient vectors. The ridge penalty distinguishes them by penalizing their coefficient sizes.
Using an Observation-Sized Matrix
The direct formula requires inverting a \(p\times p\) matrix. When \(p\) is much larger than \(n\), it can be more convenient to invert an \(n\times n\) matrix instead. The equivalent dual expression is
This is not a different ridge fit. To see why the two formulas agree, first observe that
Both matrices in parentheses are invertible when \(\lambda>0\), by the same positive-definiteness argument. Multiplying the identity on the left and right by their inverses gives
The left side produces the direct formula, and the right side produces the dual one. The dual calculation first finds an \(n\)-component vector \((XX^\top+\lambda I_n)^{-1}y\), then maps it back to the \(p\)-component coefficient vector by multiplying by \(X^\top\). That intermediate vector is not itself the slope vector.
Worked Example: Compute Ridge Coefficients with More Predictors Than Cases
A hypothetical study has two observations and three prepared predictors. Use the no-intercept ridge objective above, with
There are \(p=3\) predictors and \(n=2\) observations, so \(p>n\). Use the dual method, which requires a \(2\times2\) inverse.
Step 1: Form the observation-sized matrix. The two rows of \(X\) have inner products \(2\), \(1\), and \(2\), so
Step 2: Solve for the intermediate vector. The inverse is
Step 3: Map to the three coefficients. Since \(X^\top=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix}\),
Check. The fitted values are \(X\widehat{\beta}=(7/8,11/8)^\top\), so the residual vector is \(y-X\widehat{\beta}=(1/8,5/8)^\top\). Multiplying by \(X^\top\) gives \((1/8,5/8,3/4)^\top=\lambda\widehat{\beta}\), as required by the ridge normal equations. This verifies the computed estimate. The method avoids trying to invert the singular \(3\times3\) matrix \(X^\top X\).
What the Penalty Resolves—and What It Does Not
A positive ridge penalty resolves coefficient non-uniqueness; it does not create additional observations or make every predictor independently identifiable from the data. Several coefficient patterns can still describe similar fits, but the penalty selects one unique minimizer of the stated objective. That uniqueness is conditional on the chosen design matrix and positive \(\lambda\); it is not evidence that the selected coefficients are true effects or causal relationships.
The fitted values need not equal the observed responses. Ridge balances residual error against coefficient size, so the residuals generally do not have zero inner product with every predictor column. Instead, the normal equations require
This equation is a useful check on a hand calculation: the residual-predictor products equal the penalized coefficients, not necessarily zero. When a model has an unpenalized intercept, the intercept is handled separately; do not add it to the penalized slope vector in this formula.
Worked Example: Duplicate Predictors and the Unique Ridge Fit
Suppose a hypothetical two-case data set has two identical first predictors and a third predictor that records the second case only:
With ordinary least squares, exact fit requires \(\beta_1+\beta_2=2\) and \(\beta_3=1\). There are many choices for the first two coefficients, so the coefficient vector is not unique.
Step 1: Use the dual matrix. Here \(XX^\top=\begin{pmatrix}2&0\\0&1\end{pmatrix}\), so \(XX^\top+I_2=\begin{pmatrix}3&0\\0&2\end{pmatrix}\). Applying its inverse to \(y\) gives \(\alpha=(2/3,1/2)^\top\).
Step 2: Recover the coefficients. Multiplication by \(X^\top\) gives \(\widehat{\beta}=X^\top\alpha=(2/3,2/3,1/2)^\top\). The identical predictor columns receive equal coefficients in this symmetric problem.
Check. The fitted values are \((4/3,1/2)^\top\), and the residuals are \((2/3,1/2)^\top\). Thus \(X^\top(y-X\widehat{\beta})=(2/3,2/3,1/2)^\top=\lambda\widehat{\beta}\). The ridge estimate is unique even though the unpenalized exact-fit coefficients are not. The common mistake is to conclude that identical columns make ridge coefficients non-unique; with \(\lambda>0\), the full penalized objective still has a unique minimizer.
Worked Example: Verify a Ridge Fit Without Assuming Zero Residuals
For a third hypothetical design, let
Step 1: Form the dual system. The row inner products give \(XX^\top=\begin{pmatrix}1&0\\0&2\end{pmatrix}\), and therefore \(XX^\top+I_2=\begin{pmatrix}2&0\\0&3\end{pmatrix}\). Its inverse applied to \(y\) is \(\alpha=(3/2,2/3)^\top\).
Step 2: Recover slopes and fitted values. Multiplying by \(X^\top\) gives \(\widehat{\beta}=(3/2,2/3,2/3)^\top\). Hence \(X\widehat{\beta}=(3/2,4/3)^\top\), and the residual vector is \((3/2,2/3)^\top\).
Check. The residuals are not zero, as expected for a penalized fit. Their products with the predictor columns are \(X^\top(y-X\widehat{\beta})=(3/2,2/3,2/3)^\top=\lambda\widehat{\beta}\). This confirms the solution. Assuming instead that each residual must be orthogonal to every predictor is the ordinary least-squares condition, not the ridge condition.
Common Exam Traps
- Trying to invert \(X^\top X\) when \(p>n\). That matrix is singular because its rank is at most \(n<p\). Use \(X^\top X+\lambda I_p\) for the direct ridge formula.
- Using a zero penalty and still claiming uniqueness. The positive-definiteness argument relies on \(\lambda>0\). At \(\lambda=0\), the ridge formula reduces to ordinary least squares, whose coefficients are non-unique in this setting.
- Confusing the dual intermediate vector with the coefficients. The vector \((XX^\top+\lambda I_n)^{-1}y\) has length \(n\). Multiply it by \(X^\top\) to obtain the length-\(p\) slope vector.
- Penalizing an intercept by accident. The formulas here apply to slopes. With an intercept, treat it as unpenalized, or fit the slopes to centered data and restore the intercept afterward.
The central result is compact: adding \(\lambda I_p\) makes the slope system invertible for every \(\lambda>0\), while the dual formula can reduce the matrix inversion from \(p\times p\) to \(n\times n\). Understanding both forms is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Check Your Understanding
Work each problem through the dimensions and matrix products before checking the final answer.
- A design has \(n=5\) observations and \(p=8\) predictors. For a fixed \(\lambda=0.4\), state the dimensions of the direct penalized matrix and the dual penalized matrix, and say which is smaller. (Answer: The direct matrix is \(8\times8\); the dual matrix is \(5\times5\), so the dual matrix is smaller.)
- Let \(X=\begin{pmatrix}1&0&1\\0&1&1\end{pmatrix}\), \(y=(2,0)^\top\), and \(\lambda=1\). Find the ridge coefficient vector using the dual form. (Answer: \((3/4,-1/4,1/2)^\top\).)
- Let \(X=\begin{pmatrix}1&1&0\\0&1&1\end{pmatrix}\), \(y=(1,3)^\top\), and \(\lambda=1\). Find the ridge coefficient vector and verify its fitted values. (Answer: \((0,1,1)^\top\), with fitted values \((1,2)^\top\).)
- Two identical predictor columns and one separate column give \(X=\begin{pmatrix}1&1&0\\0&0&1\end{pmatrix}\), \(y=(4,2)^\top\), and \(\lambda=2\). Find the ridge coefficient vector using the dual matrix. (Answer: \((1,1,2/3)^\top\).)
- For \(p>n\), explain why \(X^\top X\) is singular but \(X^\top X+\lambda I_p\) is invertible when \(\lambda>0\). (Answer: \(\operatorname{rank}(X^\top X)\le n<p\), so \(X^\top X\) is singular; for every nonzero \(z\), \(z^\top(X^\top X+\lambda I_p)z=\lVert Xz\rVert^2+\lambda\lVert z\rVert^2>0\), so the penalized matrix is positive definite and invertible.)