Ridge and Lasso When Predictors Outnumber Cases

More Predictors Than Cases

A regression data set can have more candidate predictors than observed cases. This occurs, for example, when a small data set contains measurements on many features. In that setting, the number of coefficients to estimate can exceed the number of independent pieces of information in the observed outcomes. Ordinary least squares may then have multiple coefficient vectors that fit equally well, or may fail to define a unique coefficient vector at all.

Ridge and lasso address this computational difficulty by adding a penalty to the residual sum of squares. Ridge shrinks coefficients and, for a positive penalty, gives a unique coefficient vector even when predictors are linearly dependent. Lasso also penalizes coefficients, but its absolute-value penalty can produce exact zeros. Neither property implies that the fitted coefficients reveal which predictors are truly responsible for an outcome.

With an intercept, a common convention is to center the response and each predictor before fitting the penalized slopes, then estimate the intercept separately and leave it unpenalized. Standardizing predictors is also common so that the penalty treats them on comparable scales. The examples below instead specify uncentered data and explicitly omit an intercept, so their arithmetic does not rely on a centering claim. The matrix results apply to whatever predictor and response values enter the stated objective.

The fits discussed here are solutions to optimization problems for a given design matrix and a given positive penalty value. Their computation does not require normally distributed errors. When a model is used to describe a stochastic response, assumptions about the response and errors matter for that broader interpretation; a penalized fit alone does not establish them.

Why Ridge Can Still Be Computed

For a specified \(\lambda>0\), define the ridge coefficient vector as the minimizer of residual sum of squares plus a squared-coefficient penalty. The intercept, if included, is not penalized. Here the objective is written without a factor of one-half, so \(\lambda\) has the scale shown in the formula.

Even if \(p>n\), the matrix \(X^\mathsf{T}X\) is only guaranteed to be positive semidefinite, not invertible. Ridge adds \(\lambda I_p\). For every nonzero \(v\), \(v^\mathsf{T}(X^\mathsf{T}X+\lambda I_p)v=\|Xv\|^2+\lambda\|v\|^2>0\). Thus the matrix is positive definite and invertible. The resulting ridge coefficient vector is unique.

There is also a useful formula involving an \(n \times n\) matrix rather than a \(p \times p\) matrix. When \(p\) is much larger than \(n\), this can make the computation more manageable.

To verify the identity, note that \((X^\mathsf{T}X+\lambda I_p)X^\mathsf{T}=X^\mathsf{T}(XX^\mathsf{T}+\lambda I_n)\). Both matrices to be inverted are positive definite for \(\lambda>0\). Multiplying by their inverses gives the stated equality. The dual formula does not change the fit or the penalty; it expresses the same coefficient vector through an observation-sized system.

How Lasso Can Give a Sparse Fit

Lasso replaces the squared-coefficient penalty with a penalty on the sum of absolute coefficient values. We use the objective with a factor of one-half on the residual sum of squares below. Its particular numerical \(\lambda\) scale differs from the ridge convention above, so a value of \(\lambda\) should always be interpreted relative to the stated objective.

The absolute-value penalty has corners at zero, making exact zero coefficients possible. When \(p>n\), there is an important sparse-solution result: at least one lasso minimizer can be chosen with no more than \(\operatorname{rank}(X)\) nonzero coefficients. Since \(\operatorname{rank}(X)\leq n\), this guarantees the existence of a solution with at most \(n\) active predictors. It does not say that every minimizer must be sparse, or that the active predictors are uniquely determined.

Here is a justification. Take a lasso minimizer and let \(z=X\hat{\beta}\) be its fitted-value vector. Among all coefficient vectors giving this same \(z\), the minimizer must have the smallest possible sum of absolute values; otherwise replacing it with a lower-penalty representation of \(z\) would lower the full lasso objective. Finding a minimum-\(\ell_1\) representation of \(z\) can be written as minimizing \(\sum_j(u_j^++u_j^-)\), subject to \(X(u^+-u^-)=z\) and \(u_j^+,u_j^-\geq0\). A basic feasible optimum for these equality constraints uses at most \(\operatorname{rank}(X)\) positive variables. Also, both \(u_j^+\) and \(u_j^-\) cannot be positive at an optimum, since reducing both by the same small amount leaves the representation unchanged and lowers the objective. Thus the corresponding coefficient vector has at most \(\operatorname{rank}(X)\) nonzero entries. It gives the same fit and penalty as the original minimizer, so it is also a lasso solution.

What a Computable Fit Does—and Does Not—Tell You

The ridge inverse guarantees a unique numerical coefficient vector for the stated design and penalty. Lasso can produce exact zeros and has a sparse solution even when predictors outnumber cases. These are useful properties for constructing predictions, but they do not create information absent from the data. In particular, the penalty does not make a predictor’s individual association identifiable when its column is redundant with other columns.

In the example, let \(v=(1,0,-1)^\mathsf{T}\). Since \(Xv=0\), adding any multiple of \(v\) to a coefficient vector leaves all training fitted values unchanged. For ordinary least squares, every vector \((t,1,2-t)^\mathsf{T}\) gives fitted values \((2,1)^\mathsf{T}\). The data distinguish the sum of the first and third coefficients, not their separate contributions. Ridge and lasso select coefficient allocations by their penalties, but that selection should not be mistaken for evidence that one predictor has a uniquely established effect.

The limitation also matters for inference. Penalized coefficients are not ordinary least-squares estimates: shrinkage changes their values, and lasso selection makes the set of reported coefficients depend on the penalty and data. Standard OLS standard errors and tests do not automatically apply to these coefficients or account for that selection. A unique ridge solution is not, by itself, a confidence interval or a test of an individual coefficient.

Common Mistakes

• Assuming \(p>n\) makes every penalized fit impossible. The unpenalized cross-product matrix can be singular, but ridge with \(\lambda>0\) has a positive-definite penalized matrix and a unique fit.
• Claiming lasso always selects a unique set of predictors. Duplicate or dependent columns can yield multiple coefficient solutions. A sparse solution exists, but it need not be unique.
• Equating a lasso zero with proof of no relationship. A zero is the result of a penalized optimization at a specified \(\lambda\); correlated or redundant predictors can share information.
• Treating coefficients as separately identified because the algorithm returned numbers. When \(Xv=0\), coefficient vectors differing by \(v\) have identical fitted values. A penalty selects among representations; the observed design may not distinguish their individual contributions.
• Calling data centered when they are not. If an example uses an intercept or claims centered variables, check column and response means. Otherwise state that the fit is uncentered and specify whether an intercept is included.