Why the Number of Predictors Matters
A regression may include a large set of candidate variables even when the available data are limited. For example, a hypothetical insurer might have a few dozen claims but hundreds of engineered rating and experience features. When the number of predictor columns exceeds the number of observations, fitting an ordinary least squares model is not just difficult: the coefficients may not be uniquely determined. Penalized regression provides a way to choose a defined solution by adding a constraint or penalty, but it does not create information that the sample does not contain.
The key is to distinguish the number of named variables from the number of columns actually supplied to the fitting algorithm. A categorical variable expanded into indicator columns can add several predictors; interactions and transformations can add still more. The relevant count is the column count of the final model matrix, not the number of entries in a formula or source data table.
For a linear regression with response vector \(y\), ordinary least squares minimizes the residual sum of squares, \(\|y-X\beta\|_2^2\). If the columns of \(X\) are dependent, different coefficient vectors can produce the same fitted values. When \(p>n\), there is necessarily a nonzero vector \(v\) such that \(Xv=0\). Thus \(\beta\) and \(\beta+v\) give identical predictions because \(X(\beta+v)=X\beta\). The fitted values might be determined even though the individual coefficients are not.
Penalties in a High-Dimensional Model
For the examples below, use the squared-error elastic-net objective
Here \(\beta_0\) is the intercept, \(\beta\) is the vector of slopes, \(\lambda\geq 0\) controls penalty strength, and \(0\leq\alpha\leq1\) determines the mixture of penalties. The intercept is not penalized in this definition. At \(\alpha=0\), the fit is ridge regression; at \(\alpha=1\), it is lasso; values strictly between zero and one give elastic net. This objective describes a squared-error linear regression, not a logistic regression or a model with a different loss function.
R implementations may standardize predictors and apply implementation-specific conventions for the loss scaling. Therefore, hand calculations should use the stated objective, while interpreting software output requires checking how the predictors and objective were handled. In practice, the model matrix and preprocessing must be formed using training data only; within resampling, those steps are repeated separately for each training fold.
Worked Example: Seeing Nonuniqueness in an OLS Fit
A small study has \(n=3\) observations and \(p=4\) centered predictors. The columns are \(x_1=(1,0,-1)^\mathsf{T}\), \(x_2=(0,1,-1)^\mathsf{T}\), \(x_3=x_1+x_2\), and \(x_4=2x_1\). The response is \(y=2x_1-x_2\). Show two different OLS coefficient vectors that fit perfectly, and explain the implication.
Step 1: Count the columns and identify the dependence. There are four predictor columns and only three observations, so \(p>n\). In addition to that dimension constraint, the given columns satisfy \(x_3=x_1+x_2\) and \(x_4=2x_1\). Consequently, the predictor matrix has rank 2.
Step 2: Write one exact fit. The coefficient vector \((2,-1,0,0)^\mathsf{T}\) gives \(2x_1-x_2=y\), so every residual is zero.
Step 3: Write a different exact fit. The vector \((0,-1,2,0)^\mathsf{T}\) gives \(2x_3-x_2=2(x_1+x_2)-x_2=2x_1+x_2\), which is not the stated response. This tempting substitution is incorrect. A valid alternative uses \(x_4=2x_1\): the vector \((0,-1,0,1)^\mathsf{T}\) gives \(-x_2+x_4=-x_2+2x_1=y\).
Check. Both valid vectors yield zero residual sum of squares, but assign the effect to different columns. Ordinary least squares cannot distinguish these representations from these data. The common mistake is to interpret one coefficient representation returned by a software routine as uniquely established by the observations.
What Ridge, Lasso, and Elastic Net Do
| Method | Penalty component | High-dimensional behavior to inspect |
|---|---|---|
| Ridge | \(\lambda\|\beta\|_2^2/2\) | Shrinks slopes toward zero; ordinarily retains all nonzero predictor columns rather than performing exact selection. |
| Lasso | \(\lambda\|\beta\|_1\) | Can set slopes exactly to zero; among redundant predictors, the selected coefficient representation may be nonunique or change with the data. |
| Elastic net | A mixture of the squared and absolute-value penalties | Combines shrinkage and possible exact zeros; its quadratic component can help stabilize solutions with correlated or redundant columns. |
These descriptions concern coefficient behavior, not a claim that one method always predicts better. The appropriate penalty and tuning value depend on the prediction objective and the data. A tuned penalty can resolve the optimization problem according to its objective, but it cannot identify a causal effect or guarantee reliable estimates for every predictor.
Worked Example: Computing Ridge and Lasso Fits With More Predictors Than Rows
There are four observations and five predictor columns. Three columns are \(x_1=(1,-1,1,-1)^\mathsf{T}\), \(x_2=(1,1,-1,-1)^\mathsf{T}\), and \(x_3=(1,-1,-1,1)^\mathsf{T}\); the remaining two columns are zero. The response is \(y=2x_1+x_2\). There is an unpenalized intercept, which is zero for these centered columns and response. Under the stated objective, calculate the ridge and lasso slopes at \(\lambda=0.5\).
Step 1: Check the design. Here \(p=5>n=4\). The first three columns are mutually orthogonal and each has squared length 4; the last two columns are zero. The ordinary least squares coefficients along the first three columns are therefore \(c_1=x_1^\mathsf{T}y/4=2\), \(c_2=x_2^\mathsf{T}y/4=1\), and \(c_3=x_3^\mathsf{T}y/4=0\). The zero columns have no effect on the fitted values.
Step 2: Compute ridge. For a column with squared length \(n=4\), its normalized squared-error contribution has curvature 1 in its coefficient. The ridge objective for that coefficient is, up to a constant, \((\beta_j-c_j)^2/2+\lambda\beta_j^2/2\). Differentiating gives \(\beta_j-c_j+\lambda\beta_j=0\), so \(\beta_j=c_j/(1+\lambda)\). The three slopes are \(2/1.5=1.3333\), \(1/1.5=0.6667\), and \(0/1.5=0\). The two zero columns also have coefficient zero because any nonzero coefficient would increase the penalty without changing predictions.
Step 3: Compute lasso. For each orthogonal column, the coefficient minimizes \((\beta_j-c_j)^2/2+\lambda|\beta_j|\), giving the soft-thresholded value \(\operatorname{sign}(c_j)\max(|c_j|-\lambda,0)\). The slopes are \(1.5\), \(0.5\), and \(0\); the zero columns again have zero coefficients.
Check and common mistake. The lasso solution has two nonzero slopes, while ridge has two nonzero slopes here as well but gives different shrunk values. Lasso’s exact zeros arise from its absolute-value penalty, not from having fewer predictors than observations. Do not claim that ridge performs automatic selection just because its coefficients are smaller.
A Justification for Uniqueness and Its Limits
The ridge uniqueness result follows from the curvature of the objective. For clarity, suppose the intercept has been separated from the slope calculation, and consider ridge with \(\lambda>0\). The part of the objective depending on \(\beta\) has Hessian
For any nonzero vector \(z\), its quadratic form is \(z^\mathsf{T}X^\mathsf{T}Xz/n+\lambda z^\mathsf{T}z =\|Xz\|_2^2/n+\lambda\|z\|_2^2>0\). Thus the Hessian is positive definite, so the objective is strictly convex in the slopes and has at most one minimizer. The positive penalty also makes the objective grow as the slope vector grows, so a minimizer exists. This proves uniqueness even if \(X^\mathsf{T}X\) alone is singular.
For lasso, the absolute-value penalty is convex but not strictly convex everywhere. If two coefficient vectors fit equally well and the penalty does not distinguish them, both can minimize the objective. Elastic net with a positive squared-penalty component has the same strict-convexity argument as ridge. An unpenalized intercept is a separate parameter; the usual model must still be specified sensibly, including how intercepts and any redundant columns are represented.
Worked Example: How Elastic Net Splits a Redundant Signal
There are \(n=2\) observations and \(p=3\) slopes. The first two predictor columns are both \(x=(1,-1)^\mathsf{T}\); the third is zero. The response is \(y=2x\). Use an unpenalized intercept, \(\alpha=0.5\), and \(\lambda=1\) in the stated objective. Find the elastic-net slopes.
Step 1: Reduce the fit to the combined coefficient. Let the first two slopes be \(\beta_1,\beta_2\), and set \(s=\beta_1+\beta_2\). The fitted values from these columns are \(sx\). Since \(x\) and \(y\) are centered, the optimal intercept is zero. The third, zero-valued column has optimal coefficient zero because it cannot improve fit and is penalized.
Step 2: Write the objective in terms of \(s\). The residual is \((2-s)x\), whose squared length is \(2(2-s)^2\). With \(n=2\), the loss is \((2-s)^2/2\). For a fixed positive \(s\), the lasso component is minimized when both slopes are nonnegative, and the ridge component is minimized by splitting the sum evenly: \(\beta_1=\beta_2=s/2\). The resulting penalty is \(s^2/8+s/2\). Thus the objective is \((2-s)^2/2+s^2/8+s/2\).
Step 3: Minimize and check. Differentiating gives \(s-2+s/4+1/2=0\), so \(1.25s=1.5\) and \(s=1.2\). Hence \(\beta_1=\beta_2=0.6\), and \(\beta_3=0\). The fitted values are \(1.2x=(1.2,-1.2)^\mathsf{T}\); the residuals are \((0.8,-0.8)^\mathsf{T}\).
Common mistake avoided. Ordinary least squares can divide the total coefficient of 2 between the identical columns in infinitely many ways. The elastic-net penalty here prefers an even split, but that split is a consequence of the chosen penalty and scaling, not evidence that the two predictors have separately measured effects.
Building and Inspecting the Model Matrix in R
In an R workflow, the fitting function receives a numeric predictor matrix, often called \(x\), and a response vector. Before fitting, inspect the matrix dimensions, column names, and any constant or duplicate columns. If a factor is expanded into indicators, count those resulting columns. For example, a factor with several observed levels may contribute multiple columns, depending on the encoding and whether an intercept is included. A redundant full set of indicators together with an intercept can create exact linear dependence; use an appropriate encoding rather than treating the resulting coefficients as independent effects.
Standardization changes the scale on which a penalty acts. Check whether the fitting procedure standardizes predictors by default and whether the coefficients you inspect are on standardized or original scales. Do not standardize the response or predictors using information from held-out observations. For resampling, repeat any learned preprocessing inside each training fold so that validation observations do not influence the fitted transformations.
At the selected tuning value, inspect more than a printed list of nonzero coefficients. Record the dimensions of the training matrix, the number of nonzero slopes, the intercept and coefficient scale, and whether columns are duplicates, nearly redundant, or indicators for rare levels. A sparse lasso fit is a compact representation; it is not, by itself, proof that the selected predictors are uniquely important. Ridge may be preferable when prediction benefits from retaining many weak or correlated signals, while lasso or elastic net may be useful when a smaller representation is desired. These are modeling choices to evaluate against the stated objective, not universal rules.
Interpretation and Common Exam Traps
- Confusing a unique solution with identified effects. Ridge can return one unique coefficient vector even when many predictor combinations explain the data similarly. Uniqueness comes from the penalty, not necessarily from independent information in the sample.
- Counting source variables instead of columns. The predictor count must include indicators, interactions, and other model-matrix expansions. A formula with few named terms can still yield \(p>n\).
- Assuming lasso always selects a stable set. With highly correlated predictors, small changes in observations can alter which column receives a nonzero coefficient. Exact selection is a property of one fitted solution, not a guarantee of repeatability.
- Ignoring the intercept and encoding. Whether an intercept is included and how categorical levels are encoded affect the design matrix. A redundant encoding can introduce dependencies even before \(p\) exceeds \(n\).
- Reading penalized coefficients as ordinary least squares effects. Penalties deliberately trade some coefficient accuracy for a constrained fit. Interpret the estimates in light of shrinkage, scaling, and predictor dependence.
Check Your Understanding
Work each problem from the stated dimensions and penalty behavior before checking the answer.
- A training matrix has 8 observations and 11 predictor columns, including all encoded indicators. What is the largest possible rank of its predictor matrix, and what minimum number of independent coefficient directions must be unidentifiable to ordinary least squares? (Answer: Rank at most 8; at least 3 null-space directions.)
- Under the objective in this tutorial, a ridge model has \(\lambda=1\), and a standardized orthogonal predictor has unpenalized slope \(c=3\). What is its ridge slope? (Answer: \(3/(1+1)=1.5\).)
- For an orthogonal predictor with unpenalized slope \(c=0.7\), what lasso slope results at \(\lambda=0.5\)? (Answer: \(0.2\).)
- Two identical nonzero columns have a required combined coefficient \(s=1.8\). With a positive ridge penalty and no other constraints, how does the squared-penalty component divide this fixed sum between the two coefficients? (Answer: Equally, \(0.9\) each, because this minimizes the sum of their squared coefficients.)
- A model has 30 observations, 25 numeric predictors, and a categorical predictor expanded into 8 indicator columns; the final model matrix includes all 8 indicators and an intercept. Does the number of slopes exceed the number of observations, and what is the total column count including the intercept? (Answer: Yes; there are 33 slopes, so \(33>30\), and 34 total columns including the intercept.)