Penalty Selection by Cross-Validation and Theory

Why Penalty Choice Depends on the Goal

The lasso penalty controls how strongly regression coefficients are shrunk toward zero. A larger penalty usually produces a sparser fitted model, while a smaller penalty may improve fit to the observed sample but retain more noise. In high-dimensional regression, the penalty is not just a numerical setting: its choice affects prediction, coefficient estimation, and which variables are selected.

Two common ways to choose the penalty answer different questions. Cross-validation estimates how well a fitted model predicts observations not used to fit it. Theoretical calibration instead chooses a penalty large enough, with high probability under specified assumptions, to dominate random score fluctuations and support statistical bounds. These goals can recommend different values without contradicting one another.

Model and Penalty Conventions

Consider the linear model \(y=X\beta_0+u\), where \(y\) is an \(n\)-vector of outcomes, \(X\) is an \(n\times p\) design matrix, \(\beta_0\) is the coefficient vector, and \(u\) is an error vector. For clarity, take the columns of \(X\) to be centered and scaled to have squared length \(n\), and leave any intercept unpenalized. The lasso estimate for a given \(\lambda>0\) minimizes the normalized squared-error loss plus an \(\ell_1\) penalty:

This normalization matters: numerical values of \(\lambda\) depend on the loss scaling and predictor scaling. A penalty value from software using a differently normalized objective cannot be transferred directly without conversion. The discussion below uses this displayed objective throughout.

Cross-Validation for Prediction

Partition the observations into \(K\) folds. For each candidate penalty \(\lambda\), fit the lasso on the observations outside fold \(k\), then compute mean squared prediction error on fold \(k\). If \(I_k\) is the set of indices in fold \(k\), \(n_k\) is its size, and \(\hat{\beta}^{(-k)}(\lambda)\) is the estimate from the other folds, the \(K\)-fold cross-validation criterion is

The selected penalty is often the candidate minimizing this criterion. Its purpose is to estimate predictive performance for new observations drawn from a relevant target population. The folds must reflect the data structure: for example, observations from the same individual should not be split across training and validation folds when the target is prediction for new individuals. Time-ordered data may require time-respecting splits rather than random folds.

All data-dependent preprocessing must be learned within each training fold. If predictor means and scales, variable screening, or imputation rules are computed using the full data before splitting, information from the validation observations can leak into the fit and make its measured error optimistic. The chosen penalty is also conditional on the candidate grid, fold design, outcome loss, and target population.

Theoretical Calibration From Score Fluctuations

A common theoretical calibration starts from the score at the true coefficient, \(X'u/n\). Suppose the design is treated as fixed and the errors are independent with \(u_i\sim N(0,\sigma^2)\). With columns scaled to squared length \(n\), each coordinate \(x_j'u/n\) is normally distributed with mean zero and variance \(\sigma^2/n\). The largest absolute coordinate is the relevant quantity because the lasso penalty acts on all \(p\) coefficients.

A Gaussian tail bound and a union bound give, for any \(t>0\),

Setting the right-hand side equal to a chosen probability level \(\delta\) gives the threshold \(t=\sigma\sqrt{2\log(2p/\delta)/n}\). The lasso basic inequality is controlled when \(\lambda\) is at least twice the score maximum. Thus a convenient sufficient calibration is the following.

The factor of two is tied to the displayed lasso objective and to a standard sufficient condition for the basic inequality. Other loss normalizations, error-tail assumptions, or penalty conventions change the formula. In practice \(\sigma\) is usually unknown and must be estimated or handled using a suitable alternative calibration; substituting an estimate adds uncertainty. The bound alone also does not guarantee accurate coefficient recovery: prediction and estimation bounds require design conditions such as restricted eigenvalue or compatibility conditions, and support recovery typically requires stronger conditions, including adequate signal strength.

Different Objectives, Different Penalties

Cross-validation and theoretical calibration can disagree because neither is designed to minimize the same quantity as the other. Cross-validation chooses among its candidates to minimize an estimated prediction loss. A theoretical penalty is often chosen to control random fluctuations and obtain a bound that holds with high probability. Such a bound is sufficient rather than necessarily sharp, and its constants can make the resulting penalty conservative for prediction.

Implications for Inference and Practice

If the objective is prediction, cross-validation is a natural tuning tool, provided the folds represent the intended prediction task and the validation procedure avoids leakage. If the objective is theoretical control of coefficient error or support recovery, penalty calibration must be considered alongside the relevant assumptions. The support-recovery conditions discussed in the earlier material are stronger than a prediction-risk objective; choosing a penalty by CV alone does not verify those conditions.

Neither choice makes ordinary post-selection uncertainty statements automatically valid. Cross-validation selects a penalty for predictive performance, not for valid coefficient-level inference. A theoretically calibrated penalty likewise does not, by itself, provide valid standard errors or confidence intervals after selection. Inference requires methods and assumptions designed for that purpose.

When using cross-validation in an econometric application, also match the fold design to the sampling structure. Random folds can be inappropriate when observations are clustered, repeated over time, or otherwise dependent. Use splits that respect the independent sampling units and the intended prediction target. If prediction across future periods is the goal, for example, randomly mixing future and past observations may not estimate the relevant forecasting performance.

Common Mistake

Wrong reasoning: “Cross-validation chose this penalty, so it is large enough for support recovery and inference.” The CV criterion only compares estimated prediction errors over the candidate grid. A good validation score does not establish score domination, design conditions, or the signal-strength conditions needed for support recovery.

Correction: State what the selected penalty is meant to accomplish. Use prediction-oriented validation to tune prediction performance, and assess the assumptions and calibration required for any theoretical claim separately. Do not treat either selection method as a substitute for a valid inferential procedure.