From a Chosen Penalty to a Selected Predictor Set
A fitted penalized regression model may contain many candidate predictors, yet only some have nonzero coefficients at the penalty chosen for use. The distinction matters when summarizing a model: the list of variables supplied to the fitting procedure is not automatically the list of variables selected by it. Selection is a property of the fitted coefficient vector at a specified penalty.
This tutorial focuses on extracting that vector in R and reporting its nonzero coefficients. The same workflow applies whether the penalty value was chosen as lambda.min, lambda.1se, or another value selected through an established tuning process. The choice of penalty is taken as given here; the task is to describe the fitted model at that value accurately.
For a numeric predictor, its modeled column ordinarily corresponds to that predictor. A factor, however, is commonly represented by several indicator columns in a model matrix. In that case, the active set is first defined at the column level: a factor indicator belongs to the active set if its coefficient is nonzero. It is not safe to silently replace those indicator-level results with a claim that the entire factor was selected.
Define the Coefficient Vector Precisely
For a lasso fit, the fitted coefficients depend on both the data and the penalty. In the least-squares case, a common objective is the residual sum of squares, scaled by the sample size, plus an L1 penalty. Here, \(y_i\) is the response, \(x_{ij}\) is predictor column \(j\) for observation \(i\), \(\beta_0\) is the intercept, and \(\beta_j\) is its coefficient.
The intercept is unpenalized in this expression. The lasso penalty can set some \(\beta_j\) values to exactly zero. In elastic-net regression, the L1 component can also produce exact zeros when its mixing parameter is positive. Ridge regression uses an L2 penalty instead and generally does not produce exact zeros, so calling its nonzero coefficient columns “selected” can be misleading.
This objective describes a particular least-squares lasso setup; it is not the fitting objective for every response type or penalty. For example, a penalized logistic regression uses a likelihood-based loss rather than squared error. The extraction principle remains the same: inspect the fitted predictor coefficients at the chosen penalty, while keeping the intercept separate.
Why Lasso Coefficients Can Be Exactly Zero
A short optimality argument explains why checking for zero coefficients is useful for lasso. Consider the least-squares lasso objective above, with the intercept handled separately. For a fixed coefficient vector, define the residual for observation \(i\) as \(r_i=y_i-\beta_0-\sum_k x_{ik}\beta_k\). When considering coefficient \(j\), the derivative of the scaled squared-error term with respect to \(\beta_j\) is \(-\frac{1}{n}\sum_i x_{ij}r_i\).
The absolute-value penalty is not differentiable at zero, but its subgradient at zero is every value between \(-1\) and \(1\). At an optimum where \(\beta_j=0\), the zero-coefficient condition is therefore:
In words, if the predictor’s residual association is not large enough to overcome the penalty, zero satisfies the optimality condition. When \(\beta_j\ne 0\), the corresponding condition is \(\frac{1}{n}\sum_i x_{ij}r_i=\lambda\,\operatorname{sign}(\beta_j)\). This explains the lasso’s sparse solutions. The conditions depend on the stated least-squares objective and the predictor scaling used in the fit; they are not a formula for interpreting coefficients from every penalized model.
Extracting the Active Set in R
With glmnet, a cross-validated fit from cv.glmnet stores the fitted model and the selected penalty values. To inspect coefficients at lambda.1se, for example, request them explicitly with coef(cvfit, s = "lambda.1se"). For a manually chosen numeric value, supply that value as s instead. The key is to request the coefficient vector at the penalty you intend to report, rather than relying on whichever value happens to be the default.
Record whether the fit is being summarized at lambda.min, lambda.1se, or a specified numeric lambda. Use that same choice in the coefficient extraction.
Convert the returned sparse coefficient matrix to an ordinary matrix if convenient. Keep the coefficient names; they identify numeric columns and factor indicators.
Remove the entry named "(Intercept)" before identifying predictor columns. Do not count it as a selected predictor.
For a lasso or elastic-net fit, identify nonzero coefficient entries. If a tolerance is used for numerical reporting, state it and apply it to every coefficient consistently.
List numeric predictors and factor indicator columns separately, and state the candidate columns against which the active set was identified.
One compact R pattern is: coef_mat <- as.matrix(coef(cvfit, s = "lambda.1se")); b <- coef_mat[, 1]; b <- b[names(b) != "(Intercept)"]; active <- names(b)[b != 0]. This example assumes that the requested coefficient result has one column. For a numeric penalty, replace "lambda.1se" with the chosen numeric value. The resulting names are model-matrix column names, not necessarily the names of original data-frame variables.
glmnet may standardize predictor columns during fitting, but the coefficient vector returned for prediction is expressed on the original predictor scales. For identifying zeros, focus on the returned coefficients from the fitted model and use its column names. Keep the preprocessing and model-matrix specification that produced those columns so that the report can be interpreted correctly.
Worked Examples
Worked Example: Numeric Predictors at lambda.1se
A hypothetical insurer fits a lasso model for a continuous claim-cost outcome using four numeric candidate columns: vehicle age, annual distance, prior claims, and a safety score. The cross-validation procedure has already designated lambda.1se = 0.08. At that value, the returned coefficients are: intercept 2.40, vehicle age 0.00, annual distance -0.12, prior claims 0.35, and safety score 0.00. Identify the active set and state its size.
Method. Extract the vector at the designated penalty, exclude the intercept, and identify the predictor coefficients that are nonzero. The stated values are treated as exact for this example.
Step 1: Exclude the intercept. The intercept is 2.40, but it is not a predictor. The four predictor coefficients are 0.00, -0.12, 0.35, and 0.00, in the order given.
Step 2: Check each candidate column. Vehicle age has coefficient 0.00, so it is inactive. Annual distance has coefficient -0.12, so it is active. Prior claims has coefficient 0.35, so it is active. Safety score has coefficient 0.00, so it is inactive.
Step 3: Report the result. The active set is {annual distance, prior claims}, with size 2. The candidate set still contains all four columns; the fit has not erased the fact that vehicle age and safety score were considered.
Check. There are four candidate predictor columns and two nonzero predictor coefficients, so the active-set size is 2, not 3. A common mistake is to count the intercept as an active predictor or to call all four candidates selected. Neither matches the coefficient-based definition.
Worked Example: Reporting Factor Indicators Without Overclaiming
A hypothetical service model uses numeric variables account age and prior service calls, plus a three-level factor channel with levels Branch, Web, and Partner. Branch is the reference level, so the model matrix contains indicators channelWeb and channelPartner. At the chosen penalty, the non-intercept coefficients are: account age 0.00, prior service calls 0.20, channelWeb -0.04, and channelPartner 0.00. State the active set at the indicator-column level and explain what can be said about channel.
Method. Use the model-matrix column names as the units of extraction. A reference level is represented through the coding convention and does not have its own indicator coefficient.
Step 1: Check the numeric columns. Account age has coefficient 0.00 and is inactive. Prior service calls has coefficient 0.20 and is active.
Step 2: Check the factor indicators separately. channelWeb has coefficient -0.04 and is active. channelPartner has coefficient 0.00 and is inactive.
Step 3: State the result precisely. The active set of modeled columns is {prior service calls, channelWeb}. The candidate set also included account age and channelPartner. The report can say that the Web indicator is nonzero relative to the Branch reference coding. It should not say that every channel level was selected, because channelPartner is zero and Branch has no separately estimated indicator.
Check. There are four candidate columns and two active columns. The word “channel” alone is ambiguous here: it could mean the original factor or either of its indicator columns. A common mistake is to count the factor as one selected column without stating the rule used to combine its indicators.
Worked Example: Applying a Numerical Tolerance Consistently
A hypothetical lasso model is being summarized from a printed coefficient table at a fixed penalty. The table reports three predictor coefficients as 0.00000002, 0.0030, and 0.00000000. The analyst decides in advance to treat absolute values below \(10^{-7}\) as zero for this report. Identify the active columns under that rule and distinguish that report from exact mathematical nonzero status.
Method. Apply the declared tolerance \(\varepsilon=10^{-7}\) to the absolute value of each predictor coefficient. A column is reported active if \(|\hat{\beta}_j|\geq\varepsilon\). This is a reporting convention for values that may be affected by numerical precision; it is not a change to the fitted model.
Step 1: Evaluate the first coefficient. \(|0.00000002|=2\times10^{-8}\), and \(2\times10^{-8}<10^{-7}\). Report it as zero under the stated rule.
Step 2: Evaluate the second coefficient. \(|0.0030|=0.0030\), and \(0.0030\geq10^{-7}\). Report this column as active.
Step 3: Evaluate the third coefficient. \(|0.00000000|=0\), and \(0<10^{-7}\). Report it as zero.
Result and check. Exactly one of the three columns is active under this reporting convention: the column with coefficient 0.0030. Without a tolerance, the first displayed number is mathematically nonzero as printed. A common mistake is to choose a tolerance after looking at the result, or to apply different cutoffs to different columns. State the rule and use it uniformly.
Report Selection So It Can Be Reproduced
A useful selection summary says what was fitted, at what penalty, what counted as zero, and what names the reported columns represent. For example, a concise report might state that a lasso model at lambda.1se had two active predictor columns out of four candidates, naming the two columns and identifying any factor indicators explicitly. If a numerical tolerance was used, include its value. If no tolerance was used, report that exact nonzero coefficients defined the active set.
| Report item | What to specify | Why it matters |
|---|---|---|
| Penalty value | lambda.min, lambda.1se, or the numeric lambda used | The active set can differ across penalty values |
| Candidate set | The model-matrix columns offered to the fit | Selection is defined relative to the columns considered |
| Zero rule | Exact zero or a stated tolerance | Printed rounding and numerical values can differ |
| Factor coding | Indicator names and reference level | A factor may be represented by multiple columns |
| Intercept | Reported separately from predictor selection | It is not a candidate predictor column |
Selection is a descriptive statement about one fitted model at one penalty. A nonzero coefficient does not by itself establish causation, practical importance, or stable selection in new samples. Likewise, a zero coefficient means the fitted penalized model assigns that column a zero coefficient under the specified fitting and coding choices; it does not prove that the underlying phenomenon is irrelevant.
The same care applies when reporting a factor. If a study needs a single selection decision for an entire factor, the rule for turning several indicator coefficients into one factor-level decision must be stated explicitly. Simply listing the factor name can hide which contrasts were active. The first, reproducible step is to report the indicator columns actually returned by the model.
Check Your Understanding
For each question, identify the active set using the specified rule and keep the intercept and candidate set distinct.
- A lasso fit at lambda.min has intercept 1.1 and predictor coefficients age 0, mileage -0.2, and score 0.6. Give the active set and its size. (Answer: {mileage, score}; size 2.)
- A factor has reference level East and indicators regionNorth and regionWest. At the chosen penalty their coefficients are 0.08 and 0. The numeric predictor price has coefficient 0. At the indicator-column level, what is active? (Answer: {regionNorth}; price and regionWest are inactive.)
- A model has five candidate predictor columns. After excluding the intercept, two coefficients are nonzero and three are zero. How many active columns are there, and how many candidate columns remain inactive? (Answer: 2 active and 3 inactive.)
- Using a stated tolerance of \(10^{-6}\), classify coefficients 0.0000004, -0.000002, and 0. What is the active set if the columns are named A, B, and C in that order? (Answer: {B}; A and C are treated as zero.)
- A report says “the three-level product factor was selected” because one of its two non-reference indicators is nonzero. What is the precise correction? (Answer: Report the active indicator by name and state the reference level; do not claim both indicators or the entire factor were selected without an explicit factor-level rule.)