Selected Variables Are Not Unique Importance Rankings

Selection Is Not a Ranking

A lasso model can identify a set of predictors with nonzero fitted coefficients. That set is useful for describing the particular fitted model, but it is not, by itself, a ranking of which variables matter most. Even when coefficients are available, their sizes can reflect scaling, correlations among predictors, and the way categorical variables are encoded—not just predictive contribution.

This distinction matters when explaining a model. “The lasso retained these variables” is a statement about the fitted model at a chosen penalty. “This variable is more important than that one” requires a definition of importance and a comparison method. Without those additional choices, a single ordered list may suggest more certainty than the data support. This tutorial is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: The selected set (or active set) of a fitted lasso model is the set of predictors whose fitted coefficients are nonzero. A variable-importance ranking orders variables according to a specified measure of their contribution to a stated predictive goal. Selection describes which coefficients are nonzero in one fit; importance requires a measure, a comparison, and a defined context.

For example, a coefficient’s absolute value is one possible quantity to inspect, but it is not a universal importance measure. It depends on the predictor’s scale and on how the model allocates shared predictive information among related predictors. Moreover, a categorical predictor may be represented by several indicator columns, so ranking those coefficients individually need not rank the factor as a whole.

Correlated Predictors Share Predictive Information

Suppose a linear model uses centered predictors \(x_1,\ldots,x_p\) and response \(y\). A common lasso objective, for a fixed penalty \(\lambda>0\), is

\frac{1}{2n}\sum_{i=1}^{n}\left(y_i-\beta_0-\sum_{j=1}^{p}x_{ij}\beta_j\right)^2 +\lambda\sum_{j=1}^{p}|\beta_j|.

Here \(\beta_0\) is the intercept and the penalty applies to the slopes. If two predictors are strongly correlated, their contributions to the fitted values can overlap. The data may therefore provide limited information about how much of that shared contribution should be assigned to each coefficient. A small change in the observations can alter that allocation, and sometimes which predictor is selected.

Standardizing predictors can make coefficient magnitudes more comparable across predictor units, but it does not remove correlation or make a coefficient ranking uniquely meaningful. Even when all predictors are standardized, a coefficient is a conditional model coefficient: it describes the fitted change associated with that predictor while the other included predictors are held fixed. When predictors move together in the data, that comparison may be poorly supported.

Worked Example: Two Nearly Redundant Predictors

Consider two centered, standardized predictors with correlation \(0.90\). Suppose their sample predictor-response inner products are \(c_1=0.95\) and \(c_2=0.94\), using the convention \(c_j=x_j^\mathsf{T}y/n\). Fit the lasso with \(\lambda=0.50\) under the objective above, and assume both fitted coefficients are positive. Find the coefficients and assess whether their ordering is a stable importance ranking.

Method. For standardized columns with correlation \(\rho=0.90\), the predictor Gram matrix, divided by \(n\), has diagonal entries 1 and off-diagonal entries 0.90. For positive active coefficients, the lasso first-order conditions are

\beta_1+0.90\beta_2=c_1-\lambda=0.45,\qquad 0.90\beta_1+\beta_2=c_2-\lambda=0.44.

Subtract the second equation from the first:

0.10(\beta_1-\beta_2)=0.01, \qquad \beta_1-\beta_2=0.10.

Add the two equations:

1.90(\beta_1+\beta_2)=0.89, \qquad \beta_1+\beta_2=\frac{0.89}{1.90}=0.468421\ldots.

Combining the sum and difference gives

\beta_1=\frac{0.468421\ldots+0.10}{2}=0.284211\ldots,\qquad \beta_2=\frac{0.468421\ldots-0.10}{2}=0.184211\ldots.

Both values are positive, as assumed, and substitution checks the conditions: \(0.284211+0.90(0.184211)\approx0.45\) and \(0.90(0.284211)+0.184211\approx0.44\). Rounded to three decimals, the fitted coefficients are \(0.284\) and \(0.184\). Predictor 1 has the larger coefficient in this fit, but that small difference is not a universal importance ranking. If the two inner products were reversed, the coefficient difference would reverse too. The calculation also shows how the difference is amplified: \(\beta_1-\beta_2=(c_1-c_2)/(1-\rho)\). Here a difference of \(0.01\) in the inner products becomes a difference of \(0.10\) in the coefficients.

Common mistake avoided: Treating the larger coefficient as evidence that predictor 1 is definitively more important. The calculation establishes an ordering for this fitted model and penalty, not an invariant ranking.

Why Exact Duplicates Can Have Multiple Allocations

The limiting case of correlation is exact duplication. If two predictor columns are identical, predictions depend on the sum of their coefficients, not on how that sum is split. For coefficients with the same sign, the lasso penalty is also unchanged by splitting that sum. Thus multiple coefficient vectors can give both the same fitted values and the same objective value.

The justification follows directly from the objective. If the shared predictor is \(x\), then the fitted contribution from two duplicate columns is \(x\beta_1+x\beta_2=x(\beta_1+\beta_2)\). For a fixed nonnegative sum \(s=\beta_1+\beta_2\), any nonnegative split has penalty \(|\beta_1|+|\beta_2|=s\). The objective can choose the sum without uniquely choosing the split.

Worked Example: One Fitted Contribution, Different Coefficient Rankings

Two standardized predictor columns are exact duplicates. In the one-dimensional fitted direction, the response has projection 2. After dropping the residual component perpendicular to that direction, the lasso objective is

Q(\beta_1,\beta_2) =\frac{1}{2}(2-\beta_1-\beta_2)^2 +0.50\left(|\beta_1|+|\beta_2|\right).

Find the optimal total coefficient and compare two optimal allocations.

Method. Let \(s=\beta_1+\beta_2\). For a fixed \(s\geq0\), the smallest possible penalty is \(0.50s\), attained whenever both coefficients are nonnegative. The objective as a function of the total is therefore

q(s)=\frac{1}{2}(2-s)^2+0.50s.

Differentiate: \(q'(s)=s-2+0.50=s-1.50\). The minimum occurs at \(s=1.50\); the second derivative is \(q''(s)=1>0\), confirming a minimum. The allocations \((1.50,0)\) and \((0.75,0.75)\) both have total 1.50. For the first, \(Q=\frac12(0.50)^2+0.50(1.50)=0.125+0.75=0.875\). For the second, \(Q=\frac12(0.50)^2+0.50(0.75+0.75)=0.125+0.75=0.875\). Their fitted contributions are identical because each uses the same total coefficient on the duplicate direction.

Check. Both solutions have the same residual term, the same penalty, and the same objective value. Yet the first allocation selects one column and the second assigns equal nonzero coefficients to both. The selected variables and their coefficient ranking are therefore not unique in this example.

Common mistake avoided: Assuming that each nonzero lasso coefficient identifies a uniquely necessary predictor. With duplicate columns, the model can make the same predictions using different allocations.

Indicator Coefficients Depend on the Reference Level

A categorical predictor with multiple levels is commonly represented by indicator columns. With a reference level, each included indicator coefficient describes a contrast with that reference, conditional on the other model predictors. It is not a standalone measure of the whole factor’s predictive importance.

Changing the reference level changes which contrasts the coefficients represent. Even if a fixed set of fitted level predictions is expressed exactly under two codings, the absolute coefficient sizes—and therefore their apparent ranking—can differ. Further, refitting a penalized model under a different coding can change the fit itself: the lasso penalty acts on the individual coefficient representation, which can differ across codings.

Worked Example: Releveling a Three-Level Factor

A hypothetical model uses a categorical predictor with levels A, B, and C. With A as the reference, the fitted equation for the relevant part of the linear predictor is

\widehat{\eta}=10+0.8I(B)-0.2I(C),

where \(I(B)\) and \(I(C)\) equal 1 for the named level and 0 otherwise. Calculate the fitted value for each level, then rewrite the same predictions using B as the reference. Does the absolute coefficient ranking stay the same?

Method. For A, both indicators are zero, giving \(\widehat{\eta}_A=10\). For B, \(I(B)=1\) and \(I(C)=0\), giving \(\widehat{\eta}_B=10+0.8=10.8\). For C, \(I(B)=0\) and \(I(C)=1\), giving \(\widehat{\eta}_C=10-0.2=9.8\).

With B as the reference, the intercept is the fitted value for B, \(10.8\). The contrast for A is \(10-10.8=-0.8\), and the contrast for C is \(9.8-10.8=-1.0\). Thus the equivalent equation is

\widehat{\eta}=10.8-0.8I(A)-1.0I(C).

Checking the new equation gives A: \(10.8-0.8=10\); B: \(10.8\); C: \(10.8-1.0=9.8\). The predictions agree exactly. But with A as reference, the absolute indicator coefficients rank B (\(0.8\)) above C (\(0.2\)); with B as reference, C (\(1.0\)) is above A (\(0.8\)). The apparent ordering changes because the coefficients represent different contrasts.

Common mistake avoided: Ranking individual indicators as if each were a coding-invariant measure of the factor’s importance. For interpretation, state the reference level and the contrast. If the question concerns the factor as a whole, compare it as a group using a clearly defined method.

Define Importance Before Comparing Predictors

There is no single ranking that answers every predictive question. A comparison might ask how a model’s predictions change when a feature is removed, how much predictive error changes under a specified perturbation, or how a group of indicators contributes jointly. These questions define different quantities and may produce different orderings. The important discipline is to identify the target, comparison, and data context before interpreting the result.

When explaining a lasso fit, distinguish at least three statements:

Selection: a coefficient is nonzero in this fitted model at this penalty.
Coefficient size: the fitted conditional association has a particular value on the stated predictor and outcome scales.
Importance: a specified measure assigns a contribution or ranking for a defined predictive goal.

These statements are not interchangeable. In particular, selection does not prove that an unselected predictor contains no predictive information; correlated predictors can substitute for one another. Nor does a larger standardized coefficient necessarily mean a predictor would produce a larger change in predictive performance if removed. A useful report states how predictors were coded and scaled, notes important correlations or grouped indicators, and avoids claiming a unique ranking when the chosen method does not support one.

Key takeaway: A lasso active set is a property of a particular fit; it is not an intrinsic importance list. Correlation can make coefficient allocation unstable or non-unique, and factor coding can change individual indicator magnitudes while preserving fitted level predictions. Any importance ranking must state its measure and context.

Check Your Understanding

For each problem, distinguish what the fitted coefficients establish from what would require a separately defined importance measure.

A standardized lasso fit has coefficients \(0.40\) and \(0.15\) for two predictors correlated at \(0.92\). Both are selected. What can be said about selection and coefficient size, and what cannot be concluded from those facts alone? (Answer: Both predictors are selected in this fit, and the first has the larger absolute fitted coefficient. This alone does not establish a unique ranking of predictive importance.)
Two duplicate predictors have a fitted total coefficient of \(1.2\), with both coefficients constrained to be nonnegative. Give two allocations with the same predictions and the same lasso penalty. (Answer: \((1.2,0)\) and \((0.6,0.6)\); both have total coefficient \(1.2\) and penalty contribution \(1.2\lambda\).)
A three-level factor has fitted level values A \(=7\), B \(=9\), and C \(=8\). With A as reference, state the indicator coefficients. With B as reference, state the intercept and the coefficients for A and C. (Answer: With A as reference, the coefficients are B \(=2\), C \(=1\), and the intercept is 7. With B as reference, the intercept is 9, the A coefficient is \(-2\), and the C coefficient is \(-1\). The fitted level values are unchanged.)
For standardized correlated predictors, \(\rho=0.80\), \(c_1=0.90\), \(c_2=0.88\), and \(\lambda=0.40\). Assuming both coefficients are positive, solve the lasso active equations and explain why the ordering is not a universal ranking. (Answer: The equations have right sides \(0.50\) and \(0.48\). Thus \(0.20(\beta_1-\beta_2)=0.02\), so the difference is \(0.10\); \(1.80(\beta_1+\beta_2)=0.98\), so the sum is \(0.544444\ldots\). Therefore \(\beta_1=0.322222\ldots\) and \(\beta_2=0.222222\ldots\), both positive. This is the ordering for this fit; reversing the two inner products reverses the coefficient ordering.)
A report ranks one factor’s levels by the absolute values of its dummy coefficients, but does not state the reference category. What information is missing, and why does it matter? (Answer: The reference category and the coding must be stated. Each dummy coefficient is a contrast with the reference, so changing the reference can change coefficient magnitudes and their ranking even when the fitted level predictions are the same.)

What You'll Learn