Lasso Selection Among Factor Indicators

Reference Coding and Lasso

A categorical predictor with several levels is usually represented in a regression model by a set of indicator variables. With an ordinary unpenalized fit, a change of reference level is a reparameterization: it changes the coefficient labels but not the fitted values. Lasso adds a penalty to individual coefficients, so the chosen coding matters. In particular, a lasso model can retain some non-reference levels of a factor and shrink other level indicators to zero.

The key is to distinguish the factor’s contrasts from the model’s fitted values. An indicator coefficient describes a contrast against the reference level, while a fitted value also contains the intercept and contributions from other predictors.

Definition: For a categorical predictor with \(K\) levels, reference-level (or treatment) coding chooses one level as the reference and creates \(K-1\) indicators. Each indicator equals 1 for its corresponding non-reference level and 0 otherwise. Its regression coefficient is the contrast in the linear predictor between that level and the reference, holding other predictors fixed.

For clarity, consider a Gaussian response and the lasso objective

\frac{1}{2n}\sum_{i=1}^{n}(y_i-\beta_0-\mathbf{x}_i^\mathsf{T}\boldsymbol{\gamma}-\sum_{j=1}^{K-1}\beta_j z_{ij})^2 +\lambda\left(\sum_{j=1}^{K-1}|\beta_j|+\text{penalties on any other penalized slopes}\right).

Here \(z_{ij}\) is the indicator for non-reference level \(j\), \(\beta_0\) is the intercept, and \(\lambda\geq 0\) controls the penalty. We assume the intercept is not penalized. The displayed scaling of the residual term is a convention; software can use an equivalent scaling with a correspondingly different value of \(\lambda\). Unless stated otherwise, examples below use unscaled indicators and the displayed objective.

What the Coefficients Mean

Suppose a factor has reference level A and non-reference levels B and C. Its indicators are \(z_B\) and \(z_C\), with coefficients \(\beta_B\) and \(\beta_C\). The factor’s contrasts relative to A are \(0\), \(\beta_B\), and \(\beta_C\). For observations agreeing on all other predictors, the fitted values apart from the common contributions of those predictors are \(\beta_0\), \(\beta_0+\beta_B\), and \(\beta_0+\beta_C\). Thus, \(\beta_0\) is not itself the factor’s contribution.

If \(\widehat{\beta}_C=0\), the fitted linear predictor for C equals that for the reference level A when all other predictors are held fixed. This does not say that the factor has no effect: B may still differ from A. It also does not establish that the true population effect is exactly zero. It says that the fitted lasso solution has shrunk that particular contrast to zero at the chosen penalty and coding.

Worked Example: Interpreting Selected Indicators

A hypothetical cost model uses service zone with North as the reference. At a specified profile, the common baseline fitted value for North—including the intercept and all other predictor contributions—is 5.4 thousand dollars. The lasso coefficients are 0.8 for Central, 0 for East, and \(-0.3\) for South. Find the fitted value for each zone and interpret which indicators are selected. Assume a Gaussian model with an identity link, and that the stated common baseline is fixed across zones.

Method. Add each zone’s contrast to the common baseline. The factor contrasts are 0 for North, 0.8 for Central, 0 for East, and \(-0.3\) for South.

North: \(5.4+0=5.4\) thousand dollars. Central: \(5.4+0.8=6.2\) thousand dollars. East: \(5.4+0=5.4\) thousand dollars. South: \(5.4-0.3=5.1\) thousand dollars.

The Central and South indicators have nonzero coefficients, so those contrasts are selected. East’s coefficient is zero, so the model gives East and North the same fitted value at this fixed profile. The mistake to avoid is treating the intercept alone as the common baseline when other predictors contribute to the prediction. Here the problem explicitly supplies their combined baseline contribution.

Check. Each fitted value equals 5.4 thousand dollars plus its stated contrast; the non-reference contrasts are not added to one another.

Why Some Levels Can Be Selected and Others Shrunk to Zero

Under this lasso formulation, each indicator coefficient receives its own absolute-value penalty. The factor is not automatically treated as one indivisible group. If the data support a sufficiently large conditional contrast for one level, its coefficient can be nonzero; another contrast can be shrunk to zero. The indicators can be correlated through the intercept and the other columns of the design matrix, so “separate penalties” does not mean that each coefficient is estimated in isolation.

The optimality conditions make this distinction precise. Let \(r_i=y_i-\widehat{y}_i\) be the residual at a fitted solution, and consider one indicator \(z_j\) with coefficient \(\beta_j\). For the objective with residual term \(\frac{1}{2n}\sum_i r_i^2\) and penalty \(\lambda|\beta_j|\), the derivative of the residual term with respect to \(\beta_j\) is \(-\frac{1}{n}\sum_i z_{ij}r_i\). At a nonzero coefficient, the penalty derivative is \(\lambda\operatorname{sign}(\beta_j)\). At zero, the absolute-value penalty has subgradients from \(-\lambda\) to \(\lambda\).

\begin{aligned} \widehat{\beta}_j\ne 0 &\quad\Longrightarrow\quad \frac{1}{n}\sum_i z_{ij}r_i =\lambda\operatorname{sign}(\widehat{\beta}_j),\\ \widehat{\beta}_j=0 &\quad\Longleftrightarrow\quad \left|\frac{1}{n}\sum_i z_{ij}r_i\right|\leq\lambda, \end{aligned}

provided the intercept and all other coefficients also satisfy their own optimality conditions. The quantity \(\frac{1}{n}\sum_i z_{ij}r_i\) is the indicator’s residual association at the candidate fit. If its magnitude is no greater than the penalty threshold, setting that coefficient to zero is optimal. This explains why a level can be dropped even when the factor as a whole remains useful.

Justification: For a nonzero coefficient, differentiating the objective with respect to that coefficient and setting the derivative to zero gives the first condition. At zero, \(|\beta_j|\) has subgradient interval \([-1,1]\); multiplying by \(\lambda\) means the residual-term derivative can be balanced by the penalty exactly when its magnitude is at most \(\lambda\). These are the lasso subgradient conditions for that coordinate.

Worked Example: Checking a Partially Selected Factor

Consider three observations, one in each of levels A, B, and C, with responses 0, 4, and 0.5. Use A as the reference, fit an intercept and indicators \(z_B,z_C\), and minimize \(\frac{1}{2}\sum_i(y_i-\widehat{y}_i)^2+|b|+|c|\). Determine whether \(\widehat{\beta}_0=0.75,\widehat{b}=2.25,\widehat{c}=0\) satisfies the lasso conditions. Assume no other predictors.

Method. Compute fitted values and residuals, then check the intercept condition, the active-coefficient condition, and the zero-coefficient condition. Here \(\lambda=1\), because the penalty coefficient on each absolute value is 1.

The fitted values are A: \(0.75\), B: \(0.75+2.25=3\), and C: \(0.75+0=0.75\). Using residual \(r=y-\widehat{y}\), the residuals are A: \(0-0.75=-0.75\), B: \(4-3=1\), and C: \(0.5-0.75=-0.25\).

The intercept is unpenalized, so its derivative condition is that the residuals sum to zero: \(-0.75+1-0.25=0\). For the positive coefficient \(b=2.25\), its indicator-weighted residual sum is 1, equal to \(\lambda\operatorname{sign}(b)=1\). For \(c=0\), the absolute indicator-weighted residual sum is \(|-0.25|=0.25\), which is at most \(\lambda=1\). Therefore all three conditions hold. Because the objective is convex, this candidate satisfies the global lasso optimality conditions.

For a numerical check, the residual sum of squares is \((-0.75)^2+1^2+(-0.25)^2=0.5625+1+0.0625=1.625\). The objective is \(\frac{1}{2}(1.625)+|2.25|+|0|=0.8125+2.25=3.0625\). The B indicator is selected, while C’s indicator is zero. The common mistake is to require every non-reference level to be selected whenever the factor has any selected level.

Coding and Scaling Affect the Penalty

Reference choice matters because the penalty is applied to the coefficients in the chosen parameterization. Consider fitted factor-level contributions \(0,2,0\) for A, B, and C, with an intercept of 10 and no other predictors. With A as reference, the coefficients are \(\beta_0=10,\beta_B=2,\beta_C=0\). The indicator penalty is \(|2|+|0|=2\).

Now use B as the reference. To preserve the same fitted values, the intercept becomes 12, and the contrasts are \(-2\) for A and \(-2\) for C. The indicator penalty is then \(|-2|+|-2|=4\). Both coefficient sets describe the same three fitted values, but the lasso assigns them different penalty amounts. Consequently, changing the reference level can change which fitted solution is preferred; it is not merely a harmless relabeling once the penalty is imposed.

Scaling is another part of the parameterization. If an indicator \(z\) is centered and divided by its standard deviation \(s\), then \(z=(z-\bar z)/s\) in the scaled design. A coefficient \(\theta\) on the scaled column corresponds to an original-scale coefficient \(\beta=\theta/s\), with an intercept adjustment for centering. A penalty \(\lambda|\theta|\) is therefore \(\lambda s|\beta|\) when expressed in the original coefficient. Thus, scaling can change the effective penalty on a contrast. For an unscaled binary indicator with prevalence \(p\), its standard deviation is \(\sqrt{p(1-p)}\); rare-level indicators can therefore be affected differently by standardization. Check the software’s preprocessing and coefficient rescaling rather than assuming all indicator contrasts receive identical effective penalties.

Worked Example: Comparing Two Reference Choices

A hypothetical factor has levels A, B, and C. A candidate fitted model gives factor-level contributions 0, 2, and 0, respectively, and a common intercept of 10. Compute the fitted values and lasso indicator penalty under reference A and reference B. Assume there are no other predictors, use unscaled indicators, and use penalty \(\lambda\sum_j|\beta_j|\).

Reference A. The indicator coefficients are \(\beta_B=2-0=2\) and \(\beta_C=0-0=0\). The fitted values are A: \(10\), B: \(10+2=12\), and C: \(10+0=10\). The penalty is \(\lambda(|2|+|0|)=2\lambda\).

Reference B. The intercept must equal B’s fitted value, 12. The contrast for A is \(10-12=-2\); the contrast for C is \(10-12=-2\). The fitted values are A: \(12-2=10\), B: \(12\), and C: \(12-2=10\). The penalty is \(\lambda(|-2|+|-2|)=4\lambda\).

The fitted values agree exactly, but the penalty differs by \(4\lambda-2\lambda=2\lambda\). This calculation compares the penalty assigned to the same candidate fitted values; it does not by itself claim that either candidate is the optimum for a particular data set. The trap is to conclude that lasso selection is invariant to reference coding merely because ordinary regression fitted values can be invariant to a change of reference.

Practical Interpretation and Exam Traps

When reporting a lasso model with a factor, identify the reference level, list the nonzero indicator contrasts, and state the comparison they represent. If the response model uses a link function, the coefficients are contrasts on the linear-predictor scale; they need not be additive changes on the response scale. For example, an indicator coefficient in a log-link model represents a difference in log fitted means, not a fixed dollar difference.

The factor’s levels are not necessarily selected as one block. If a task requires all levels of a factor to enter or leave together, that is a group-selection requirement, not the ordinary coefficient-by-coefficient lasso described here. Also, a nonzero coefficient is a selection outcome under the chosen data, coding, scaling, and penalty; it is not by itself evidence of causation or statistical significance.

Do not add the intercept when naming a factor contrast. The reference contrast is zero; the non-reference contrasts are their indicator coefficients.
Do not infer the full fitted value from the intercept alone. Other predictor contributions must be included, or a common baseline that already contains them must be specified.
Do not equate a zero indicator with a zero factor effect. Other indicators can remain nonzero, and the zero coefficient only equates that level’s fitted linear predictor to the reference at fixed values of other predictors.
Do not assume reference coding and scaling are irrelevant. They can alter the coefficient penalty and therefore the selected representation.

Key takeaway: Ordinary lasso penalizes the individual coefficients for non-reference indicators. A factor can therefore retain some contrasts and shrink others to zero. Interpret those coefficients relative to the stated reference, include all common predictor contributions when calculating fitted values, and account for coding and scaling when discussing selection.

Check Your Understanding

For each problem, identify the reference-level contrasts before interpreting the fitted model.

A factor has reference level R. At a fixed profile, the common baseline fitted value, including the intercept and all other predictors, is 6.5. The lasso coefficients for S and T are \(-0.4\) and 0. Find the fitted values for R, S, and T. (Answer: R = 6.5, S = 6.1, T = 6.5.)
With reference level A, a factor has indicator coefficients \(\widehat{\beta}_B=0\) and \(\widehat{\beta}_C=1.3\). What do the coefficients say about the fitted linear predictors when other predictors are fixed? (Answer: A and B have equal fitted linear predictors; C is 1.3 higher than A.)
For one indicator, \(\lambda=0.6\) and the residual association \(\frac{1}{n}\sum_i z_{ij}r_i\) is \(-0.42\). Is a zero coefficient compatible with the lasso condition? (Answer: Yes, because \(|-0.42|=0.42\leq0.6\).)
Factor-level contributions are 1, 4, and 2 for levels A, B, and C. With A as reference, calculate the two indicator coefficients and their total absolute penalty. (Answer: Coefficients are 3 and 1; total absolute penalty is 4.)
A binary indicator has prevalence \(p=0.25\). Find its standard deviation after centering, and state the effective penalty on the original-scale coefficient if the standardized coefficient is penalized by \(\lambda|\theta|\) and \(\theta=s\beta\). (Answer: \(s=\sqrt{0.25(0.75)}=\sqrt{0.1875}\approx0.4330\); the original-scale penalty is \(0.4330\lambda|\beta|\).)

What You'll Learn