Penalizing Factor Indicators And Group Effects

Why Factor Coding Matters Under Regularization

A categorical predictor with several levels cannot enter a regression as a single ordinary numeric value. It is represented by indicator columns, and a regularization method acts on the coefficients of those columns. The resulting model can therefore retain some level contrasts and shrink others to zero. Understanding that distinction is important when interpreting a selected model: ordinary lasso selects coefficients, not necessarily whole categorical predictors.

The examples below focus on treatment coding, the common encoding that represents a factor with \(k\) levels by \(k-1\) indicators when a model has an intercept. This is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For a factor with \(k\) levels, treatment coding chooses one level as the reference and creates one indicator for each of the other \(k-1\) levels. A non-reference coefficient describes the change in the model's linear predictor for that level relative to the reference, holding other predictors fixed. In an ordinary lasso or elastic-net fit, these indicator coefficients are penalized separately; the intercept is ordinarily not penalized.

What the Penalty Sees

With an intercept and a three-level factor, suppose the levels are North, South, and Coastal, and North is the reference. The model matrix contributes two columns: an indicator for South and an indicator for Coastal. Each row has a 1 in the column for its level, except that rows in the reference level have 0 in both columns.

Level	South indicator	Coastal indicator
North (reference)	0	0
South	1	0
Coastal	0	1

For a Gaussian model, a simplified statement of the lasso objective used for illustration is

\frac{1}{2n}\sum_{i=1}^{n}(y_i-\beta_0-\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta})^2 +\lambda\sum_{j=1}^{p}|\beta_j|.

Here \(n\) is the number of observations, \(\beta_0\) is the intercept, \(\boldsymbol{\beta}\) contains the predictor coefficients, and \(\lambda\) is a nonnegative penalty strength. The intercept is excluded from the sum. In an elastic net, the penalty is a combination of an absolute-value penalty and a squared-coefficient penalty; both still act on the encoded coefficients rather than automatically treating all indicators from a factor as one group. For generalized linear models, the squared-error term is replaced by the model's appropriate loss, while the coefficient-penalty principle remains.

In R, a factor is commonly expanded into indicator columns in the model matrix before fitting with glmnet. With the usual intercept and treatment coding, the reference level has no separate coefficient. The other level indicators enter the penalty as separate columns. By default, glmnet standardizes predictors internally; the penalty applies in the fitting procedure after that preprocessing, while reported coefficients are returned on the original predictor scale. Setting standardize = FALSE changes that preprocessing. The hand calculations below explicitly state when standardization is turned off.

Worked Example: Reading Treatment-Coded Coefficients

A hypothetical Gaussian model predicts an annual claim amount using territory, with North as the reference. For a policyholder with the other modeled characteristics held fixed, suppose the common contribution to the linear predictor is 0.80, and the fitted territory coefficients are \(\hat\beta_{\text{South}}=-0.12\) and \(\hat\beta_{\text{Coastal}}=0.35\). Find the linear predictor for each territory and interpret the coefficients.

Method. Add the indicator contributions to the common contribution. The North indicators are both zero, the South indicator is one only for South, and the Coastal indicator is one only for Coastal.

For North, the linear predictor is \(0.80\). For South, it is \(0.80-0.12=0.68\). For Coastal, it is \(0.80+0.35=1.15\).

The South coefficient says that, at these fixed predictor values, South's linear predictor is 0.12 lower than North's. The Coastal coefficient says Coastal's is 0.35 higher than North's. The coefficients are contrasts with North, not standalone estimates of the territories' outcomes.

Check. Subtracting the North predictor gives \(0.68-0.80=-0.12\) for South and \(1.15-0.80=0.35\) for Coastal, matching the coefficients. The common mistake avoided is interpreting an indicator coefficient as the territory's full predicted value.

Why Lasso Does Not Select a Factor as One Unit

The lasso's absolute-value penalty encourages individual coefficients to become exactly zero. Because treatment coding gives a factor multiple coefficients, each one can be retained or removed separately. A factor with three levels can therefore have one non-reference contrast retained and another set to zero. The zero means that, in the fitted parameterization, that level's contrast with the reference is zero; it does not mean the entire factor was excluded.

A short derivation makes the separate action of the penalty explicit. Consider one coefficient \(\beta\) in a least-squares problem after the other terms have been held fixed. Suppose its predictor column is centered and scaled so that its mean squared value is 1, and let \(c\) be its association with the current residual, measured as the mean product of the column and residual. The part of the lasso objective that depends on \(\beta\) is, up to a constant,

\frac{1}{2}\beta^2-c\beta+\lambda|\beta|.

For \(\beta>0\), differentiating gives \(\beta-c+\lambda=0\), so a positive solution is \(\beta=c-\lambda\) when \(c>\lambda\). For \(\beta<0\), the derivative gives \(\beta-c-\lambda=0\), so a negative solution is \(\beta=c+\lambda\) when \(c<-\lambda\). If \(-\lambda\leq c\leq\lambda\), the minimum is at zero: the left and right slopes there straddle zero. Thus the solution is

\hat\beta=\operatorname{sign}(c)\max(|c|-\lambda,0).

This is the soft-thresholding rule. Applied to encoded factor indicators, it acts on their coefficients separately. It does not impose a condition that all coefficients for a factor must be zero together or nonzero together.

Worked Example: One Contrast Survives and Another Is Zero

A hypothetical dataset has one observation at each level of a three-level factor: \(y=4\) for A, \(y=1\) for B, and \(y=0\) for C. Fit a Gaussian lasso with an intercept, treatment coding using C as the reference, \(\lambda=0.50\), and standardize = FALSE. Show that it is possible for the A indicator to be nonzero while the B indicator is zero.

Method. Write the fitted values as \(\mu+\beta_A\) for A, \(\mu+\beta_B\) for B, and \(\mu\) for C. The objective is

J(\mu,\beta_A,\beta_B) =\frac{1}{6}\left[(4-\mu-\beta_A)^2+(1-\mu-\beta_B)^2+(0-\mu)^2\right] +0.50\left(|\beta_A|+|\beta_B|\right).

The factor \(1/6\) is \(1/(2n)\) because \(n=3\). Consider \(\beta_A=1.25\), \(\beta_B=0\), and \(\mu=1.25\). The residuals for A, B, and C are respectively \(4-1.25-1.25=1.50\), \(1-1.25=-0.25\), and \(0-1.25=-1.25\).

The residuals sum to zero, which satisfies the intercept condition. For the positive coefficient \(\beta_A\), the coefficient condition is \(r_A=3\lambda=1.50\), which holds. For the zero coefficient \(\beta_B\), the lasso condition is \(|r_B|\leq 3\lambda\); here \(0.25\leq1.50\), so zero is permitted. These conditions are sufficient because the objective is convex.

The squared-error part is \((1.50^2+(-0.25)^2+(-1.25)^2)/6=(2.25+0.0625+1.5625)/6=3.875/6\), or approximately \(0.6458\). The penalty is \(0.50(1.25+0)=0.625\). The total objective is approximately \(1.2708\). The fitted values are \(2.50\) for A and \(1.25\) for both B and C.

Check. B and C have the same fitted value because \(\hat\beta_B=0\), while A differs from C because \(\hat\beta_A=1.25\). This explicitly demonstrates partial selection of the factor's contrasts. The common mistake avoided is assuming that retaining the factor requires retaining every one of its indicator coefficients.

Reference Levels Change the Penalty Representation

For an unpenalized model, changing the reference level changes the coefficient labels and values but not the fitted values, provided the same factor information is represented. Under a coefficient penalty, however, a change of coding can change the penalty assigned to an otherwise identical set of fitted values. The penalty is calculated from the coordinates used by the model, not from a coding-independent measure of the factor's overall effect.

Worked Example: Comparing Two Reference Codings

Use the same three group responses as in the preceding example: A has response 4, B has response 1, and C has response 0, with one observation per group. Compare the unpenalized treatment-coded representations using C as the reference and A as the reference. Take \(\lambda=0.20\) and turn standardization off so the penalty contributions can be compared directly.

Method. First represent the group means exactly. With C as reference, the intercept is 0, the A coefficient is \(4-0=4\), and the B coefficient is \(1-0=1\). The fitted values are therefore \(4,1,0\), and the sum of absolute penalized coefficients is \(|4|+|1|=5\). The penalty contribution is \(0.20(5)=1.00\).

With A as reference, the intercept is 4, the B coefficient is \(1-4=-3\), and the C coefficient is \(0-4=-4\). These also give fitted values \(4,1,0\). But the sum of absolute penalized coefficients is \(|-3|+|-4|=7\), so the penalty contribution is \(0.20(7)=1.40\).

The two representations produce the same unpenalized predictions, yet their lasso penalty contributions differ by \(1.40-1.00=0.40\). Consequently, the penalty can favor one coding's coefficient representation over another. With a nonzero penalty, changing the reference level may change the fitted model and its predictions; it is not merely a relabeling guaranteed to leave the penalized fit unchanged.

Check. Both codings reconstruct group means \(4,1,0\), but the penalized coefficient sums are 5 and 7. The common mistake avoided is assuming that equivalent unpenalized model matrices must remain equivalent after a coordinate-dependent penalty is applied.

Interpreting Penalized Factor Effects in a GLM

In a generalized linear model, coefficients describe changes in the linear predictor, and the link function determines how those changes translate to the outcome scale. For logistic regression, the linear predictor is the log odds. A factor coefficient is therefore a log-odds contrast with the reference level, conditional on the other included predictors. Exponentiating the coefficient gives the corresponding odds ratio.

Worked Example: Reading a Penalized Logistic Factor

A hypothetical penalized logistic model uses sales channel with Online as the reference. Suppose the fitted coefficients are \(0.40\) for Agent and \(-0.20\) for Broker. For a particular set of other predictors, their combined contribution to the linear predictor is \(0.20\). Calculate the predicted probabilities by channel and interpret the two coefficients.

Method. Add the appropriate factor coefficient to the common contribution. The linear predictors are \(0.20\) for Online, \(0.20+0.40=0.60\) for Agent, and \(0.20-0.20=0.00\) for Broker.

For logistic regression, the probability corresponding to a linear predictor \(\eta\) is \(1/(1+\exp(-\eta))\). The probabilities are approximately \(0.5498\) for Online, \(0.6457\) for Agent, and \(0.5000\) for Broker. The Agent coefficient corresponds to an odds ratio of \(\exp(0.40)\approx1.4918\) versus Online. The Broker coefficient corresponds to an odds ratio of \(\exp(-0.20)\approx0.8187\) versus Online.

Check. Agent versus Broker differs by \(0.60-0.00=0.60\) in the linear predictor, so their odds ratio is \(\exp(0.60)\approx1.8221\). The common mistake avoided is treating a coefficient as a probability difference: the coefficient is a log-odds contrast, not a change of 0.40 in probability.

Practical Interpretation and Common Traps

A factor's coefficient pattern should be interpreted with its coding and reference level in view. A nonzero coefficient signals a retained contrast in that parameterization. A zero coefficient says that the corresponding contrast has been shrunk to zero, not that the whole factor has no role. Other levels of the same factor may still have nonzero coefficients.

These distinctions matter when reporting selected predictors or comparing models. Saying “the factor was selected” can conceal which contrasts survived. A clearer report names the reference level and identifies the retained comparisons. If an analysis needs a categorical predictor to enter or leave only as a whole, ordinary coefficient-wise lasso does not impose that requirement; a group-penalty method is designed to penalize a set of coefficients together.

Do not count columns as separate original variables. Several model-matrix columns may represent one factor.
Do not interpret a zero contrast as proof of no overall factor association. It is a statement about one comparison with the chosen reference under the fitted penalty.
Do not assume the reference level is penalized like the others. With an intercept and treatment coding, it is represented by the intercept and has no separate indicator coefficient.
Do not assume recoding leaves a penalized fit unchanged. The coefficient penalty depends on the representation, even when the unpenalized predictions can be represented identically.

Key takeaway: Ordinary lasso and elastic-net penalties act on the encoded factor coefficients individually. Under treatment coding, each non-reference coefficient is a contrast against the reference level. Some contrasts can be zero while others remain nonzero, and changing the reference coding can change how the penalty treats equivalent fitted relationships.

Check Your Understanding

For each problem, identify the relevant coding or penalty implication before calculating.

A factor has levels Cedar, Birch, and Ash, with Cedar as reference. The fitted linear predictor includes coefficients \(0.30\) for Birch and \(-0.10\) for Ash. If all other terms contribute \(0.50\), find the linear predictor for each level. (Answer: Cedar \(0.50\), Birch \(0.80\), Ash \(0.40\).)
A treatment-coded factor has four levels and an intercept. How many indicator coefficients represent the factor, and how many coefficients are penalized separately by an ordinary lasso? (Answer: Three indicator coefficients; each of the three is penalized separately.)
In a logistic model, a factor coefficient for Clinic B relative to reference Clinic A is \(-0.25\). Find the odds ratio for B versus A, rounded to four decimals. (Answer: \(\exp(-0.25)=0.7788\).)
For a three-level factor, the same exact group means are represented under two treatment codings. The penalized coefficient sums are 6 under one coding and 8 under the other, with \(\lambda=0.15\). Find each lasso penalty contribution and their difference. (Answer: \(0.90\) and \(1.20\); difference \(0.30\).)
A fitted treatment-coded model has three non-reference levels. Two indicator coefficients are zero and one is nonzero. What can be concluded about factor selection under ordinary lasso? (Answer: One contrast is retained and two are shrunk to zero; ordinary lasso has not imposed all-or-none selection for the factor.)

What You'll Learn