Why Factor Encoding Matters in Model Comparisons
A factor such as territory, vehicle class, or claim channel describes membership in categories, not a numerical quantity. A model still needs a way to use that information. Generalized linear models (GLMs) commonly represent a factor using contrasts, such as reference-level dummy variables. Tree algorithms may use categorical splits directly, or may instead receive a numeric or dummy-coded representation. Those choices affect coefficient interpretation, the splits a tree can consider, and—when coefficients are penalized—the fitted model itself.
A fair comparison therefore requires more than giving each algorithm a column with the same name. The model inputs must represent the same categories, and the analyst must understand what each algorithm does with them. This distinction is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Reference-Level Encoding in a GLM
Suppose a GLM has a factor with levels A, B, and C, and A is the reference. For an observation in category \(g\), define \(I(B)=1\) if \(g=B\) and 0 otherwise, and define \(I(C)\) similarly. The linear predictor is
The intercept is the linear predictor for A. The coefficient for B is the change in linear predictor from A to B, and the coefficient for C is the change from A to C. The model’s response mean is obtained through its link function: \(\mu=g^{-1}(\eta)\). Thus, for a logistic GLM, the coefficients are differences in log odds, not differences in probabilities.
In an unpenalized GLM, changing the reference level changes the coefficient values but not the set of category-specific linear predictors the model can represent, provided the model remains identifiable and otherwise has the same terms. This matters when reading coefficients: a coefficient is always relative to the chosen reference, while fitted predictions need not depend on that choice.
Worked Example: Changing the Reference in a Logistic GLM
A hypothetical portfolio has four observations in each of territories A, B, and C. The event counts are 1, 2, and 3, respectively. Fit the category-specific event probabilities using a logistic GLM with only territory as a predictor. Compare the coefficient interpretation when A and B are the reference levels.
Method and assumptions. Each observation has a binary outcome, and the model gives each territory its own event probability. Assume independent Bernoulli outcomes within and across territories. With no complete separation, the maximum-likelihood estimate in each territory is its observed event proportion.
For A, \(\hat p_A=1/4=0.25\), so its estimated odds are \(0.25/(1-0.25)=1/3\), and its estimated log odds are \(\ln(1/3)=-1.0986\). For B, \(\hat p_B=2/4=0.50\), giving odds \(1\) and log odds \(0\). For C, \(\hat p_C=3/4=0.75\), giving odds \(3\) and log odds \(\ln(3)=1.0986\).
Reference A. The intercept is \(\beta_0=-1.0986\). The B coefficient is \(0-(-1.0986)=1.0986\), and the C coefficient is \(1.0986-(-1.0986)=2.1972\). The predicted probabilities are \(\operatorname{logit}^{-1}(-1.0986)=0.25\), \(\operatorname{logit}^{-1}(-1.0986+1.0986)=0.50\), and \(\operatorname{logit}^{-1}(-1.0986+2.1972)=0.75\).
Reference B. The intercept is now \(0\). The A coefficient is \(-1.0986-0=-1.0986\), and the C coefficient is \(1.0986-0=1.0986\). The linear predictors for A, B, and C are \(-1.0986\), \(0\), and \(1.0986\), so the predicted probabilities are again \(0.25\), \(0.50\), and \(0.75\).
Check. Both encodings recover the three observed proportions. The coefficients differ because their comparisons differ; the category-specific predictions agree. The common mistake this example avoids is treating a change in reference level as a change in the underlying unpenalized model.
Why Unpenalized GLM Predictions Are Invariant
The reason is a reparameterization: changing the reference changes how the same category-specific linear predictors are written, not which predictors are available. For three levels, write their linear predictors as \(\eta_A,\eta_B,\eta_C\). Under reference A, the coefficients are \(\beta_0=\eta_A\), \(\beta_B=\eta_B-\eta_A\), and \(\beta_C=\eta_C-\eta_A\). Under reference B, they are \(\beta'_0=\eta_B\), \(\beta'_A=\eta_A-\eta_B\), and \(\beta'_C=\eta_C-\eta_B\). Substitution gives the same \(\eta\) for every category under either encoding.
A GLM likelihood is determined by the observed responses and the fitted means (or equivalently, the linear predictors and link). Since every observation retains the same linear predictor under the reparameterization, the likelihood and fitted response means are unchanged. This argument requires the same model space and no coefficient-dependent constraint or penalty that changes with the parameterization. It does not imply that coefficients have the same values or interpretation.
where \(r\) is the chosen reference level. The formula applies to a GLM with reference-level contrasts for the factor; other model terms can be present, but the comparison holds only when those terms and the represented model space are held fixed.
Penalized Regression: The Coding Can Change the Fit
Ridge and lasso regression add a penalty on coefficient values to a data-fit objective. A typical objective is the negative log-likelihood plus \(\lambda\) times a penalty, where \(\lambda\geq 0\) controls penalty strength. For ridge, the penalty is a sum of squared coefficients; for lasso, it is a sum of absolute coefficient values. Often the intercept is not penalized, but the treatment of factor coefficients must be checked for the particular modeling procedure.
Unlike an unpenalized likelihood, a coefficient penalty is not automatically invariant to a change in reference coding. The same category-specific predictions can be described using different coefficient values, and the penalty is calculated from those values. Thus, even if the unpenalized model can represent identical predictions under both encodings, a penalized fit may shrink them differently. The reference level can effectively receive special treatment.
Worked Example: A Reference Change Alters the Ridge Penalty
A hypothetical Gaussian regression has three territory-specific fitted means: A has 2, B has 4, and C has 8. Compare the ridge penalty for these same fitted means under reference A and reference B, using the convention that the intercept is unpenalized and all dummy coefficients receive the same penalty.
Method. Use the ridge penalty \(P=\sum_k\beta_k^2\) over the dummy coefficients only. This calculation compares the penalty assigned to a fixed set of fitted means; it is not a claim that these means are the resulting ridge estimates for any particular \(\lambda\).
With A as reference, the intercept is 2, the B coefficient is \(4-2=2\), and the C coefficient is \(8-2=6\). Therefore, \(P_A=2^2+6^2=4+36=40\).
With B as reference, the intercept is 4, the A coefficient is \(2-4=-2\), and the C coefficient is \(8-4=4\). Therefore, \(P_B=(-2)^2+4^2=4+16=20\).
Check. Both encodings describe exactly the same means: \((2,4,8)\). Yet the ridge penalty is 40 with A as reference and 20 with B as reference. For positive \(\lambda\), the penalized objectives therefore assign different costs to this same prediction vector. In general, minimizing those different objectives can produce different predictions. The common mistake is to infer that unchanged unpenalized predictions guarantee unchanged penalized fits.
The same issue arises for lasso: for the example above, the dummy-coefficient penalties are \(|2|+|6|=8\) with A as reference and \(|-2|+|4|=6\) with B as reference. A coefficient set to zero also has a coding-dependent interpretation: it says that a category matches the reference on the modeled effect, not necessarily that the factor has no effect overall.
When comparing penalized models, document the reference level and contrast scheme, and verify that the penalty applies to factor coefficients as intended. If two candidates use different encodings, a difference in fit may reflect different shrinkage geometry rather than an algorithmic advantage. Do not assume that merely tuning \(\lambda\) removes this difference.
How Trees Can Treat the Same Factor Differently
A tree predicts by recursively splitting observations into groups. For a categorical predictor, a split may separate one level from the others or, in some implementations, divide the levels into two arbitrary subsets. The available split rules depend on the algorithm and how the factor is supplied. A tree given a factor with native categorical handling may consider category-subset splits; a tree given integer codes may treat those codes as ordered; and a tree given dummy columns may split on individual indicators. These are not necessarily equivalent searches, especially when tree depth is limited.
An integer code such as A=1, B=2, C=3 is only a label assignment unless a genuine order exists. A threshold split on that code can separate only contiguous groups in that assigned order. Changing the arbitrary code order can change the candidate splits. Dummy encoding avoids imposing that specific order, but it presents multiple predictors and may require additional splits to express a grouping of several levels. Native categorical handling can also vary across algorithms, so the model’s documentation and fitted preprocessing must be checked rather than assumed.
Worked Example: An Ordered Code Misses a Useful Tree Split
A hypothetical training set has four territories A, B, C, and D, with two observations per territory. Their binary outcomes are A: \(0,0\); B: \(10,10\); C: \(0,0\); and D: \(10,10\). Compare the best root split using a native factor split that can group any subset of levels with splits on an ordered code A=1, B=2, C=3, D=4. Use squared-error impurity.
Method and assumptions. For a node, squared-error impurity is the sum of squared deviations from that node’s mean. A candidate split’s impurity is the sum of the two child impurities. The outcomes here are deliberately numeric to make the split arithmetic explicit.
The parent has four zeros and four tens, so its mean is \(5\) and its impurity is \(4(0-5)^2+4(10-5)^2=4(25)+4(25)=200\).
A native factor split can group \(\{A,C\}\) against \(\{B,D\}\). The first child contains four zeros and has impurity 0. The second contains four tens and also has impurity 0. The split impurity is 0, an improvement of \(200-0=200\) over the parent.
For the ordered code, only threshold partitions between adjacent codes are available at the root. For A | BCD, the right child has two zeros and four tens, with mean \(40/6=6.6667\). Its impurity is \(2(0-6.6667)^2+4(10-6.6667)^2=88.8889+44.4444=133.3333\); the left child is pure. For AB | CD, each child has two zeros and two tens, mean 5 and impurity 100, so the total is 200. For ABC | D, the left child’s impurity is 133.3333 and the right child’s is 0. Thus the best ordered-code split has impurity 133.3333, not 0.
Check. The native factor split finds the exact grouping \(\{A,C\}\) versus \(\{B,D\}\), while a single threshold on the chosen order cannot. The result is not evidence that one general tree algorithm is always better: it shows that the supplied factor representation and allowed split rules can change what a tree can express at a given depth. The common mistake is to treat arbitrary integer labels as a neutral encoding.
Making a Comparison Defensible
Before comparing validation results, make the factor definition explicit. The categories must mean the same thing in each candidate, and any grouping of rare levels must be applied consistently. Fit or define preprocessing using the training data as appropriate to the validation design, then apply that same mapping to validation data. A factor level that is absent from training, or handled differently by different pipelines, can make predictions incomparable; confirm how each procedure handles it rather than silently recoding it.
Record the allowed categories, any deliberate grouping, and the treatment of missing or unexpected levels.
For a GLM, check the contrasts and reference. For a penalized regression, check which encoded coefficients are penalized. For a tree, check whether it receives native categories, ordered codes, or dummy columns.
Keep the target, data partitions, and evaluation metric aligned, and explain any encoding difference that affects the model class being compared.
A validation-score difference is not automatically an algorithm effect if one candidate had a different category grouping, penalty geometry, or set of available splits.
An encoding difference is not always a flaw. It may be an intentional part of the candidate modeling procedure. The key is to distinguish that choice from a supposedly neutral preprocessing step and to report what comparison is actually being made. Reference-level coding is primarily an interpretation choice for a standard unpenalized GLM, but it can affect penalized regression. Factor handling in a tree determines the split candidates available to that implementation.
Check Your Understanding
For each problem, distinguish the factor’s meaning from the representation supplied to the model.
- A Gaussian GLM has three levels with fitted means 5, 7, and 11. With the first level as reference, state the intercept and two dummy coefficients. Then state the intercept and two coefficients if the second level is the reference. (Answer: First reference: intercept 5, coefficients 2 and 6. Second reference: intercept 7, coefficients -2 and 4.)
- A penalized model uses an unpenalized intercept and ridge penalty on dummy coefficients. Category means are 1, 3, and 6. Calculate the penalties with the first level and then the second level as reference. (Answer: First reference: \(2^2+5^2=29\). Second reference: \((-2)^2+3^2=13\).)
- A tree receives codes X=1, Y=2, Z=3, W=4. Which of the following groups cannot be separated from its complement by one threshold split: \(\{X,Y\}\), \(\{X,Z\}\), or \(\{Z,W\}\)? (Answer: \(\{X,Z\}\); it is not a contiguous group in the supplied order.)
- A binary logistic GLM estimates event proportions 1/5 for region R and 3/5 for region S. With R as reference, what are the intercept and S coefficient? (Answer: Intercept \(\ln(1/4)=-1.3863\); S coefficient \(\ln(3/2)-\ln(1/4)=\ln(6)=1.7918\).)
- Two candidate models use the same train/test rows, but one groups two rare categories while the other retains them separately. Can a validation-score difference be attributed solely to the learning algorithms? (Answer: No; the factor representation and available distinctions differ, so the comparison also reflects that modeling choice.)