Factor Levels and Indicator Columns

From a Categorical Predictor to Columns

A generalized linear model uses a linear predictor, such as \(\eta_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}\), to connect predictors to the response distribution through a link function. For a categorical predictor, the model needs a numerical representation of each observed category. Indicator columns provide that representation without pretending that category labels have a meaningful numerical distance.

Suppose a factor \(A\) has \(K\) levels, labelled \(a_1,\ldots,a_K\). The labels might be treatment names, hospital names, or disease categories. For observation \(i\), an indicator for level \(a_k\) is 1 if that observation has level \(a_k\), and 0 otherwise. The indicators record membership; they do not assign a score to the level.

Definition: For a factor \(A\) with levels \(a_1,\ldots,a_K\), the indicator for level \(a_k\) on observation \(i\) is \(D_{ik}=I(A_i=a_k)\), where \(I(\cdot)\) equals 1 when its condition is true and 0 otherwise. The \(K\) indicators are mutually exclusive and exhaustive: exactly one equals 1 for each observation whose factor value is one of these levels.

Using all \(K\) indicators makes the factor representation easy to see, but a model that also has an intercept cannot estimate a separate coefficient for every indicator. The reason is exact linear dependence: the indicators sum to 1 for every observation, just as the intercept column does. A common solution is to choose one level as the reference level and include indicators for the other \(K-1\) levels, along with an intercept.

Definition: In reference-indicator coding with reference level \(a_r\), the design matrix includes an intercept and one column \(I(A_i=a_k)\) for each level \(a_k\) other than \(a_r\). The reference level has 0 in every one of these factor-indicator columns.

The reference level determines which indicator columns are used, not whether the factor is categorical. For example, if “control” is the reference level, the columns can indicate “low dose” and “high dose.” If “high dose” is the reference, the columns instead indicate “control” and “low dose.” The row for an observation is formed from its actual level, not from the order in which the labels happen to be written.

Worked Examples: Constructing Indicator Columns

Worked Example: Three Treatment Levels

A trial has a categorical treatment factor with levels Control, Drug A, and Drug B. Choose Control as the reference level. Construct the indicator columns for four observations with treatment assignments Control, Drug A, Drug B, and Drug A.

The reference is Control, so the two columns are \(I(A_i=\text{Drug A})\) and \(I(A_i=\text{Drug B})\). For the first observation, both are 0 because it is in the reference level. For the second, the entries are 1 and 0; for the third, 0 and 1; and for the fourth, 1 and 0. Including the intercept, the design rows are therefore:

\begin{array}{c|ccc} \text{Treatment} & \text{Intercept} & I(A_i=\text{Drug A}) & I(A_i=\text{Drug B})\\ \hline \text{Control} & 1 & 0 & 0\\ \text{Drug A} & 1 & 1 & 0\\ \text{Drug B} & 1 & 0 & 1\\ \text{Drug A} & 1 & 1 & 0 \end{array}

The two Drug A observations have identical factor-indicator entries because they share the same treatment level. Other predictors, if present, would contribute their own columns. The indicator entries are determined by the treatment labels, not by the outcomes.

Worked Example: A Different Reference Level

Use the same three treatment levels, but now choose Drug B as the reference. The columns are \(I(A_i=\text{Control})\) and \(I(A_i=\text{Drug A})\). For the same four assignments, the rows become:

\begin{array}{c|ccc} \text{Treatment} & \text{Intercept} & I(A_i=\text{Control}) & I(A_i=\text{Drug A})\\ \hline \text{Control} & 1 & 1 & 0\\ \text{Drug A} & 1 & 0 & 1\\ \text{Drug B} & 1 & 0 & 0\\ \text{Drug A} & 1 & 0 & 1 \end{array}

The chosen reference level is the level whose row is all zero in the factor-indicator columns. Thus, changing the reference changes the names and entries of the columns. It does not change the observed treatment assignments or the definition of the factor.

Worked Example: Factor Labels Are Not Scores

Suppose disease severity has the ordered levels Mild, Moderate, and Severe. A researcher records them as the numbers 1, 2, and 3. Compare an indicator representation with a model that uses this single numeric score.

With Mild as reference, the indicator representation has columns \(I(A_i=\text{Moderate})\) and \(I(A_i=\text{Severe})\). The three level-specific contributions to the linear predictor can be specified independently by an intercept and these two indicators.

By contrast, treating the recorded values 1, 2, and 3 as a numeric predictor with one slope \(\beta\) imposes contributions \(\alpha+\beta\), \(\alpha+2\beta\), and \(\alpha+3\beta\). The difference from Mild to Moderate must equal the difference from Moderate to Severe: both are \(\beta\). For instance, contributions \(0,1,0\) cannot satisfy this requirement: the first difference is \(1-0=1\), while the second is \(0-1=-1\). The fact that the levels are ordered does not by itself justify imposing equal numerical steps.

A numeric score can be appropriate if that equal-step structure is intended and justified. Otherwise, the level indicators preserve the categorical specification.

What the Indicator Representation Can Express

The following result describes exactly what an intercept and \(K-1\) reference indicators can represent for a factor with \(K\) observed levels. It is an algebraic statement about the linear predictor; it does not depend on whether the GLM uses a logit, log, or another link.

Theorem: Suppose every one of the \(K\) factor levels occurs in the data, and the model contains an intercept and the \(K-1\) indicators for all levels other than reference level \(a_r\). Then these columns can represent any specified set of \(K\) level-specific contributions to the linear predictor, and the representation is unique.

Proof. Let the desired contribution for level \(a_k\) be \(\gamma_k\). Set the intercept coefficient to \(\beta_0=\gamma_r\), and for each nonreference level \(a_k\), set the coefficient of its indicator to \(\beta_k=\gamma_k-\gamma_r\). For an observation in the reference level, all indicators are zero, so its contribution is \(\beta_0=\gamma_r\). For an observation in level \(a_k\ne a_r\), only that level’s indicator is 1, so its contribution is

\beta_0+\beta_k =\gamma_r+(\gamma_k-\gamma_r) =\gamma_k.

This establishes that every specified set of contributions can be represented. To show uniqueness, suppose two coefficient sets produce the same contribution at every observed level. At the reference level, the contribution is the intercept, so the two intercepts must be equal. At each nonreference level, the contribution is the intercept plus that level’s indicator coefficient. Since the intercepts are equal, equality of contributions at that level forces the corresponding indicator coefficients to be equal. Thus all coefficients agree, proving uniqueness. \(\square\)

A related rank fact explains why the reference indicator columns avoid the dependence created by including all \(K\) indicators and an intercept. The assumption that every level occurs matters: a declared level absent from the data has an all-zero indicator column and cannot supply independent information.

Result: If every one of the \(K\) levels is observed, the intercept and the \(K-1\) reference-indicator columns are linearly independent.

Proof. Consider a linear combination of these columns that is zero for every observation. Write its coefficients as \(c_0\) for the intercept and \(c_k\) for the indicator of level \(a_k\ne a_r\). For observations at the reference level, the combination equals \(c_0\). Since that level occurs, \(c_0=0\). For observations at any nonreference level \(a_k\), the combination equals \(c_0+c_k\). That level occurs, so this quantity is zero; with \(c_0=0\), it follows that \(c_k=0\). Every coefficient is zero, so the columns are linearly independent. \(\square\)

Why Numeric Codes Are Different

A factor’s labels may be words or numbers. Either way, using indicators treats the labels as categories. Replacing those indicators with one numeric column changes the model: it assumes that differences in the recorded scores have quantitative meaning and that the linear predictor changes linearly with those scores.

For example, if three levels are coded 1, 2, and 3, a single numeric slope requires equally spaced changes in the linear predictor across those codes. With reference indicators, the contributions for the three levels need not follow that pattern. A numeric representation may be a useful, more restrictive model when its assumptions are suitable; it is not merely a shorter way to write the same categorical predictor.

Common Mistakes and Checks

Including all \(K\) indicators and an intercept without addressing dependence. The \(K\) indicator columns sum to the intercept column. Use \(K-1\) indicators with a reference level, or use a suitable alternative parameterization.
Assuming the first label is automatically the reference. The reference is a modeling choice. Identify it explicitly, then list the columns included.
Treating codes as measurements just because they are numbers. Codes 1, 2, and 3 may be category labels. A numeric predictor with one slope imposes a specific quantitative structure that indicator columns do not.
Assuming an unobserved level can be estimated from its indicator column. If a level does not occur in the data, its column is all zero. The data then contain no information to estimate a separate contribution for that level.

Key takeaway: First identify the factor’s levels. Then choose a reference level and create one indicator column for each other level. Keep category labels distinct from numerical scores: they encode different assumptions about the linear predictor.

Check Your Understanding

For each question, identify the design-matrix structure or the modeling assumption involved.

A factor has four observed levels, and level 2 is the reference. How many factor-indicator columns are included with an intercept, and which level has all zeros in those columns?
For a factor with levels North, South, and West, write the indicator entries for an observation in South when North is the reference.
Why are an intercept and all \(K\) level indicators linearly dependent?
A three-level factor is coded 0, 1, and 2 and entered as one numeric predictor. What restriction does its single slope impose on the three level contributions?
What happens to the indicator column for a declared factor level that does not occur in the sample?