Representing a Factor With K Minus One Indicators

Why Indicator Columns Are Useful

A factor records membership in a category, such as a clinic, treatment type, or soil class. Regression models work with numerical predictors, so a factor must be represented numerically before it can enter the model. Indicator coding does this by recording whether an observation belongs to particular categories.

For a factor with more than two categories, the same basic idea used for a two-level factor extends to several categories: choose one category as the reference, and create one indicator column for each of the other categories. The reference category is represented by zeros in every indicator column. This makes the baseline visible in the data and gives the fitted model a clear comparison point.

Definition and Model

Definition: Suppose a factor \(G\) has \(K\) distinct categories. Choose one category as the reference category (also called the baseline). For each of the other \(K-1\) categories, define an indicator variable that equals 1 when an observation belongs to that category and 0 otherwise. An observation in the reference category has 0 in all \(K-1\) indicator columns.

Call the reference category category 1, and label the remaining categories \(2,\ldots,K\). For observation \(i\), let \(D_{ij}\) be the indicator for category \(j\): \(D_{ij}=1\) if observation \(i\) is in category \(j\), and \(D_{ij}=0\) otherwise. A factor-only regression model using this coding is

Y_i=\beta_0+\beta_2D_{i2}+\beta_3D_{i3}+\cdots+\beta_KD_{iK}+\varepsilon_i.

Here \(Y_i\) is the response for observation \(i\), \(\beta_0\) is the intercept, and \(\varepsilon_i\) is the model error. For interpreting the model as group means, assume that the conditional mean of the error is zero in every category: \(E(\varepsilon_i\mid G_i=k)=0\) for each category \(k\). The coding itself is a deterministic description of category membership; this mean-zero condition is what lets the model’s fitted values be interpreted as conditional means. The coding does not require the response to be normally distributed.

How to Read the Columns

Each indicator column answers one yes-or-no question: “Does this observation belong to the category named by this column?” Exactly one indicator is 1 for an observation in a nonreference category; all the other indicators are 0. For an observation in the reference category, every indicator is 0. Thus, the omitted category is not missing from the representation. Its membership is conveyed by the all-zero pattern.

For instance, if a factor has five categories and category 1 is the reference, there are four columns: one marking category 2, one marking category 3, one marking category 4, and one marking category 5. An observation in category 4 has the pattern \((0,0,1,0)\), with the entries ordered as \(D_2,D_3,D_4,D_5\). An observation in category 1 has \((0,0,0,0)\).

Worked Example: Coding a Four-Category Factor

A hypothetical data set records the type of material used for each test piece: oak, pine, maple, or birch. Choose oak as the reference category and state the indicator columns and patterns for all four materials.

Step 1: Name the columns. Since oak is the reference, make one column for each other material: \(D_{\text{pine}}\), \(D_{\text{maple}}\), and \(D_{\text{birch}}\). Each column is 1 for its named material and 0 otherwise.

Step 2: Assign the patterns. In the column order \(D_{\text{pine}},D_{\text{maple}},D_{\text{birch}}\), an oak piece has \((0,0,0)\), a pine piece has \((1,0,0)\), a maple piece has \((0,1,0)\), and a birch piece has \((0,0,1)\). For example, a hypothetical observation recorded as maple receives 1 in the maple column and 0 in each other column.

There are four categories and three indicators, so the number of columns is \(K-1=4-1=3\). The coding identifies the category of every observation, including oak, without giving oak its own indicator column.

What the Baseline Means in the Model

The all-zero pattern makes the intercept the model’s fitted mean for the reference category. To see this, set all indicators to zero in the model. For any observation in the reference category, the model reduces to \(Y_i=\beta_0+\varepsilon_i\), so its conditional mean is \(\beta_0\), using the mean-zero error condition.

For an observation in category \(j\), where \(j\) is not the reference, its own indicator equals 1 and the other indicators equal 0. Its conditional mean is therefore \(\beta_0+\beta_j\). Thus, \(\beta_j\) is the difference between the fitted mean for category \(j\) and the fitted mean for the reference category. This is the same reference-group interpretation used for a binary factor, applied separately to each nonreference category.

\begin{aligned} E(Y_i\mid G_i=\text{reference})&=\beta_0,\\ E(Y_i\mid G_i=j)&=\beta_0+\beta_j,\qquad j=2,\ldots,K. \end{aligned}

This relationship follows directly from the indicator values; it does not require estimating the coefficients first. Estimation determines the numerical values of the coefficients, while the coding determines what those coefficients represent.

Worked Example: Calculating Fitted Means From Indicator Coefficients

Suppose a hypothetical model predicts a clinic’s average waiting time, in minutes, from a four-category clinic-type factor. Type A is the reference, and the estimated model is

\widehat{Y}=18+4D_B-3D_C+6D_D.

The indicators mark types B, C, and D; type A has all indicators equal to zero. Calculate the fitted mean for each type.

Type A. Substitute \((D_B,D_C,D_D)=(0,0,0)\):

\widehat{Y}_A=18+4(0)-3(0)+6(0)=18\text{ minutes}.

Type B. Substitute \((1,0,0)\):

\widehat{Y}_B=18+4(1)-3(0)+6(0)=22\text{ minutes}.

Type C. Substitute \((0,1,0)\):

\widehat{Y}_C=18+4(0)-3(1)+6(0)=15\text{ minutes}.

Type D. Substitute \((0,0,1)\):

\widehat{Y}_D=18+4(0)-3(0)+6(1)=24\text{ minutes}.

The intercept, 18 minutes, is the fitted mean for type A. The coefficient 4 means that type B’s fitted mean is 4 minutes higher than type A’s; the coefficient \(-3\) means type C’s fitted mean is 3 minutes lower; and the coefficient 6 means type D’s fitted mean is 6 minutes higher. These are differences in fitted means, not claims that clinic type causes waiting time to change.

A Short Justification of the Fitted-Mean Formula

The general fitted-mean interpretation can be checked by evaluating the model category by category. In the reference category, every indicator is zero, so the linear predictor is \(\beta_0\). In a nonreference category \(j\), the indicator \(D_j\) equals 1 and every other indicator equals 0, so the linear predictor is \(\beta_0+\beta_j\). Taking the conditional expectation removes the error term because the model assumes \(E(\varepsilon_i\mid G_i=k)=0\). Therefore:

\begin{aligned} E(Y_i\mid G_i=\text{reference}) &=\beta_0+E(\varepsilon_i\mid G_i=\text{reference})=\beta_0,\\ E(Y_i\mid G_i=j) &=\beta_0+\beta_j+E(\varepsilon_i\mid G_i=j)=\beta_0+\beta_j. \end{aligned}

This calculation also shows why it is important to know which category each column marks. Without that information, a coefficient’s comparison category cannot be interpreted correctly.

Worked Example: Decoding a Five-Category Observation

A hypothetical agricultural experiment classifies plots by soil class: sandy, loamy, clay, silt, or peat. Sandy soil is chosen as the reference. A recorded observation has indicator values \((0,1,0,0)\), ordered as \(D_{\text{loamy}},D_{\text{clay}},D_{\text{silt}},D_{\text{peat}}\). Identify its soil class. Then use the hypothetical fitted model below to calculate the predicted yield.

\widehat{Y}=42+5D_{\text{loamy}}+2D_{\text{clay}}-4D_{\text{silt}}+7D_{\text{peat}},

The indicator for clay is the only one equal to 1, so the observation is a clay-soil plot. Substitute its pattern \((0,1,0,0)\) into the model:

\widehat{Y}=42+5(0)+2(1)-4(0)+7(0)=44.

The predicted yield for this observation is 44 units of yield. The baseline is sandy soil, whose fitted mean is 42 units; the clay coefficient of 2 indicates a fitted mean 2 units higher than sandy soil. There are five categories and four indicator columns, consistent with \(K-1=5-1=4\).

Common Mistakes

Giving the baseline its own indicator column. In this reference-category scheme, the baseline is identified by zeros in all the indicator columns. First name the reference category, then create columns for the other categories.
Reading a coefficient as the category’s entire mean. A nonreference coefficient is a difference from the baseline. Add it to the intercept to obtain that category’s fitted mean.
Forgetting the column order. The pattern \((0,1,0)\) has no category meaning unless the indicators’ order is known. Write the column names before interpreting a pattern.
Reversing the comparison. A coefficient describes the nonreference category minus the reference category. A negative coefficient means the nonreference category has the lower fitted mean, under the stated model.

Key takeaway: A factor with \(K\) categories is represented by \(K-1\) indicator columns when one category is chosen as the reference. Each column marks one nonreference category; the all-zero pattern marks the reference. In a factor-only model with a mean-zero error in each category, the intercept is the reference fitted mean and each indicator coefficient is that category’s fitted difference from the reference.

Check Your Understanding

Answer each question using the column order stated.

A factor has six categories and one is designated as the reference. How many indicator columns are used in this coding?
A four-category factor uses indicators \(D_B,D_C,D_D\), with category A as the reference. What pattern represents category C?
In a model with category A as the reference, the intercept is 12 and the coefficient for category D is \(-2.5\). What is the fitted mean for category D, and what does its coefficient compare?
For indicators ordered \(D_{\text{east}},D_{\text{north}},D_{\text{west}}\), what category is represented by \((0,0,0)\) if south is the reference?
Explain why the coefficient of an indicator is not, by itself, the fitted mean for that category.