Why Categories Need a Numerical Representation
A regression model uses numerical predictor columns, but many useful predictors describe groups: treatment type, clinic, region, or machine setting. A tempting approach is to assign the categories numbers such as 1, 2, and 3 and put those numbers directly into the model. That treats the categories as though they lie on a quantitative scale: it assumes that the difference between categories 1 and 2 has the same meaning as the difference between 2 and 3. For labels without that meaning, this is not an appropriate representation.
Indicator variables provide a different approach. Each indicator records whether an observation belongs to a specified category. For a predictor with \(k\) levels, a model with an intercept uses \(k-1\) indicators. This represents the group means while avoiding an exact linear dependence among the predictor columns.
Building the Indicator Columns
Suppose a predictor has levels A, B, and C. Choose one level to leave out; call it A for this illustration. Define \(D_B=1\) for an observation in B and \(D_B=0\) otherwise. Define \(D_C\) in the same way for C. An observation in A has \(D_B=0\) and \(D_C=0\). The resulting coding is:
| Category | Intercept column | \(D_B\) | \(D_C\) |
|---|---|---|---|
| A | 1 | 0 | 0 |
| B | 1 | 1 | 0 |
| C | 1 | 0 | 1 |
The regression model for a response \(Y_i\) is
Here \(\beta_0\), \(\beta_B\), and \(\beta_C\) are model coefficients, and \(\varepsilon_i\) is the observation's error relative to the model mean. For an observation in A, both indicators are zero, so its model mean is \(\beta_0\). For B, the mean is \(\beta_0+\beta_B\); for C, it is \(\beta_0+\beta_C\). Thus, one intercept and two indicators can represent three category-specific means. The indicators are not category scores: they mark membership in particular groups.
Worked Example: Coding Three Treatment Groups
In a hypothetical experiment, a response is measured in points for participants assigned to treatment A, B, or C. The observed responses are A: 10 and 12; B: 14 and 16; C: 9 and 13. Use A as the omitted level and find the fitted category means from a regression containing only the group indicators and an intercept.
First encode each observation. An A observation has \((D_B,D_C)=(0,0)\), a B observation has \((1,0)\), and a C observation has \((0,1)\). The model is \(Y_i=\beta_0+\beta_BD_{Bi}+\beta_CD_{Ci}+\varepsilon_i\).
With only these group indicators in the model, the fitted mean for each group is that group's sample mean. The three means are
For A, the fitted mean is \(\hat{\beta}_0\), so \(\hat{\beta}_0=11\) points. For B, it is \(\hat{\beta}_0+\hat{\beta}_B\), giving \(\hat{\beta}_B=15-11=4\) points. For C, it is \(\hat{\beta}_0+\hat{\beta}_C\), giving \(\hat{\beta}_C=11-11=0\) points. The fitted model is therefore \(\hat{Y}=11+4D_B+0D_C\), in points. Substituting the codes gives fitted means of 11 for A, \(11+4(1)+0(0)=15\) for B, and \(11+4(0)+0(1)=11\) for C.
The zero coefficient for \(D_C\) in this example means the fitted mean for C equals the fitted mean for A. It does not mean that C lacks a code: the \(D_C\) column is still needed to allow the model to represent a C mean that could differ from A in another data set.
Why Not Use All k Indicators?
For every observation, exactly one category applies. If all \(k\) indicators are included, their values therefore add to 1 for every row. But a regression model with an intercept already has a column of 1s. The intercept column is consequently the sum of all \(k\) category-indicator columns. This exact dependence is called perfect multicollinearity, or a rank deficiency: the design matrix columns are not independent, so the coefficients cannot be uniquely determined.
For three levels, if \(D_A,D_B,D_C\) are all included, the row for every observation satisfies \(1=D_A+D_B+D_C\). In matrix terms, the intercept column equals the sum of those three columns. The model cannot decide uniquely how to distribute a constant amount among the intercept and the three coefficients, even though it may still be possible to calculate fitted values.
For example, the two coefficient sets \((\beta_0,\beta_A,\beta_B,\beta_C)=(10,0,2,5)\) and \((11,-1,1,4)\) give exactly the same predictions with all three indicators. For category A, the first set predicts \(10+0=10\), while the second predicts \(11-1=10\). For B, they predict \(10+2=12\) and \(11+1=12\). For C, they predict \(10+5=15\) and \(11+4=15\). The predictions agree, but the coefficients are not uniquely identified.
Leaving out one indicator removes this particular dependence. For the three-level coding above, the intercept column and the \(D_B,D_C\) columns are not related by a fixed linear combination that holds across all three category rows. This gives the model a unique way to express three distinct group means, provided observations are available in every level.
A Short Derivation: The Indicators Can Fit Each Group Mean
The claim that this model can represent any set of \(k\) group means follows directly from its form. Label the omitted category A and the other levels \(2,\ldots,k\). Let the desired fitted means be \(m_A,m_2,\ldots,m_k\). Set the intercept to \(m_A\), and set the coefficient for indicator \(j\) to \(m_j-m_A\). For an observation in A, all indicators are zero and the fitted mean is \(m_A\). For an observation in any other level \(j\), only indicator \(j\) is one, so:
Thus the coding can represent all \(k\) category means without needing \(k\) indicator coefficients in addition to the intercept. In a model containing only the categorical predictor, least squares chooses the group sample means as the fitted means. To see why, write the sum of squared residuals for group \(g\) as
where \(n_g\) is the number of observations in group \(g\), \(\bar{y}_g\) is its sample mean, and \(\hat{m}_g\) is its fitted mean. The first term does not depend on the fitted mean, and the second term is minimized at \(\hat{m}_g=\bar{y}_g\). Since the indicator coding can set each group's fitted mean independently, least squares fits each group mean at its sample mean.
Worked Example With Four Levels
Worked Example: Constructing Four Indicator Codes
A hypothetical analyst records processing time, in minutes, for items handled under settings A, B, C, and D. To illustrate the model representation, suppose the fitted category means are 20, 23, 18, and 27 minutes, respectively. Use A as the omitted level and write the indicator coding and fitted model.
There are four levels, so the model uses \(4-1=3\) indicators. Let \(D_B,D_C,D_D\) mark settings B, C, and D. Their codes are: A \((0,0,0)\), B \((1,0,0)\), C \((0,1,0)\), and D \((0,0,1)\). The model has the form \(\hat{Y}=\hat{\beta}_0+\hat{\beta}_BD_B+\hat{\beta}_CD_C+\hat{\beta}_DD_D\).
For A, all indicators are zero, so \(\hat{\beta}_0=20\) minutes. The other coefficients are the differences between their fitted means and 20:
The fitted equation is \(\hat{Y}=20+3D_B-2D_C+7D_D\), with fitted time in minutes. Substituting B's code gives \(20+3(1)-2(0)+7(0)=23\) minutes. For C it gives \(20+3(0)-2(1)+7(0)=18\) minutes, and for D it gives \(20+3(0)-2(0)+7(1)=27\) minutes. The three indicators and intercept represent all four stated means.
If all four indicators were added while retaining the intercept, then \(D_A+D_B+D_C+D_D=1\) for every item. The columns would be redundant, so there would be no unique set of four indicator coefficients and an intercept.
Using the Coding Carefully
The \(k-1\) rule addresses one specific structural problem: an intercept and all \(k\) category indicators have an exact linear dependence. It does not guarantee that every regression design matrix is free of other dependencies. For example, another predictor might itself be an exact copy of an indicator column. Nor does the rule repair a data set in which a category has no observations; a mean for an unobserved category cannot be estimated from those data.
A categorical predictor with more than two levels should generally be represented with indicators rather than a single numeric code unless the numbers genuinely represent a quantitative scale and the intended model treats them that way. A code such as A=1, B=2, C=3 imposes a single linear change across the category labels. Indicator coding instead permits each category mean to differ without imposing equal spacing.
The same construction can be used alongside other predictors. The indicators then add category-specific shifts to the model's fitted value, while the other predictors have their own columns and coefficients. The algebraic reason for omitting one indicator remains the same whenever the model includes an intercept. The coding changes how the coefficients are represented, not the information in the observed responses or the set of category means the model can fit.
Common Mistake
- Wrong reasoning: “There are three categories, so the regression needs three indicators, plus its intercept, to identify all three groups.”
- Correction: With an intercept, three indicators sum to the intercept column for every observation. That exact dependence makes the coefficients non-unique. Use two indicators; the intercept and two indicators can still represent three separate group means.
- Another mistake: Treating the labels A, B, and C as scores 1, 2, and 3 without a substantive reason. Indicator columns avoid imposing a numerical order or equal spacing on category labels.
Check Your Understanding
Try these questions without looking back.
- A predictor has five observed categories and the regression includes an intercept. How many indicator columns should be used?
- For levels A, B, and C, with A omitted, what are the \((D_B,D_C)\) codes for an observation in C?
- Explain why the intercept column is redundant with all \(k\) category indicators.
- A model has an intercept and indicators for B and C, with A omitted. Write its fitted mean for an observation in A.
- Does using \(k-1\) indicators guarantee that no other predictors in a larger model are redundant? Explain briefly.