Indicator Columns for Categorical Predictors

Representing Categories as Columns

A categorical predictor records which group an observation belongs to: for example, a region, product type, or treatment label. Unlike a numerical predictor, its labels do not automatically carry meaningful distances. Coding the labels as numbers such as 1, 2, and 3 can therefore suggest a numerical order and spacing that the categories do not have. Indicator columns instead record category membership directly, using 1 for membership and 0 otherwise.

Suppose there are \(k\) observed categories, labelled \(C_1,\dots,C_k\), and \(m\) observations. For each category \(C_j\), its indicator column has entry 1 in rows belonging to \(C_j\) and entry 0 in all other rows. We assume each observation belongs to exactly one category. When using these columns in a linear model, the choice of which indicators to include matters: an intercept together with indicators for all \(k\) categories is redundant.

With reference category \(C_r\), the model has the form \(\mathbf{y}=\beta_0\mathbf{1}+\sum_{j\ne r}\beta_j\mathbf{d}_j+\boldsymbol{\varepsilon}\). Here \(\mathbf{y}\in\mathbb{R}^m\) is the response vector, \(\mathbf{1}\in\mathbb{R}^m\) is the column of ones, \(\beta_0\) and \(\beta_j\) are coefficients, and \(\boldsymbol{\varepsilon}\) represents the differences between observed and fitted responses. In design-matrix notation, the columns are \(\mathbf{1}\) followed by the non-reference indicators. The fitted value for an observation is the corresponding entry of the model’s prediction vector.

For an observation in the reference category, every included indicator is zero, so its fitted value from the categorical part of the model is \(\beta_0\). For an observation in category \(C_j\ne C_r\), its fitted value is \(\beta_0+\beta_j\). Thus \(\beta_0\) is the reference-category level and \(\beta_j\) is the difference from that level, not the level of category \(C_j\) itself.

Constructing and Reading a Design Matrix

Proof. For an observation in \(C_r\), all non-reference indicators are zero, so the model gives fitted level \(\beta_0\). To represent the specified level \(\mu_r\), it is necessary that \(\beta_0=\mu_r\). For any category \(C_j\ne C_r\), exactly its own included indicator is 1, so the fitted level is \(\beta_0+\beta_j\). Requiring this to equal \(\mu_j\), and using \(\beta_0=\mu_r\), gives \(\beta_j=\mu_j-\mu_r\). These values do produce every specified level by substitution. They are unique because the reference level fixes \(\beta_0\), and each remaining category level then fixes its own \(\beta_j\). \(\square

This result concerns the fitted category levels represented by the model. When other predictors are included additively, the intercept and category coefficients have the same baseline-and-difference interpretation at fixed values of those other predictors. If category interactions are included, the meaning of a coefficient depends on the interaction terms as well.

Estimating Category Levels from Data

For a model containing only an intercept and a categorical predictor, the least-squares fitted level for each observed category is that category’s sample mean. One way to see this without solving a large system is to consider one category’s responses \(y_1,\dots,y_s\), with mean \(\bar y\). For any proposed fitted level \(a\), the sum of squared errors satisfies \(\sum_{i=1}^s(y_i-a)^2=\sum_{i=1}^s(y_i-\bar y)^2+s(a-\bar y)^2\). The cross term vanishes because \(\sum_i(y_i-\bar y)=0\). Since the last term is nonnegative and is zero only when \(a=\bar y\), the sum is minimized at the category mean.

Why Including Every Indicator Is Redundant

If every observation belongs to exactly one of \(k\) categories, then the indicators satisfy \(\mathbf{d}_1+\cdots+\mathbf{d}_k=\mathbf{1}\). This identity explains a common design-matrix problem. If the model already contains an intercept column \(\mathbf{1}\), adding all \(k\) indicator columns gives a linear dependence among the columns. The data can determine fitted category levels, but they cannot uniquely determine all \(k+1\) coefficients in that redundant parameterization.

Proof. Fix any observation \(i\). It belongs to exactly one category, say \(C_j\). Therefore the \(i\)th entry of \(\mathbf{d}_j\) is 1, and the \(i\)th entries of all other category indicators are 0. The \(i\)th entry of \(\sum_{j=1}^k\mathbf{d}_j\) is consequently 1, which is the \(i\)th entry of \(\mathbf{1}\). This holds for every observation, so \(\mathbf{1}=\sum_{j=1}^k\mathbf{d}_j\). Rearranging gives a linear combination of the intercept and indicator columns equal to \(\mathbf{0}\), with nonzero coefficients; hence the columns are linearly dependent. \(\square

Common Pitfalls and Practical Checks

Do not omit the intercept while treating one category as the baseline. If the model contains only B and C indicators, an observation in A has both indicators equal to zero, so its fitted value is zero. In the previous example, that would incorrectly fit A at 0 rather than at its mean of 5. With an intercept and A as reference, A’s fitted level is the intercept; the B and C columns encode differences from A.

Do not include an intercept and every category indicator as though all coefficients were separately meaningful. Their columns are dependent, so distinct coefficient vectors can give identical fitted values. Choose a reference category and omit its indicator, or use a different explicitly stated parameterization without an intercept. Do not interpret a coefficient without checking which coding was used.

Only observed categories have estimable levels in this model. If a category is declared but has no observations, its indicator column is all zeros. The data then place no restriction on the coefficient for that category: changing that coefficient changes no fitted value. Remove the unused level or collect observations before attempting to interpret its coefficient.

The reference category changes the coefficient descriptions, but not the set of category fitted levels that the model can represent. To compare coefficient tables from different reference choices, translate each coefficient into a fitted category level first; the differences between those levels are the interpretable comparisons.