Why a Complete Set of Indicators Can Cause a Problem
A categorical predictor with several groups is often represented by indicator variables: numerical columns that mark which group each observation belongs to. The difficulty arises when a regression includes both an intercept and an indicator for every category. For each observation, exactly one of those indicators equals 1, so their sum duplicates the intercept column. This is exact collinearity, not merely a strong association among predictors.
The consequence is specific: the observed responses can still have well-defined fitted values, but the individual coefficients in this redundant parameterization cannot be uniquely determined. Choosing a reference category or using a model without an intercept removes the duplication. These choices change how coefficients are represented and interpreted, not the category-specific fitted means.
The Redundant Model
Consider a response \(Y_i\) and a categorical predictor with \(K\) categories. An intercept-plus-complete-indicators model is
Here \(\beta_0\) is the intercept, \(\beta_j\) is the coefficient attached to category \(j\), and \(\varepsilon_i\) is the error for observation \(i\). For modeling category means, the assumptions are that the conditional mean is represented by the stated category-specific linear expression and that the errors have mean zero given the category. Those assumptions describe the model's mean structure; the redundancy shown below is an algebraic fact and does not depend on a normality assumption or an error variance assumption.
Because exactly one indicator is 1 for each observation, the predictor columns satisfy
The right-hand side is exactly the intercept column: a column containing 1 for every observation. Thus one model column is an exact linear combination of the others. This is the design-matrix definition of exact collinearity. It differs from near collinearity, where a combination of columns is close to, but not exactly, another column.
Worked Example: Finding the Duplicated Column
Suppose six hypothetical patients are classified by clinic, with categories North, Central, and South. Their indicator values are shown below.
| Patient | Intercept | North | Central | South |
|---|---|---|---|---|
| 1 | 1 | 1 | 0 | 0 |
| 2 | 1 | 1 | 0 | 0 |
| 3 | 1 | 0 | 1 | 0 |
| 4 | 1 | 0 | 1 | 0 |
| 5 | 1 | 0 | 0 | 1 |
| 6 | 1 | 0 | 0 | 1 |
For each patient, the sum of the three category indicators is 1. In vector form, the North column plus the Central column plus the South column equals the intercept column:
Therefore the four columns (intercept, North, Central, and South) are linearly dependent. Including all four does not provide four independently varying pieces of information. The dependency is exact even though no two of the category-indicator columns are identical.
Why the Coefficients Are Not Identifiable
A coefficient is identifiable in this model when the data and model specification determine its value uniquely. With the intercept and all category indicators, they do not. For any number \(c\), replace the coefficients by
For an observation in category \(g\), the original fitted value is \(\beta_0+\beta_g\). Under the changed coefficients it is
Every fitted value is unchanged, no matter what value of \(c\) is chosen. Consequently, the data cannot distinguish among these coefficient sets. Least squares can determine the fitted category means, but it cannot select a unique intercept and a unique coefficient for every category in this redundant model.
More formally, if all \(K\) categories are represented in the data, the \(K\) indicator columns are linearly independent: each nonempty category has rows on which its indicator is 1 and the other indicators are 0. Adding the intercept creates one dependent column, because the intercept equals the sum of those \(K\) columns. The design matrix therefore has rank \(K\), not \(K+1\). This rank statement describes the redundancy; the number of observations does not remove it.
Worked Example: Multiple Coefficient Sets, Identical Fits
For the six hypothetical patients in the table, suppose the recorded response is a clinic waiting time, in minutes: North has values 8 and 10, Central has 12 and 14, and South has 15 and 17. The category means are
One coefficient set for the redundant model is \(\beta_0=9\), \(\beta_{\text{North}}=0\), \(\beta_{\text{Central}}=4\), and \(\beta_{\text{South}}=7\). It gives fitted values \(9+0=9\) for North, \(9+4=13\) for Central, and \(9+7=16\) for South.
Take \(c=2\) in the coefficient transformation. A second set is \(\beta_0'=7\), \(\beta_{\text{North}}'=2\), \(\beta_{\text{Central}}'=6\), and \(\beta_{\text{South}}'=9\). Its fitted values are \(7+2=9\), \(7+6=13\), and \(7+9=16\), respectively. The coefficients differ, but every patient receives the same fitted value under both sets.
These fitted category means minimize the sum of squared residuals. Within a group with observations \(y_1,\ldots,y_m\), the sum of squared deviations from a proposed fitted value \(a\) is minimized at that group's mean. Here the fitted means give residuals \(-1,1\) in each clinic, so the total squared error is
The least-squares fit is therefore well-defined, while the intercept and all three category coefficients in this particular coding are not individually identifiable.
Removing the Redundancy
The most common solution is reference-category coding. Select one category as the reference and include indicators for the other \(K-1\) categories along with the intercept. For example, taking North as the reference gives
For North observations both indicators are 0, so their fitted mean is \(\beta_0\). For Central observations the fitted mean is \(\beta_0+\beta_{\text{Central}}\), and for South observations it is \(\beta_0+\beta_{\text{South}}\). The intercept is the reference-category mean; each indicator coefficient is the difference between that category's mean and the reference mean. The omitted category is not omitted from the data: it is represented by the intercept.
Worked Example: Interpreting Reference-Category Coefficients
Use the same hypothetical waiting times: North \(8,10\), Central \(12,14\), and South \(15,17\). Take North as the reference category. Its mean is \(9\) minutes, so the fitted intercept is \(\hat{\beta}_0=9\). The Central and South differences from North are
The fitted mean for a Central patient is \(9+4=13\) minutes, and for a South patient it is \(9+7=16\) minutes. For North, both indicators equal 0, giving \(9+0+0=9\) minutes. The reference-coded model thus reproduces all three category means using three coefficients, rather than four.
The coefficient of 4 minutes describes a difference in mean waiting time between the Central and North categories in this fitted model. It does not establish that being treated at one clinic rather than another causes a change in waiting time.
A second valid choice is to omit the intercept and include indicators for all categories. In that parameterization, each indicator coefficient directly represents its category's fitted mean. It has \(K\) coefficients for \(K\) category means, with no separate intercept duplicating their indicators. This is not the same coefficient interpretation as reference coding, though it gives the same fitted means when the model includes only the categorical predictor.
Worked Example: Using All Indicators Without an Intercept
For the same waiting-time data, write a model without an intercept:
A North observation has indicators \(1,0,0\), so its fitted value is \(\gamma_{\text{North}}\). To fit the North mean, set \(\hat{\gamma}_{\text{North}}=(8+10)/2=9\) minutes. Similarly, \(\hat{\gamma}_{\text{Central}}=(12+14)/2=13\) minutes and \(\hat{\gamma}_{\text{South}}=(15+17)/2=16\) minutes. Thus the three fitted category means are 9, 13, and 16 minutes, exactly as before. Here each coefficient is a category mean; unlike in reference coding, the Central and South coefficients are not differences from North.
Checking the Coding and Avoiding Common Mistakes
The exact dependency depends on the coding and on which observations are represented. With an intercept and a complete, mutually exclusive set of category indicators, the row-wise indicator sum is 1, so the intercept is redundant. If a category is omitted from the indicators, the indicators are not complete and this particular identity no longer holds. Other dependencies could still arise for other reasons, but they are not the complete-indicator redundancy discussed here.
Before fitting a model, check that the categories are defined to be exhaustive for the observations being analyzed and that each observation belongs to exactly one category. If some observations have no recorded category or can belong to multiple categories, the sum-to-one identity may fail. Also check whether a category has any observations: an indicator that is zero for every row cannot be estimated from those data.
- Mistake: treating the four coefficients as separately meaningful. With an intercept and three complete indicators, the data do not uniquely determine all four coefficients. Many coefficient sets yield the same fitted values.
- Correction: Use a nonredundant coding and interpret coefficients according to that coding. With a reference category, the intercept is its mean and the other coefficients are differences from it.
- Mistake: thinking a reference-category model discards the reference group. Its observations remain in the analysis; they are represented by the intercept.
- Correction: Explain which category is the reference and give coefficients in context and units. If using all indicators without an intercept, explain that the coefficients represent category means instead.
A software package may flag the redundant column, fail to compute a unique coefficient vector, or automatically remove a column. Such behavior is a response to the rank deficiency, not evidence that one category has no relationship with the response. The substantive fitted category means can remain estimable even when the redundant coefficient representation is not unique.
Check Your Understanding
Answer each question by referring to the model's columns and coefficient meanings.
- A predictor has four mutually exclusive categories. Why is an intercept plus four category indicators exactly collinear?
- In an intercept-plus-complete-indicators model, explain why adding the same constant to every category coefficient and subtracting it from the intercept leaves all fitted values unchanged.
- With reference-category coding, what does the intercept represent, and what does a coefficient for a non-reference category represent?
- In a model with all category indicators but no intercept, what does each indicator coefficient represent?
- Why does omitting a category's indicator under reference coding not remove observations from that category?