Why the Intercept and Every Indicator Are Redundant
A categorical predictor with \(k\) levels can be represented by \(k\) indicator variables: one for each level. But if a model also contains an intercept, those \(k\) indicators contain the intercept already. This is not an approximation or a problem caused by a particular sample size. It is an exact linear dependence whenever every observation belongs to exactly one of the listed levels.
The consequence is important for interpreting a fitted model. The data can determine fitted values while failing to determine a unique value for each coefficient. Treatment coding resolves this particular redundancy by omitting the indicator for a reference category. The explanation below is aligned with the SOA Exam PA (Predictive Analytics) syllabus.
Consider a model with an intercept and one indicator for every category. Its categorical part of the linear predictor is \(\beta_0+\sum_{j=1}^{k}\beta_jD_{ij}\). Because the indicators sum to 1, the intercept column is the sum of the \(k\) indicator columns. One column is therefore an exact linear combination of the others.
This relationship is often called the dummy-variable trap or perfect multicollinearity. The word “trap” refers to the redundant parameterization, not to a defect in the observations. The indicators correctly record category membership; it is their combination with the intercept that introduces redundancy.
What Redundancy Means for Coefficients
A coefficient vector is identifiable from a model matrix only if different coefficient vectors cannot produce the same fitted values. Here they can. To show this directly, write the model matrix as \(X=[\mathbf{1},D_1,\ldots,D_k]\), where \(\mathbf{1}\) is the intercept column and \(D_j\) is the column of indicators for category \(j\). The relation among its columns gives a nonzero vector \(c=(1,-1,\ldots,-1)^\mathsf{T}\) for which \(Xc=0\).
If \(\boldsymbol{\beta}\) is any coefficient vector, then \(\boldsymbol{\beta}+tc\) produces the same fitted values for every real number \(t\). The model matrix cannot distinguish among these coefficient vectors. This proves that the full set of coefficients is not uniquely identifiable. It does not imply that the fitted values must be different or that the model cannot make predictions for observed categories.
For ordinary least squares, the fitted values minimize the sum of squared residuals. That minimization can still have a well-defined set of fitted values even when more than one coefficient vector attains the minimum. However, the usual inverse formula involving \((X^\mathsf{T}X)^{-1}\) does not apply when the columns of \(X\) are linearly dependent: \(X^\mathsf{T}X\) is singular. Software may identify a redundant column or impose a computational convention, but such a convention does not make the original coefficients uniquely determined by the data.
Omitting a Reference Indicator
Treatment coding keeps the intercept and omits the indicator for one chosen category, called the reference category. Suppose category 1 is the reference. The predictor becomes \(\beta_0+\sum_{j=2}^{k}\beta_jD_{ij}\). An observation in category 1 has all included indicators equal to 0, so its fitted linear predictor is \(\beta_0\). An observation in category \(j\), for \(j\ne1\), has its own indicator equal to 1, so its predictor is \(\beta_0+\beta_j\).
When all \(k\) categories occur in the data and there are no other column dependencies, this coding removes the redundancy between the intercept and the category indicators. There are \(k\) coefficients: an intercept plus \(k-1\) indicator coefficients. The intercept represents the reference category’s fitted linear predictor, and each indicator coefficient is the difference in linear predictor from that reference. For a model with a link function, these statements concern the linear predictor; they do not necessarily describe differences on the response scale.
Worked Examples
Worked Example: Three Categories and Two Equivalent Coefficient Sets
A hypothetical study records a numeric response for six accounts, classified into service tiers A, B, and C. The observed responses are A: 8 and 12; B: 13 and 15; C: 18 and 20. Fit a categorical-only least-squares model conceptually, demonstrate the redundancy when all indicators are included, and then express the fitted category means using A as the reference.
Method. In a categorical-only least-squares model, the fitted value for each observed category is that category’s sample mean. Compute the means first:
For A, the mean is \((8+12)/2=10\). For B, it is \((13+15)/2=14\). For C, it is \((18+20)/2=19\). Thus the fitted values for A, B, and C are 10, 14, and 19, respectively.
With an intercept and all three indicators, one coefficient set giving those fitted values is \(\beta_0=2,\ \beta_A=8,\ \beta_B=12,\ \beta_C=17\). For A the prediction is \(2+8=10\); for B it is \(2+12=14\); for C it is \(2+17=19\).
Now add \(t=3\) to the intercept and subtract 3 from every indicator coefficient. The new coefficients are \(\beta_0=5,\ \beta_A=5,\ \beta_B=9,\ \beta_C=14\). The predictions remain \(5+5=10,\ 5+9=14,\ 5+14=19\). This verifies the exact dependence with actual values.
With A as the reference, omit \(D_A\). The intercept is 10. The B indicator coefficient is \(14-10=4\), and the C indicator coefficient is \(19-10=9\). The resulting predictions are A: \(10\), B: \(10+4=14\), and C: \(10+9=19\).
Check. The fitted values agree under both parameterizations. The residual sum of squares is \((8-10)^2+(12-10)^2+(13-14)^2+(15-14)^2+(18-19)^2+(20-19)^2=4+4+1+1+1+1=12\). The common mistake this example avoids is treating a coefficient set from the full-indicator model as the only possible one.
Worked Example: Omitting a Reference Does Not Fix an Unobserved Level
A hypothetical portfolio analysis declares four risk bands, A through D, but the training data contain only observations in A and B. The recorded responses are A: 6 and 8; B: 10 and 12. Determine the rank of the design matrix after using A as the reference, and identify what can and cannot be estimated from these data.
Method. First calculate the observed group means: A has mean \((6+8)/2=7\), and B has mean \((10+12)/2=11\). After omitting the A indicator, the design matrix has columns for the intercept and indicators \(D_B,D_C,D_D\). Its rows have only two possible forms:
The A rows have the first form and the B rows the second. These two row types are linearly independent, so the matrix has rank 2. It has four columns, and therefore still has two unidentifiable coefficient directions. In particular, the \(D_C\) and \(D_D\) columns are all zero in the training data.
The observed fitted means determine an intercept of 7 and a B coefficient of \(11-7=4\). The coefficients for C and D do not affect any training fitted value, so the training data do not determine them. A prediction for an observation in C or D would depend on those undetermined coefficients.
Check. The prediction for observed A is 7, and for observed B it is \(7+4=11\), as required. Omitting A removes the dependence caused by having an intercept and every indicator, but it cannot create information about categories C and D that never occur. The common mistake is to conclude that dropping one indicator guarantees a full-rank design matrix.
Worked Example: The Dependence Persists Alongside a Numeric Predictor
A hypothetical pricing analysis uses service tier A, B, or C and a numeric exposure measure \(z\). The observed pairs \((z,y)\) are A: \((0,5),(1,7)\); B: \((0,8),(1,10)\); and C: \((0,4),(1,6)\). Consider an additive linear model with tier indicators and a common slope for \(z\). Show one full-indicator parameterization, then express the same fitted values with A as the reference.
Model and method. Use the additive linear predictor \(\eta_i=\beta_0+\beta_AD_{iA}+\beta_BD_{iB}+\beta_CD_{iC}+\gamma z_i\). The data show an increase of 2 in each tier when \(z\) increases from 0 to 1, so the common slope is \(\gamma=2\). At \(z=0\), the fitted values by tier are 5, 8, and 4.
One full-indicator coefficient set is \(\beta_0=1,\ \beta_A=4,\ \beta_B=7,\ \beta_C=3,\ \gamma=2\). At \(z=0\), the predictions are A: \(1+4=5\), B: \(1+7=8\), and C: \(1+3=4\). At \(z=1\), add 2 to each, giving 7, 10, and 6, which matches every recorded response.
Adding 2 to the intercept and subtracting 2 from each tier coefficient gives another set: \(\beta_0=3,\ \beta_A=2,\ \beta_B=5,\ \beta_C=1,\ \gamma=2\). For example, the B prediction at \(z=1\) is \(3+5+2=10\), unchanged. The numeric-predictor coefficient did not need to change because the redundant direction concerns only the intercept and the complete indicator set.
With A as reference, the intercept is 5, the B coefficient is \(8-5=3\), the C coefficient is \(4-5=-1\), and the common slope remains 2. These give predictions A: \(5+2z\), B: \(5+3+2z\), and C: \(5-1+2z\).
Check. Substitution at \(z=0\) gives 5, 8, and 4; at \(z=1\) it gives 7, 10, and 6. The common mistake is to assume that adding another predictor removes the dependence. It does not: for every observation, the complete set of tier indicators still sums to the intercept column.
Common Exam Trap: Misreading the Intercept
Suppose the full-indicator coefficients in the first example are \(\beta_0=2,\ \beta_A=8,\ \beta_B=12,\ \beta_C=17\). A tempting but incorrect interpretation is that the intercept, 2, is the fitted mean for A. Under this coding, the A indicator is also 1 for an A observation, so the fitted value is \(2+8=10\), not 2. In fact, the coefficients \(\beta_0=5,\ \beta_A=5,\ \beta_B=9,\ \beta_C=14\) give the same fitted values, even though the intercept is now 5.
The correction is to identify the coding before interpreting any coefficient. With an intercept and all indicators, the intercept alone is not a category mean, and the individual coefficients are not uniquely determined. With A omitted, the intercept is the fitted linear predictor for A, and the other coefficients are differences from A. Also check that each declared category is represented in the estimation data: an unobserved level can leave further unidentified coefficients even after a reference indicator is omitted.
Ask whether the model contains an intercept and one indicator for every mutually exclusive, exhaustive category.
If the indicators sum to 1 for every observation, their columns sum to the intercept column, so the full-indicator parameterization is redundant.
Use the intercept for the reference category and indicators for differences from it. Confirm that all remaining category levels actually occur and that no other predictor columns create dependence.
Check Your Understanding
For each question, identify the relevant design-matrix relationship before interpreting the coefficients.
- A factor has three levels, all observed, and a model includes an intercept and all three indicators. How many columns are in this part of the model matrix, what is its rank if there are no other dependencies, and how many independent linear dependencies are present? (Answer: 4 columns, rank 3, and 1 independent linear dependency.)
- With A as the reference, observed category means are A: 12, B: 15, and C: 9. What are the treatment-coded intercept and indicator coefficients? (Answer: Intercept 12, B coefficient 3, and C coefficient \(-3\).)
- A four-level factor has observed levels A and B only. With A omitted, the model has an intercept and indicators for B, C, and D. Assuming both A and B occur, what is the rank of these four columns, and why? (Answer: Rank 2; the intercept and B columns are independent, while the C and D columns are zero.)
- A model has an intercept, indicators for all three factor levels A, B, and C, and an additional numeric predictor \(z\). Give a nonzero coefficient-change direction that leaves every fitted value unchanged, listing changes in the order intercept, A, B, C, \(z\). (Answer: \((1,-1,-1,-1,0)\), or any nonzero scalar multiple.)