Why Categorical Coding Changes Coefficients
A regression model cannot include a category label such as “treatment group” as a numerical predictor without representing its categories in some way. Several coding schemes can represent the same groups and produce the same fitted values. The coefficient estimates may look different under those schemes, but a particular group comparison should not change merely because the coding changed.
The key is to identify the quantity being compared. A coefficient in one coding may equal a group difference, while a coefficient in another coding may equal half that difference or its negative. To compare results across codings, translate the target comparison into the appropriate coefficient or linear contrast, then calculate its estimate and standard error.
The Model and the Conditions for Equivalent Tests
Consider an outcome \(Y\) and two groups, A and B. In a reference-cell coding with A as the reference group, let \(x=0\) for A and \(x=1\) for B. The regression model is
The fitted mean for group A is \(\beta_0\), and the fitted mean for group B is \(\beta_0+\beta_1\). Thus \(\beta_1\) is the B-minus-A difference. In this coding, testing equal group means is the coefficient test \(H_0:\beta_1=0\).
For a valid t test in this simple regression, assume the model has the correct group-specific means, observations have independent errors with mean zero and common variance \(\sigma^2\), and errors are normally distributed for an exact finite-sample t procedure. The model must also be identifiable: its columns cannot be redundant. Under these conditions the residual degrees of freedom and standard errors are calculated from the fitted model. A different coding preserves the same inference only if it represents the same model and the same target comparison.
More generally, write a regression model as \(Y=X\beta+\varepsilon\). Suppose a new design matrix is formed as \(X^*=XA\), where \(A\) is an invertible matrix. The same fitted mean can then be written \(X^*\beta^*\) with \(\beta^*=A^{-1}\beta\). A target \(c^\mathsf{T}\beta\) becomes \(c^\mathsf{T}A\beta^*\). Its estimate is unchanged by this re-expression, and its standard error is unchanged when the covariance is transformed along with the coefficients. The individual coefficient estimates need not be unchanged.
Worked Example: Comparing Two Groups With A as the Reference
Worked Example: Comparing Two Groups With A as the Reference
Suppose a hypothetical class exercise records quiz scores, in points, for 12 fictional students in group A and 12 in group B. The group means are 70 and 80 points. The fitted two-group model has residual sum of squares \(1188\) points squared and \(22\) residual degrees of freedom. Assume the independent, equal-variance normal-error model described above is reasonable for these hypothetical observations.
With A as the reference, the estimated intercept is the A mean, and the estimated coefficient for the B indicator is the difference:
The residual mean square is \(1188/22=54\) points squared. Since each group has 12 observations, the standard error of the difference is
To test whether the group means differ, use \(H_0:\beta_1=0\) against \(H_a:\beta_1\ne0\). The test statistic is
The two-sided p-value is approximately \(0.003014\). At the \(0.05\) significance level, reject \(H_0\): the hypothetical data provide evidence that the population mean quiz scores differ between the groups. The estimated difference is 10 points, with group B higher. The p-value is calculated under the null model; it is not the probability that the null hypothesis is true.
Changing the Reference Category
If B becomes the reference group, the indicator is instead \(z=1\) for A and \(z=0\) for B. The fitted mean for B is now the intercept, and the coefficient for \(z\) is the A-minus-B difference. This reverses the sign of the coefficient but does not change the absolute value of the test statistic.
Worked Example: Reversing the Reference Category
Use the same hypothetical quiz-score data, model, and assumptions as in the previous example. With B as the reference, the fitted mean for B is 80 points, and the coefficient of the A indicator is \(70-80=-10\) points. The contrast of interest is still the difference between the two group means; it is now expressed as A minus B rather than B minus A.
The groups and residual variation have not changed, so the standard error of this difference remains 3 points. Testing equal means gives
For a two-sided test, changing the sign of \(t\) leaves the p-value unchanged: \(p=0.003014\). The conclusion is the same as before. The estimated A-minus-B difference is \(-10\) points; equivalently, the estimated B-minus-A difference is 10 points. The choice of reference changes the coefficient’s sign and interpretation, not the underlying comparison.
Alternative Coding: Effect Coding
A common alternative for two groups assigns \(x=-1\) to A and \(x=+1\) to B. In the model \(E(Y\mid x)=\beta_0+\beta_1x\), the fitted means are \(\beta_0-\beta_1\) for A and \(\beta_0+\beta_1\) for B. Therefore, the B-minus-A difference is \(2\beta_1\), not \(\beta_1\). In a balanced two-group comparison, the intercept is the average of the two group means.
Worked Example: Testing the Difference With Effect Coding
Return to the same hypothetical class data. Under effect coding, the group means imply
The coefficient \(\beta_1\) is half the B-minus-A difference. With 12 observations coded \(-1\) and 12 coded \(+1\), the sum of squared deviations of \(x\) from its mean is \(24\). Thus the standard error of \(\hat{\beta}_1\) is
The group difference is the contrast \(2\beta_1\). Its estimate and standard error are \(2(5)=10\) points and \(2(1.5)=3\) points, respectively. Consequently,
This is the same two-sided test as before, so \(p=0.003014\), with the same conclusion. Testing \(\beta_1=0\) gives the same statistic here because multiplying both the estimate and its standard error by 2 cancels in the ratio. But the coefficient’s numerical value is not itself the group difference: it is 5 points, while the difference is 10 points.
Three Groups: A Comparison as a Contrast
With more than two groups, a group comparison may involve multiple coefficients even when one category is the reference. The standard error must account for covariance between those coefficients. Re-expressing the model does not remove that covariance requirement; it changes how the target contrast is written.
Worked Example: Comparing Two Nonreference Groups
Consider a hypothetical regression comparison of three groups, A, B, and C, under the same independent, equal-variance normal-error assumptions. With A as reference, suppose the estimated coefficients are \(\hat{\beta}_B=8\) and \(\hat{\beta}_C=12\) outcome units. Suppose the fitted model reports \(SE(\hat{\beta}_B)=3\), \(SE(\hat{\beta}_C)=4\), and \(\operatorname{Cov}(\hat{\beta}_B,\hat{\beta}_C)=6\), in squared outcome units. The residual degrees of freedom are 27.
The B-minus-C comparison is not either coefficient alone. It is the contrast \(\beta_B-\beta_C\), estimated as \(8-12=-4\) outcome units. Its standard error uses the covariance:
The test statistic for equal B and C means is \(t=-4/\sqrt{13}=-1.109\), with \(df=27\). Since \(|t|<2.052\), the two-sided \(0.05\)-level test does not reject equality of the B and C means. This is not evidence that the means are exactly equal; it means the data do not provide sufficient evidence of a difference at that significance level.
An alternative coding could include a coefficient defined directly as \(\theta=\mu_B-\mu_C\), where \(\mu_A,\mu_B,\mu_C\) denote the group means. That coefficient would be estimated as \(-4\), with standard error \(\sqrt{13}\), and its test of \(\theta=0\) would have the same statistic and conclusion. The reference-cell coding uses a contrast of two coefficients; the alternative coding puts that contrast into one coefficient.
Why the Same Test Is Preserved
The invariance can be seen directly. If a contrast estimate is \(c^\mathsf{T}\hat{\beta}\), its standard error is the square root of \(c^\mathsf{T}\widehat{\operatorname{Var}}(\hat{\beta})c\). A change of coordinates alters both the coefficient vector and its covariance matrix. When the contrast is translated to the new coordinates, its estimate and variance remain the same. Therefore, its t statistic, degrees of freedom, p-value, and confidence interval are the same, apart from rounding.
This conclusion requires care about what is held fixed. The model must have the same fitted-mean possibilities, such as the same group effects and other included predictors, and the hypothesis must concern the same comparison. If a model includes interactions, for example, the group difference can depend on the value of another predictor; the contrast must then specify the value at which the comparison is made. A coefficient test that answers a different question is not made equivalent merely by changing its label.
Common Mistake: Treating Every Coefficient as the Same Group Difference
A common mistake is to compare coefficients from two output tables as though each coefficient directly represented the same quantity. For example, the reference-cell coefficient in the two-group example was 10 points, while the effect-coded coefficient was 5 points. Concluding that the group difference was cut in half would be wrong: under effect coding, the group difference is \(2\beta_1\). Similarly, reversing the reference category changed the coefficient from 10 to \(-10\) because it changed the direction of subtraction.
The correction is to write down the target comparison in terms of group means first. Then express that same comparison using the coefficients in each coding. Check the estimate, its standard error, and the test together. A numerical coefficient by itself does not identify the comparison it represents.
Check Your Understanding
Use the coding definitions and results in this tutorial to answer the following.
- With A as reference, what does the coefficient of a B indicator represent? How does its meaning change if B is the reference?
- Under effect coding \(x=-1\) for A and \(x=+1\) for B, how is the B-minus-A difference expressed using the slope coefficient?
- Why does reversing a two-group reference category change the sign of a coefficient but not the two-sided p-value?
- For a contrast \(\hat{\beta}_B-\hat{\beta}_C\), why must the covariance between the coefficient estimates be included when calculating its standard error?
- Give one condition required for two coding schemes to yield equivalent tests of a group comparison.