Comparing Categorical Predictor Blocks

Why Test a Categorical Predictor as a Block?

A regression model can include both quantitative predictors and categorical predictors, such as treatment group, region, or type of device. A categorical predictor with several levels generally requires several coefficients. The question is often not whether one particular coefficient differs from zero, but whether the categorical predictor as a whole improves the model after accounting for the other predictors.

The appropriate comparison is between a reduced model that omits the categorical predictor and a full model that includes it. A partial F test evaluates the reduction in residual sum of squares, or RSS, from adding the entire block of indicator coefficients. The number of restrictions tested is the number of coefficients in that block, not simply the number of category labels.

Definition: Suppose a categorical predictor has \(k\) observed levels and the regression model includes an intercept. Under reference-level indicator coding, one level is the reference and the remaining \(k-1\) levels each have an indicator coefficient. Testing the categorical predictor as a block means testing the joint null hypothesis that all \(k-1\) indicator coefficients equal zero, while the other predictors remain in both models.

Set Up the Nested Models and Hypotheses

Let \(Y_i\) be the outcome for observation \(i\), let \(Z_{i1},\ldots,Z_{im}\) denote other predictors included in both models, and let a categorical predictor have \(k\) levels. Choose one level as the reference. For \(j=2,\ldots,k\), define \(D_{ij}=1\) when observation \(i\) is in level \(j\), and \(D_{ij}=0\) otherwise. A full model can be written as

Y_i=\beta_0+\sum_{\ell=1}^{m}\gamma_\ell Z_{i\ell} +\sum_{j=2}^{k}\delta_j D_{ij}+\varepsilon_i.

The reduced model keeps the intercept and the \(m\) other predictors, but omits all \(k-1\) indicators. The block test asks

H_0:\delta_2=\delta_3=\cdots=\delta_k=0 \qquad\text{versus}\qquad H_a:\text{at least one }\delta_j\ne 0.

Under this coding, the null says that, conditional on the other predictors, the categorical predictor contributes no differences from the reference level in the model’s mean outcome. The alternative does not specify which levels differ or in what direction; it says only that the entire block is not zero.

With an intercept and \(k\) represented levels, the block contains \(q=k-1\) coefficients. More generally, use \(q\), the number of independent restrictions being tested. This count gives the numerator degrees of freedom for the partial F statistic. It is not the number of levels \(k\).

The Partial F Statistic for a Categorical Block

Let \(RSS_R\) be the residual sum of squares from the reduced model, \(RSS_F\) the residual sum of squares from the full model, and \(df_F\) the residual degrees of freedom in the full model. The partial F statistic is

F=\frac{(RSS_R-RSS_F)/q}{RSS_F/df_F}.

The numerator measures the reduction in residual variation per added coefficient. The denominator is the full model’s residual mean square, which estimates the error variance under the model. When the null hypothesis is true and the regression conditions hold, the statistic has an \(F\) distribution with \(q\) numerator degrees of freedom and \(df_F\) denominator degrees of freedom.

For a full model with an intercept and \(p_F\) predictor coefficients, \(df_F=n-(p_F+1)\). Here \(p_F\) counts all slope coefficients, including the \(k-1\) indicators. The test’s p-value is the probability, assuming \(H_0\) and the model conditions, of obtaining an \(F\) statistic at least as large as the observed one from an \(F(q,df_F)\) distribution.

Conditions for the usual partial F test: The reduced model is obtained by setting the tested block’s coefficients to zero in the full model. Both models use the same outcome and observations. The regression mean is correctly specified for the predictors included; errors are independent, have mean zero and a common variance, and are normally distributed for the exact small-sample \(F\) reference distribution. The design must also allow the tested coefficients to be estimated without exact collinearity.

Worked Calculations

Worked Example: Testing a Three-Level Predictor

Suppose a hypothetical data set contains \(n=30\) observations. A response is modeled using a quantitative baseline score and a three-level program type. The reduced model includes the intercept and baseline score; the full model adds two indicators for program type, with the third level as reference. The models produce \(RSS_R=120\) and \(RSS_F=72\). Test the program-type block at \(\alpha=0.05\).

There are \(k=3\) levels, so the block contains \(q=k-1=2\) indicator coefficients. The full model estimates an intercept, one baseline-score slope, and two indicator coefficients, for four parameters total. Thus \(df_F=30-4=26\).

The test statistic is

F=\frac{(120-72)/2}{72/26} =\frac{24}{2.7692} \approx 8.667.

Under \(H_0\), this is compared with an \(F(2,26)\) distribution. The upper-tail p-value is approximately \(0.00131\), which is below \(0.05\). Reject \(H_0\): these data provide evidence that program type, considered as a two-coefficient block, is associated with the response after accounting for baseline score. This result does not by itself identify which program levels differ.

The usual test is appropriate here only if the two models use these same 30 observations and the stated regression error conditions are reasonable. In particular, the full model must be estimable without exact collinearity among its predictors.

Worked Example: Counting the Restrictions Correctly

In a separate hypothetical analysis, a model includes an intercept, two quantitative predictors, and a four-level categorical predictor. The reduced model omits the categorical predictor and has \(RSS_R=150\); the full model has \(RSS_F=132\). There are \(n=40\) observations. Test whether the four-level predictor adds evidence of association after adjustment for the two quantitative predictors.

With an intercept, a four-level factor uses \(q=4-1=3\) indicators. The full model has five slope coefficients: two quantitative slopes and three indicator coefficients. Including the intercept, it estimates six parameters, so \(df_F=40-6=34\).

Substituting into the partial F formula gives

F=\frac{(150-132)/3}{132/34} =\frac{6}{3.8824} \approx 1.545.

The reference distribution is \(F(3,34)\), and the upper-tail p-value is approximately \(0.22\). At \(\alpha=0.05\), do not reject the joint null hypothesis. This analysis does not provide sufficient evidence that the four-level predictor improves the model after adjustment for the two quantitative predictors. The numerator degrees of freedom are 3 because the test concerns three indicator coefficients, not four category labels.

For this conclusion to use the standard partial F reference distribution, assume independent, mean-zero errors with constant variance and a correctly specified regression mean. Normal errors give the exact reference distribution in this sample; without normality, the usual small-sample calibration is not guaranteed.

Why the Joint Test Has an F Distribution

The statistic compares two estimates of error variation. Under the null, adding the \(q\) indicators does not improve the true mean model. The reduction \(RSS_R-RSS_F\) then reflects fitting \(q\) unnecessary coefficients to random noise. The full-model residual sum of squares also reflects random error, using \(df_F\) residual degrees of freedom.

In the normal linear model, the standard sum-of-squares results under \(H_0\) are

\frac{RSS_R-RSS_F}{\sigma^2}\sim\chi^2_q, \qquad \frac{RSS_F}{\sigma^2}\sim\chi^2_{df_F},

and these two quantities are independent. Dividing each by its degrees of freedom gives two independent estimates of \(\sigma^2\). Their ratio is therefore \(F(q,df_F)\), which is exactly

\frac{(RSS_R-RSS_F)/q}{RSS_F/df_F}.

This explains why the numerator degrees of freedom count the restrictions added by the categorical block and why the denominator degrees of freedom come from the full model.

Reference Categories and Interpretation

The individual indicator coefficients depend on which category is selected as the reference. Changing the reference changes the coefficient descriptions and their numerical values, but it does not change the underlying comparison between the model with the categorical predictor and the model without it. The block test concerns whether the factor contributes any independent differences in the fitted mean, not whether one particular coding produces a coefficient with a particular value.

In a model with other predictors, the test is conditional on those predictors: it asks whether adding the categorical block improves the fit beyond the reduced model containing them. It does not establish that the categorical predictor causes changes in the outcome. A statistically significant result also does not, on its own, indicate whether differences are large enough to matter in practice.

Worked Example: A Two-Level Factor and the Block Test

A hypothetical sample has \(n=24\) observations. The reduced model contains an intercept and one quantitative covariate; the full model adds a three-level categorical predictor. Suppose \(RSS_R=87\) and \(RSS_F=60\). Test the categorical predictor as a block at \(\alpha=0.05\).

The three-level factor requires \(q=2\) indicators. The full model estimates an intercept, the quantitative-covariate slope, and two indicator coefficients: four parameters in total. Therefore \(df_F=24-4=20\).

The partial F statistic is

F=\frac{(87-60)/2}{60/20} =\frac{13.5}{3} =4.500.

For \(F(2,20)\), the upper-tail probability at 4.500 is approximately \(0.0243\). Since \(0.0243<0.05\), reject the joint null. The data provide evidence that the three-level predictor contributes to the model after adjustment for the quantitative covariate. This is a block-level conclusion: it does not say that each of the two indicators is individually significant.

The reduced and full models must be fit to the same 24 observations for this RSS comparison to have the stated interpretation. The standard exact p-value also relies on the normal, independent, constant-variance error assumptions under the null.

Common Mistake: Testing Each Indicator Separately

A frequent error is to treat a multi-level categorical predictor as though it were one coefficient, or to test each indicator separately and conclude that the predictor is irrelevant whenever no individual coefficient has a small p-value. These are not the same question. A factor with \(k\) levels contributes \(k-1\) coefficients under reference coding, and the partial F test evaluates their joint null. Several coefficients can contribute jointly even when the separate tests are not individually compelling.

The correction is to specify the reduced and full models, count the independent coefficients added, and use that count as \(q\). For example, a five-level factor with an intercept adds four indicators, so its joint test has four numerator degrees of freedom. Do not use five merely because there are five category labels, and do not replace the joint test with a collection of one-coefficient tests.

Key takeaway: To test a \(k\)-level categorical predictor with an intercept, compare models that differ by its \(k-1\) indicator coefficients. Use \(q=k-1\) in the numerator of the partial F statistic, and use the full model’s residual degrees of freedom in the denominator.

Check Your Understanding

Answer each question using the joint block-test framework.

A model has an intercept and a five-level categorical predictor. How many indicator coefficients are required under reference-level coding, and what is the numerator degrees of freedom for the joint test?
A full model has \(n=50\), seven estimated parameters including the intercept, and adds three restrictions to a reduced model. What is \(df_F\), and what are the numerator and denominator degrees of freedom of the partial F reference distribution?
In a block test, \(RSS_R=210\), \(RSS_F=180\), \(q=3\), and \(df_F=30\). Calculate the partial F statistic.
Why should a researcher test the indicators jointly rather than infer that a categorical predictor has no contribution just because no single indicator has a small individual p-value?
What does rejecting the block null allow you to conclude about the categorical predictor after adjustment for other predictors, and what does it not establish?