Predicting Across Categories with Indicator Models

Predicting for Every Category

A categorical predictor can describe features such as plan design, region, or service channel. A fitted regression model can then answer questions such as: what response does it predict for each plan design if the other predictors are held at the same values? The important step is to calculate each category’s fitted response from the model, rather than interpreting a coefficient as if it were itself the full prediction.

This tutorial considers indicator-coded models without interactions. In such a model, category differences are constant across the values of any included numeric predictors. The resulting predictions are useful for comparing categories on a like-for-like basis, but they are not necessarily the observed average responses in those categories.

Definition: Suppose a categorical predictor $G$ has $K$ levels, labelled $0,1,\ldots,K-1$, and level $0$ is the reference category. Define the indicator $I(G=j)$ to equal 1 when an observation is in level $j$, and 0 otherwise. A reference-coded additive regression model is $$ Y_i=\beta_0+\sum_{j=1}^{K-1}\beta_j I(G_i=j)+\boldsymbol{x}_i^\mathsf{T}\boldsymbol{\gamma}+\varepsilon_i. $$ Here $\boldsymbol{x}_i$ contains the other predictors, and the model has no products between the category indicators and those predictors. The fitted response for a category is obtained by substituting that category’s indicator values and the specified values of $\boldsymbol{x}$ into the fitted equation.

For the model to define unique coefficients under this coding, the design matrix must have full column rank. In particular, the reference level and each included non-reference level must be represented in the fitting data, and the predictors must not create exact linear dependencies. For the usual interpretation of the equation as a conditional mean model, assume $E[\varepsilon_i\mid G_i,\boldsymbol{x}_i]=0$. These conditions do not require normally distributed errors just to calculate fitted responses. A fitted response estimates a conditional mean; it is not an individual future outcome or a prediction interval.

Reading Predictions from Reference Coding

Let $\widehat{\beta}_0,\widehat{\beta}_1,\ldots,\widehat{\beta}_{K-1}$ and $\widehat{\boldsymbol{\gamma}}$ denote the fitted coefficients. For a specified covariate vector $\boldsymbol{x}$, the reference-category prediction sets every indicator to zero. A prediction for a non-reference category adds that category’s coefficient:

\widehat{Y}(G=0,\boldsymbol{x}) =\widehat{\beta}_0+\boldsymbol{x}^{\mathsf T}\widehat{\boldsymbol{\gamma}}, \qquad \widehat{Y}(G=j,\boldsymbol{x}) =\widehat{\beta}_0+\widehat{\beta}_j+\boldsymbol{x}^{\mathsf T}\widehat{\boldsymbol{\gamma}}.

Thus $\widehat{\beta}_j$ is the fitted difference between category $j$ and the reference category when the other predictors are held fixed. For two non-reference categories $j$ and $l$, subtract their predictions: the common intercept and covariate contribution cancel, leaving $\widehat{\beta}_j-\widehat{\beta}_l$. This comparison is conditional on the specified values of the other predictors.

The absence of interactions matters. In this model, the difference between two categories is the same at every value of $\boldsymbol{x}$. If the real relationship allows category-specific slopes, this additive model does not represent that feature; an interaction model is needed.

Worked Example: Comparing Three Designs at a Fixed Tenure

A hypothetical insurer models annual service cost $Y$, measured in thousands of dollars, using plan design and customer tenure $t$, measured in years. Design A is the reference category. The fitted model is $\widehat{Y}=8+1.5I(G=B)-0.8I(G=C)-0.4t$. Calculate the fitted cost for each design at five years of tenure, identify the highest prediction, and compare designs B and C.

Method. Substitute $t=5$, then set the indicators according to each design. The model is additive, so the same tenure contribution applies to all three categories.

Design A. Both indicators are zero: $\widehat{Y}_A=8+1.5(0)-0.8(0)-0.4(5)=8-2=6.0$ thousand dollars.

Design B. $I(G=B)=1$ and $I(G=C)=0$: $\widehat{Y}_B=8+1.5(1)-0.8(0)-0.4(5)=8+1.5-2=7.5$ thousand dollars.

Design C. $I(G=B)=0$ and $I(G=C)=1$: $\widehat{Y}_C=8+1.5(0)-0.8(1)-0.4(5)=8-0.8-2=5.2$ thousand dollars.

Design B has the highest fitted cost. The fitted difference between B and C is $7.5-5.2=2.3$ thousand dollars. It also follows directly from their coefficients: $1.5-(-0.8)=2.3$. The predictions check because their common contribution, $8-0.4(5)=6$, is shared: adding $1.5$ gives $7.5$, and adding $-0.8$ gives $5.2$.

Common mistake avoided: reporting $1.5$ as the prediction for design B. It is only B’s fitted difference from A at the same tenure; the full prediction also includes the intercept and tenure contribution.

Why Categorical-Only Predictions Are Group Means

When a model contains only a categorical predictor, the fitted response for each category is its sample mean. This is a special property of the categorical-only model, not a general rule that holds for every model containing a categorical predictor.

Justification: For category $g$, let its observed responses be $y_{g1},\ldots,y_{gn_g}$, and let the model assign that category a fitted value $\mu_g$. Its contribution to the residual sum of squares is $\sum_{r=1}^{n_g}(y_{gr}-\mu_g)^2$. Differentiating with respect to $\mu_g$ and setting the derivative to zero gives $$ -2\sum_{r=1}^{n_g}(y_{gr}-\mu_g)=0 \quad\Longrightarrow\quad \widehat{\mu}_g=\frac{1}{n_g}\sum_{r=1}^{n_g}y_{gr}. $$ Each category has its own fitted value, so minimizing the total residual sum of squares minimizes each category’s contribution at its sample mean.

With additional predictors, the fitted category comparison is adjusted for those predictors in the specified model. The raw category means may differ because the categories contain different mixes of the numeric predictors. Substituting the same covariate values for each category makes the model-based comparison explicit.

Worked Example: Fitting Category Means from Observed Data

A hypothetical study records processing times, in minutes, for three routing methods. Method A has times 12, 16, and 14; method B has times 19, 21, and 20; method C has times 9 and 15. Fit the categorical-only reference-coded model using A as the reference, calculate the fitted response for each method, and compare B with C.

Method. In a categorical-only least-squares model, calculate each method’s sample mean. The intercept is the mean for A, and each indicator coefficient is that category’s mean minus A’s mean.

For A, the sum is $12+16+14=42$, with $n_A=3$, so $\widehat{\mu}_A=42/3=14$.

For B, the sum is $19+21+20=60$, with $n_B=3$, so $\widehat{\mu}_B=60/3=20$.

For C, the sum is $9+15=24$, with $n_C=2$, so $\widehat{\mu}_C=24/2=12$.

The fitted model is $\widehat{Y}=14+6I(G=B)-2I(G=C)$: the intercept is 14, the B coefficient is $20-14=6$, and the C coefficient is $12-14=-2$. Substitution gives fitted responses of 14 for A, 20 for B, and 12 for C. B exceeds C by $20-12=8$ minutes; equivalently, the difference between their indicator coefficients is $6-(-2)=8$.

As a check, the fitted value within each method equals that method’s observed mean. The residuals sum to zero within every method: for A, $(12-14)+(16-14)+(14-14)=0$; for B, $(19-20)+(21-20)+(20-20)=0$; for C, $(9-12)+(15-12)=0$.

Common mistake avoided: treating the C coefficient, $-2$, as C’s fitted time. It is C’s difference from A. The full prediction for C is $14-2=12$.

Using a Different Contrast Coding

Reference coding is not the only way to represent a categorical predictor. Under contrast coding, the model can use other numerical columns to represent category membership. The calculation rule remains the same: insert the code assigned to the category into the fitted equation. The coefficient values may have a different interpretation, but the fitted response for a category is determined by the complete linear predictor.

Worked Example: Predicting Under Sum-to-Zero Coding

A model for hypothetical claim-handling time has three categories, A, B, and C, and uses the coding A: $(1,0)$, B: $(0,1)$, C: $(-1,-1)$. There are no other predictors, and the fitted equation is $\widehat{Y}=15+3c_1-2c_2$. Calculate all three fitted responses, rank the categories, and express the results using A as a reference.

Method. Substitute each category’s pair of codes. Then, for reference coding with A as the reference, use A’s fitted value as the intercept and subtract it from each other category’s fitted value to obtain the corresponding indicator coefficient.

For A, $(c_1,c_2)=(1,0)$, so $\widehat{Y}_A=15+3(1)-2(0)=18$.

For B, $(c_1,c_2)=(0,1)$, so $\widehat{Y}_B=15+3(0)-2(1)=13$.

For C, $(c_1,c_2)=(-1,-1)$, so $\widehat{Y}_C=15+3(-1)-2(-1)=15-3+2=14$.

The ranking is A (18), C (14), then B (13). The reference-coded form is $\widehat{Y}=18-5I(G=B)-4I(G=C)$, because $13-18=-5$ and $14-18=-4$. Checking that form gives 18 for A, $18-5=13$ for B, and $18-4=14$ for C, matching the original predictions.

Common mistake avoided: reading the contrast coefficients 3 and $-2$ as category-specific differences from A. They are coefficients on the chosen contrast columns, not reference-coded category differences.

Comparison Traps and Practical Checks

A reliable way to compare categories is to write down the entire fitted expression for each one, including the common covariate terms. This makes it easier to catch omitted contributions and coding errors.

Set every indicator deliberately. For a single observation in a reference-coded model, the reference category has all its non-reference indicators equal to zero; a non-reference category has its own indicator equal to one and the others equal to zero.
Hold other predictors fixed. A category comparison is meaningful only after stating the values at which numeric predictors are evaluated. Comparing predictions at different covariate values mixes category and covariate differences.
Separate fitted differences from fitted responses. In reference coding, a category coefficient is a difference from the reference category, not the category’s full prediction.
Do not substitute raw means for adjusted predictions. Category means equal fitted values in a categorical-only least-squares model. In an additive model with other predictors, predictions require evaluating the fitted equation at specified covariate values.
Check whether the model has interactions. Without interactions, category differences do not vary with the numeric predictors. Do not apply that conclusion to a model that includes category-by-predictor products.

For a fitted comparison, a useful final check is to calculate the predictions separately and then subtract them. The difference should also agree with the algebraic coefficient difference when the model is reference-coded and additive. Agreement between these calculations helps detect a missing indicator, a sign error, or a covariate value applied inconsistently.

Key takeaway: A category’s fitted response is the model’s full linear predictor evaluated using that category’s code and the stated values of all other predictors. Under reference coding, non-reference coefficients are differences from the reference category; under other coding schemes, substitute the assigned codes before interpreting the resulting fitted responses.

Check Your Understanding

For each question, calculate the requested fitted responses from the stated model.

A reference-coded model has categories A, B, C, and D, with fitted intercept 42 and indicator coefficients $\widehat{\beta}_B=-3$, $\widehat{\beta}_C=5$, and $\widehat{\beta}_D=1$. Calculate all category predictions and identify the largest. (Answer: A = 42, B = 39, C = 47, D = 43; C is largest.)
A hypothetical model has reference category R and fitted equation $\widehat{Y}=2+0.7I(G=D)-0.4I(G=E)+0.3z$. At $z=10$, calculate all three category predictions and the fitted difference between D and E. (Answer: R = 5.0, D = 5.7, E = 4.6; D exceeds E by 1.1.)
Under the coding A: $(1,0)$, B: $(0,1)$, and C: $(-1,-1)$, a fitted model is $\widehat{Y}=30+4c_1-c_2$. Calculate each category prediction and verify their unweighted average. (Answer: A = 34, B = 29, C = 27; average = 30.)
A categorical-only data set has observations A: 4, 8; B: 10, 14; and C: 3, 9. With A as reference, calculate the fitted category means and the two indicator coefficients. (Answer: fitted means A = 6, B = 12, C = 6; coefficients for B and C are 6 and 0.)
A reference-coded model has fitted equation $\widehat{Y}=100-12I(G=B)+8I(G=C)+2x$. At $x=6$, calculate all three predictions and the difference between C and B. (Answer: A = 112, B = 100, C = 120; C exceeds B by 20.)

What You'll Learn