An Integrated Multiple Regression Estimation Case

Why an Integrated Fit Matters

A regression model with a categorical predictor and a continuous predictor can allow both the intercept and the slope to differ between categories. The interaction term is what permits the group-specific slopes to differ. Estimating such a model requires keeping the coding, coefficient interpretation, and arithmetic aligned: a correctly calculated coefficient for the wrong design-matrix column still gives the wrong model.

This tutorial works through a complete least-squares fit, including the design matrix, estimated coefficients, fitted values, residuals, and residual sum of squares (RSS). The examples use hypothetical insurance and service settings; their numbers are invented for instruction.

Model: Let $D_i$ be an indicator equal to 0 for the reference category and 1 for the other category, and let $x_i$ be a continuous predictor. The interaction model is $$ Y_i=\beta_0+\beta_1D_i+\beta_2x_i+\beta_3D_ix_i+\varepsilon_i. $$ The coefficient vector is estimated by least squares when the design matrix has full column rank. For interpreting the fitted regression as a conditional-mean model, assume $E[\varepsilon_i\mid D_i,x_i]=0$. Normal errors are not needed to calculate the least-squares estimates or fitted values.

Read the Model by Category

Set $D_i=0$ in the model for the reference category. Its fitted line is $\hat{Y}=\hat{\beta}_0+\hat{\beta}_2x$, so its intercept is $\hat{\beta}_0$ and its slope is $\hat{\beta}_2$. For the category with $D_i=1$, the fitted line is $\hat{Y}=(\hat{\beta}_0+\hat{\beta}_1)+(\hat{\beta}_2+\hat{\beta}_3)x$. Thus, $\hat{\beta}_1$ is the difference in intercepts and $\hat{\beta}_3$ is the difference in slopes, relative to the reference category.

The product $D_ix_i$ is essential: it is zero in the reference category and equals $x_i$ in the other category. Without that product, this model would allow different intercepts but would impose one common slope. The fitted value for observation $i$ is $\hat{Y}_i$, and its residual is $e_i=Y_i-\hat{Y}_i$. The RSS is the sum of the squared residuals.

Least-squares result: Write the model for all observations as $\mathbf{Y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}$, where each row of $\mathbf{X}$ is $(1,D_i,x_i,D_ix_i)$. If the columns of $\mathbf{X}$ are linearly independent, the least-squares estimate is $$ \hat{\boldsymbol{\beta}}=(\mathbf{X}^{\mathsf T}\mathbf{X})^{-1}\mathbf{X}^{\mathsf T}\mathbf{Y}. $$ The residual sum of squares is $RSS=\sum_i(Y_i-\hat{Y}_i)^2$.

Why the Normal Equations Give the Fit

Least squares chooses coefficients to minimize the total squared difference between observed and fitted responses. In matrix form, the quantity being minimized is $(\mathbf{Y}-\mathbf{X}\boldsymbol{\beta})^{\mathsf T}(\mathbf{Y}-\mathbf{X}\boldsymbol{\beta})$. Differentiating with respect to the coefficient vector and setting the derivative equal to zero gives the normal equations:

-2\mathbf{X}^{\mathsf T}(\mathbf{Y}-\mathbf{X}\hat{\boldsymbol{\beta}})=\mathbf{0} \quad\Longrightarrow\quad \mathbf{X}^{\mathsf T}\mathbf{X}\hat{\boldsymbol{\beta}}=\mathbf{X}^{\mathsf T}\mathbf{Y}.

When $\mathbf{X}$ has full column rank, $\mathbf{X}^{\mathsf T}\mathbf{X}$ is invertible, giving the stated solution. If the columns are not linearly independent, the inverse does not exist and the coefficient vector is not uniquely determined; that is why checking the model columns matters. In this tutorial, the example design matrices have full column rank.

Worked Example: Estimate All Four Coefficients

A hypothetical insurer records a response score $Y$ for six policies. Category A is the reference category ($D=0$); category B has $D=1$. The continuous predictor $x$ is a centered service-change measure. Fit the interaction model by least squares, then compute the fitted values and RSS.

Category	$D$	$x$	$Y$
A	0	-1	9
A	0	0	8
A	0	1	13
B	1	-1	14
B	1	0	9
B	1	1	16

Step 1: Construct the columns. In the order $(1,D,x,Dx)$, the rows are $(1,0,-1,0)$, $(1,0,0,0)$, $(1,0,1,0)$, $(1,1,-1,-1)$, $(1,1,0,0)$, and $(1,1,1,1)$. In particular, the interaction column is the product $D_ix_i$, not a second copy of $x_i$.

Step 2: Form the normal equations. Adding the cross-products of the design columns gives

\mathbf{X}^{\mathsf T}\mathbf{X}= \begin{pmatrix} 6&3&0&0\\ 3&3&0&0\\ 0&0&4&2\\ 0&0&2&2 \end{pmatrix}, \qquad \mathbf{X}^{\mathsf T}\mathbf{Y}= \begin{pmatrix}69\\39\\6\\2\end{pmatrix}.

For example, the first entry of $\mathbf{X}^{\mathsf T}\mathbf{Y}$ is $9+8+13+14+9+16=69$; its second entry is the category-B total $14+9+16=39$. The normal equations are therefore $6\hat{\beta}_0+3\hat{\beta}_1=69$, $3\hat{\beta}_0+3\hat{\beta}_1=39$, $4\hat{\beta}_2+2\hat{\beta}_3=6$, and $2\hat{\beta}_2+2\hat{\beta}_3=2$.

Step 3: Solve the equations. Subtracting the second equation from the first gives $3\hat{\beta}_0=30$, so $\hat{\beta}_0=10$. Substitution into the second gives $30+3\hat{\beta}_1=39$, hence $\hat{\beta}_1=3$. Subtracting the fourth equation from the third gives $2\hat{\beta}_2=4$, so $\hat{\beta}_2=2$. Substitution into the fourth gives $4+2\hat{\beta}_3=2$, hence $\hat{\beta}_3=-1$.

The fitted model is $\hat{Y}=10+3D+2x-Dx$. Category A therefore has intercept 10 and slope 2. Category B has intercept $10+3=13$ and slope $2-1=1$.

Step 4: Check fitted values, residuals, and RSS. For A, the fitted values at $x=-1,0,1$ are $8,10,12$; the residuals are $9-8=1$, $8-10=-2$, and $13-12=1$. For B, the fitted values are $13-1=12$, $13$, and $13+1=14$; the residuals are $14-12=2$, $9-13=-4$, and $16-14=2$. Thus

$$ RSS=1^2+(-2)^2+1^2+2^2+(-4)^2+2^2=1+4+1+4+16+4=30. $$

As a check, the residuals sum to zero within each category: $1-2+1=0$ and $2-4+2=0$. The estimates also reproduce the group-specific lines calculated above.

Use the Coefficients for a Prediction

Once the coefficients are estimated, prediction requires substituting the indicator and continuous-predictor values into the complete model. It is safer to do this explicitly than to apply a group slope while forgetting the group-specific intercept, or to add the interaction coefficient without multiplying it by both $D$ and $x$.

Worked Example: Estimate from Group Summaries and Predict

A hypothetical claims-support team studies response time $Y$ by support channel, with channel A coded $D=0$ and channel B coded $D=1$. The continuous predictor $x$ is workload in stated units. Each category has observations at $x=0,2,4$. The observed responses are A: $21,24,33$, and B: $24,30,30$. Estimate the coefficients, calculate the prediction for channel B at $x=3$, and find the RSS.

Step 1: Fit the channel-A line. Its mean response is $(21+24+33)/3=78/3=26$, and its mean $x$ is $(0+2+4)/3=2$. The centered cross-product sum is $(0-2)(21-26)+(2-2)(24-26)+(4-2)(33-26)=10+0+14=24$. The centered $x$ sum of squares is $(-2)^2+0^2+2^2=8$. Thus the slope is $24/8=3$, and the intercept is $26-3(2)=20$.

Step 2: Fit the channel-B line. Its mean response is $(24+30+30)/3=84/3=28$, while the mean $x$ remains 2. The centered cross-product sum is $(-2)(24-28)+0(30-28)+2(30-28)=8+0+4=12$, so the slope is $12/8=1.5$. The intercept is $28-1.5(2)=25$.

The model coefficients follow from the two lines: $\hat{\beta}_0=20$, $\hat{\beta}_1=25-20=5$, $\hat{\beta}_2=3$, and $\hat{\beta}_3=1.5-3=-1.5$. Hence $\hat{Y}=20+5D+3x-1.5Dx$.

Step 3: Predict and check RSS. For channel B at $x=3$, the prediction is $20+5+3(3)-1.5(1)(3)=25+9-4.5=29.5$. For the observed A values, fitted values at $x=0,2,4$ are $20,26,32$, giving residuals $1,-2,1$. For B, fitted values are $25,28,31$, giving residuals $-1,2,-1$. Therefore $RSS=1^2+(-2)^2+1^2+(-1)^2+2^2+(-1)^2=12$. The residuals sum to zero in each category, as expected for these fitted lines with intercepts.

Check a Fit When the Group Sizes Differ

The same model can be fit when the categories have different numbers of observations. A useful way to check the arithmetic is to estimate a separate line within each category: the reference-category line gives $\hat{\beta}_0$ and $\hat{\beta}_2$, while the other line gives the differences that determine $\hat{\beta}_1$ and $\hat{\beta}_3$. This equivalence follows directly from the model’s two group-specific lines, and it applies when each group-specific line is identifiable.

Worked Example: Verify a Fit with Unequal Group Sizes

A hypothetical service unit records an outcome for three observations in category A ($D=0$) and two in category B ($D=1$). The data are A: $(x,Y)=(0,6),(1,5),(2,10)$, and B: $(0,8),(1,12)$. Fit the interaction model and verify the fitted values and RSS.

Step 1: Fit category A. The mean of $x$ is $(0+1+2)/3=1$, and the mean response is $(6+5+10)/3=7$. The centered cross-product sum is $(-1)(6-7)+0(5-7)+1(10-7)=1+0+3=4$. The centered $x$ sum of squares is $1+0+1=2$, so the slope is $4/2=2$, and the intercept is $7-2(1)=5$.

Step 2: Fit category B. With two points, its line passes through $(0,8)$ and $(1,12)$. Its slope is $(12-8)/(1-0)=4$, and its intercept is $8$.

The model coefficients are therefore $\hat{\beta}_0=5$, $\hat{\beta}_1=8-5=3$, $\hat{\beta}_2=2$, and $\hat{\beta}_3=4-2=2$. The fitted equation is $\hat{Y}=5+3D+2x+2Dx$.

Step 3: Calculate residuals and RSS. In category A, the fitted values at $x=0,1,2$ are $5,7,9$, so the residuals are $6-5=1$, $5-7=-2$, and $10-9=1$. In category B, the fitted values at $x=0,1$ are $5+3=8$ and $5+3+2+2=12$, so both residuals are zero. Thus $RSS=1^2+(-2)^2+1^2+0^2+0^2=6$. The reference-category residuals sum to $1-2+1=0$, and the B residuals also sum to zero.

The unequal group sizes do not change the coefficient interpretation. This check works here because both groups contain enough distinct $x$-values to identify their lines; if a group had no variation in $x$, its within-group slope could not be estimated from that group alone.

Common Mistakes and a Reliable Check

A frequent interaction error is to report $\hat{\beta}_3$ as the slope for category $D=1$. It is only the change from the reference slope; the category-1 slope is $\hat{\beta}_2+\hat{\beta}_3$. For the first example, the wrong calculation would call the B slope $-1$. The correction is $2+(-1)=1$, which agrees with the fitted values $12,13,14$ as $x$ increases from $-1$ to 1.

Other useful checks are to calculate $D_ix_i$ row by row, verify that the fitted value uses all four terms, and square each residual before adding. The residuals themselves can sum to zero while the RSS is positive; for example, the first example’s residuals sum to zero, but their squared sum is 30. Do not confuse the sum of residuals with the residual sum of squares.

Key takeaway: Estimate and interpret the model through its two group-specific lines, but calculate fitted values from the full coded equation. Then compute each residual as observed minus fitted and square before summing to obtain RSS.

Check Your Understanding

For each problem, identify the requested coefficients or quantities using the stated indicator convention.

A fitted interaction model has reference-category intercept 12 and slope 2. The other category has intercept 15 and slope 0.5. What are $\hat{\beta}_0,\hat{\beta}_1,\hat{\beta}_2,\hat{\beta}_3$? (Answer: $12,3,2,-1.5$, respectively.)
Using the coefficients in Question 1, calculate the fitted response for the non-reference category at $x=4$. (Answer: $17$.)
For the reference category alone, observations at $x=0,1,2$ have responses $4,7,8$. Fit its least-squares line and calculate its RSS. (Answer: intercept $13/3$, slope $2$, RSS $2/3$.)
An interaction fit has group-specific lines $\hat{Y}_0=6+1.5x$ and $\hat{Y}_1=9+0.5x$, where group 0 is the reference. Give the four coefficients in the model $\hat{Y}=\hat{\beta}_0+\hat{\beta}_1D+\hat{\beta}_2x+\hat{\beta}_3Dx$. (Answer: $\hat{\beta}_0=6,\hat{\beta}_1=3,\hat{\beta}_2=1.5,\hat{\beta}_3=-1$.)
A fitted model produces residuals $1,-1,0$ in category 0 and $2,0,-2$ in category 1. Find the residual sum and the RSS. (Answer: residual sum $=0$; RSS $=10$.)

Category	\(D\)	\(x\)	\(Y\)
A	0	-1	9
A	0	0	8
A	0	1	13
B	1	-1	14
B	1	0	9
B	1	1	16

What You'll Learn