Least-Squares Fitting With Several Predictors

Why Fit Several Predictors Together?

A multiple regression model describes a response using more than one predictor. Fitting the model means choosing its coefficients from observed data. The goal in least-squares fitting is precise: choose the coefficients that make the squared differences between observed and fitted responses as small as possible in total.

This is a joint choice. A change to one coefficient can affect the residuals for every observation, and the best value for one coefficient can depend on the values chosen for the others. Least squares provides a systematic criterion for making these choices together.

Definition: For observations $i=1,\ldots,n$, let $y_i$ be the observed response and $x_{ij}$ the value of predictor $j$ for observation $i$, where $j=1,\ldots,p$. The multiple linear regression model for the conditional mean is $$ E(Y_i\mid x_{i1},\ldots,x_{ip})=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}. $$ For proposed coefficients $b_0,\ldots,b_p$, the fitted value is $\hat{y}_i=b_0+\sum_{j=1}^{p}b_jx_{ij}$, the residual is $e_i=y_i-\hat{y}_i$, and the sum of squared residuals is $$ SSE(b_0,\ldots,b_p)=\sum_{i=1}^{n}(y_i-b_0-b_1x_{i1}-\cdots-b_px_{ip})^2. $$ The least-squares coefficients are the values that minimize this SSE.

What the Criterion Requires

The SSE criterion is a definition of how to fit the coefficients; it is not a claim that the data must follow a particular probability distribution. Given the observed responses and predictor values, we can calculate and minimize SSE without assuming normally distributed errors. The multiple regression model for a conditional mean supplies a statistical description of the response, but the act of minimizing SSE is an optimization problem.

The fitting calculation does need the response and predictor values for each observation. It also needs enough independent information in the predictor columns to identify a unique set of coefficients. In practical terms, a predictor must not be an exact linear combination of the other predictor columns and the intercept column. If that condition fails, different coefficient choices can give the same fitted values and SSE.

When the columns are not redundant, the least-squares solution can be found by solving the normal equations. These equations say that the residuals from the fit have zero total and zero summed product with the values in each predictor column. The derivation below explains why this is true.

Why the Normal Equations Characterize the Fit

Write $x_{i0}=1$ for the intercept column, and let $b_0,\ldots,b_p$ be the coefficients being fitted. The residual for observation $i$ is $e_i=y_i-\sum_{k=0}^{p}b_kx_{ik}$. Because SSE is a sum of squares, it is a differentiable function of each coefficient. At a minimum, its derivative with respect to each coefficient must be zero.

For coefficient $b_j$, changing that coefficient changes the fitted value for observation $i$ by $x_{ij}$ per unit change in $b_j$, and hence changes the residual by $-x_{ij}$. Differentiating and setting the result to zero gives:

\frac{\partial SSE}{\partial b_j} =\sum_{i=1}^{n}2e_i(-x_{ij}) =-2\sum_{i=1}^{n}x_{ij}e_i=0, \qquad j=0,\ldots,p.

Since $x_{i0}=1$, the equation for $j=0$ says $\sum_i e_i=0$. For every predictor $j=1,\ldots,p$, the corresponding equation says $\sum_i x_{ij}e_i=0$. Substituting the expression for each residual yields the normal equations:

\sum_{i=1}^{n}x_{ij}y_i = \sum_{k=0}^{p}b_k\sum_{i=1}^{n}x_{ij}x_{ik}, \qquad j=0,\ldots,p.

These equations are not merely a convenient calculation. They identify a minimum: SSE is a quadratic function of the coefficients, and its curvature is determined by sums of products of predictor columns. For any change in the coefficient vector, the change in SSE around a point satisfying the normal equations is a sum of squares of linear combinations of those columns, so it cannot be negative. If the columns are linearly independent, that sum is positive for every nonzero coefficient change, giving a unique minimum. If they are redundant, some nonzero changes leave every fitted value unchanged, so the coefficients need not be unique.

Key result: At a least-squares fit, the residuals sum to zero and have zero summed product with each predictor column. If the intercept and predictor columns are linearly independent, the coefficients solving these equations give the unique least-squares fit.

Worked Fits

Worked Example: Fitting Two Centered Predictors

Suppose five hypothetical observations have the following predictor and response values. The predictors are centered: each column sums to zero.

Observation	$x_1$	$x_2$	$y$
1	-2	1	-2
2	-1	-2	12
3	0	2	12
4	1	-2	18
5	2	1	10

Fit the model $\hat{y}=b_0+b_1x_1+b_2x_2$. The relevant predictor sums are $\sum x_1=0$, $\sum x_2=0$, $\sum x_1x_2=0$, $\sum x_1^2=10$, and $\sum x_2^2=14$. The zero cross-product means the two centered predictor columns are orthogonal in this data set. The response mean is $\bar{x}=10$ only if the response were named $x$, so here we instead calculate its mean directly: $\bar{y}=( -2+12+12+18+10)/5=10$. The intercept is therefore $b_0=10$.

With centered, mutually orthogonal predictor columns, the predictor normal equations reduce to $b_1=\sum x_1y/\sum x_1^2$ and $b_2=\sum x_2y/\sum x_2^2$. Substitution gives:

b_1=\frac{(-2)(-2)+(-1)(12)+(0)(12)+(1)(18)+(2)(10)}{10} =\frac{30}{10}=3, $$ $$ b_2=\frac{(1)(-2)+(-2)(12)+(2)(12)+(-2)(18)+(1)(10)}{14} =\frac{-28}{14}=-2.

The fitted equation is $\hat{y}=10+3x_1-2x_2$. Its fitted values, in observation order, are $2,11,6,17,14$, so the residuals are $-4,1,6,1,-4$. Thus:

$$ SSE=(-4)^2+1^2+6^2+1^2+(-4)^2=70. $$

As a check on the calculation, the residuals sum to zero, and their summed products with $x_1$ and $x_2$ are also zero: $(-2)(-4)+(-1)(1)+(0)(6)+(1)(1)+(2)(-4)=0$, and $(1)(-4)+(-2)(1)+(2)(6)+(-2)(1)+(1)(-4)=0$. The fitted coefficients therefore satisfy the normal equations.

Worked Example: Solving Normal Equations With Related Predictors

Now consider four hypothetical observations with two predictors that are not orthogonal. Fit the same model form, $\hat{y}=b_0+b_1x_1+b_2x_2$.

Observation	$x_1$	$x_2$	$y$
1	0	0	2
2	1	1	1
3	2	1	3
4	3	2	6

The needed sums are $n=4$, $\sum x_1=6$, $\sum x_2=4$, $\sum y=12$, $\sum x_1^2=14$, $\sum x_2^2=6$, $\sum x_1x_2=9$, $\sum x_1y=25$, and $\sum x_2y=16$. Applying the normal equations gives:

$$ 4b_0+6b_1+4b_2=12, $$ $$ 6b_0+14b_1+9b_2=25, $$ $$ 4b_0+9b_1+6b_2=16. $$

Solving this system gives $b_0=1$, $b_1=2$, and $b_2=-1$. Substitution verifies all three equations: $4+12-4=12$, $6+28-9=25$, and $4+18-6=16$. The resulting fitted values are $1,2,4,5$; subtracting them from the observed responses gives residuals $1,-1,-1,1$. Therefore $SSE=1^2+(-1)^2+(-1)^2+1^2=4$. The residuals sum to zero, and their products with each predictor column sum to zero: $0(1)+1(-1)+2(-1)+3(1)=0$, and $0(1)+1(-1)+1(-1)+2(1)=0$.

Worked Example: Detecting a Nonunique Coefficient Fit

Suppose three hypothetical observations have $x_1=0,1,2$, $x_2=0,2,4$, and $y=1,3,5$, respectively. Here $x_2=2x_1$ for every observation, so the predictor columns are redundant. The fitted value for any row is:

\hat{y}=b_0+b_1x_1+b_2x_2 =b_0+(b_1+2b_2)x_1.

The data are fitted exactly if $b_0=1$ and $b_1+2b_2=2$. For example, $(b_0,b_1,b_2)=(1,2,0)$ gives fitted values $1,3,5$ and $SSE=0$. So does $(1,0,1)$, because $0+2(1)=2$. Both are least-squares solutions, and infinitely many others satisfy the same equations. The fitted values are uniquely determined here, but the separate coefficients of the redundant predictors are not.

Why the Joint Fit Matters

When predictors are related, fitting each predictor separately and then combining the results generally does not minimize the SSE of the multiple regression model. The normal equations include cross-products such as $\sum_i x_{i1}x_{i2}$, which account for how predictor columns overlap. Ignoring those terms can produce a coefficient choice that does not satisfy the joint minimization criterion.

A common mistake is to say, “I found the best coefficient for each predictor one at a time, so those coefficients must be the best multiple regression fit.” That reasoning treats each predictor as if the other predictors were absent. The correction is to minimize the full SSE over all coefficients together, either by solving the normal equations or by using a numerical least-squares algorithm. When columns are redundant, report that the coefficients are not uniquely identified rather than treating one arbitrary solution as the only fit.

Least squares matters because it gives a clear and reproducible rule for selecting the fitted model. The residuals also provide a direct check on a computed solution: with an intercept, they should sum to zero, and their summed products with each included predictor column should be zero, aside from small rounding error. These checks verify the fitting equations; they do not by themselves establish that the model is appropriate for the data.

Check Your Understanding

Try these questions before reviewing the worked examples.

For a model with an intercept and two predictors, write the fitted value and residual for observation $i$, then define the SSE.
What do the normal equations require about the sum of residuals and the summed product of residuals with each predictor column?
Why does minimizing SSE with several predictors require choosing the coefficients jointly?
If one predictor column is exactly twice another, what problem can arise for the individual fitted coefficients, even when fitted values are available?
Does calculating a least-squares fit require normally distributed errors? Distinguish the optimization criterion from assumptions used for statistical inference.

Observation	\(x_1\)	\(x_2\)	\(y\)
1	0	0	2
2	1	1	1
3	2	1	3
4	3	2	6