Intercepts, Centering, and Regression Geometry

Why the Intercept Has a Special Geometric Role

In ordinary least squares (OLS), an intercept is more than a coefficient that shifts predictions up or down. Including it places the vector of ones in the regression's column space. The OLS residual vector must be orthogonal to every column in that space, so it must also be orthogonal to the vector of ones. That geometric fact is exactly why the residuals sum to zero.

The same structure connects the intercept to sample means. In a regression with an intercept, the fitted outcome has the same sample mean as the observed outcome. If every regressor is centered by subtracting its sample mean, the slopes and fitted values stay the same, while the intercept becomes the sample mean of the outcome. These are algebraic properties of the OLS fit; they do not require a probabilistic error model.

Definition: For observations $i=1,\ldots,n$, let $y_i$ be the outcome and let $x_{ij}$ be regressor $j$, for $j=1,\ldots,k$. OLS with an intercept chooses coefficients $\hat{\beta}_0,\hat{\beta}_1,\ldots,\hat{\beta}_k$ to minimize $$ \sum_{i=1}^n \left(y_i-\beta_0-\sum_{j=1}^k \beta_j x_{ij}\right)^2. $$ The fitted value is $\hat{y}_i=\hat{\beta}_0+\sum_{j=1}^k\hat{\beta}_j x_{ij}$, and the residual is $e_i=y_i-\hat{y}_i$. In matrix notation, $y$ is the $n$-vector of outcomes, $X$ is the design matrix whose first column is the $n$-vector of ones and whose other columns are the regressors, and $\hat{\beta}$ minimizes $(y-X\beta)'(y-X\beta)$. If the columns of $X$ are linearly independent, this minimizer is unique.

The full-rank condition ensures unique coefficient estimates, but the zero-sum result follows from the OLS normal equations whenever an intercept column is included. A probability model for the outcomes is not needed for that result.

The Zero-Sum Residual Property

At an OLS minimum, the residual vector is orthogonal to each column of the design matrix. In particular, it is orthogonal to the intercept column $1$. Since $e=y-X\hat{\beta}$, the normal equations give $X'e=0$, and the intercept component of this equation is $1'e=0$.

Theorem: In an OLS regression that includes an intercept, the residuals sum to zero: $$ \sum_{i=1}^n e_i=0. $$ Consequently, the sample mean of the fitted values equals the sample mean of the observed outcomes: $$ \frac{1}{n}\sum_{i=1}^n\hat{y}_i=\bar{y}. $$

To see why the normal equations hold, consider changing the coefficient vector by a small amount in any direction $v$. The sum of squared residuals is minimized at $\hat{\beta}$, so its derivative along that direction is zero. Writing $e=y-X\hat{\beta}$, the derivative is $-2v'X'e$. It must equal zero for every $v$, which implies $X'e=0$. Taking the component associated with the column of ones gives $1'e=0$, or the stated sum of residuals.

\frac{1}{n}\sum_{i=1}^n\hat{y}_i =\frac{1}{n}\sum_{i=1}^n(y_i-e_i) =\bar{y}-\frac{1}{n}\sum_{i=1}^n e_i =\bar{y}.

In geometric terms, the fitted vector $X\hat{\beta}$ lies in the column space of $X$, and the residual vector is orthogonal to that space. Because the vector of ones belongs to the column space when there is an intercept, the residuals have zero inner product with it. The inner product is their sum. This does not mean every residual is zero, nor that the regression line or plane passes through every observation.

Worked Example: Residuals and the Intercept in a Simple Regression

Suppose four hypothetical observations have regressor values $x=(1,2,3,4)$ and outcome values $y=(3,5,4,8)$. Fit the OLS regression with an intercept and verify the residual-sum result.

The sample means are $\bar{x}=2.5$ and $\bar{y}=5$. For a simple regression with an intercept, the slope is the sum of cross-products of centered values divided by the sum of squared centered regressor values: $$ \hat{\beta}_1= \frac{\sum_{i=1}^4(x_i-\bar{x})(y_i-\bar{y})} {\sum_{i=1}^4(x_i-\bar{x})^2} =\frac{3+0-0.5+4.5}{2.25+0.25+0.25+2.25} =\frac{7}{5}=1.4. $$

The intercept is $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}=5-(1.4)(2.5)=1.5$. Thus the fitted values are $2.9, 4.3, 5.7,$ and $7.1$, giving residuals $0.1, 0.7, -1.7,$ and $0.9$. Their sum is $0.1+0.7-1.7+0.9=0$. The fitted values average to $(2.9+4.3+5.7+7.1)/4=5$, equal to $\bar{y}$.

How the Intercept Relates to Sample Means

The residual definition gives another route to the intercept formula. Sum $e_i=y_i-\hat{\beta}_0-\sum_{j=1}^k\hat{\beta}_j x_{ij}$ over observations. Because the residuals sum to zero, dividing by $n$ and rearranging yields $$ \hat{\beta}_0=\bar{y}-\sum_{j=1}^k\hat{\beta}_j\bar{x}_j, $$ where $\bar{x}_j=n^{-1}\sum_{i=1}^n x_{ij}$. The formula holds in a multiple regression as well as a simple regression. It describes the estimated intercept in terms of the observed sample means and the estimated slopes.

The intercept is the fitted value when all regressors equal zero. The formula above connects that value to the sample means, but it does not say that the intercept must equal $\bar{y}$ in the original, uncentered regression. In general, it equals $\bar{y}$ only when the slope-weighted regressor means sum to zero. Whether zero is a useful or observed regressor value is a separate question from the algebraic properties of OLS.

Centering Regressors

To center regressor $j$, replace each value by $x_{ij}^{c}=x_{ij}-\bar{x}_j$. The centered regressor has sample mean zero. Rewriting the original fitted equation using $x_{ij}=x_{ij}^{c}+\bar{x}_j$ gives $$ \hat{y}_i= \left(\hat{\beta}_0+\sum_{j=1}^k\hat{\beta}_j\bar{x}_j\right) +\sum_{j=1}^k\hat{\beta}_j x_{ij}^{c}. $$ Thus the same fitted values are represented using the same slopes and a new intercept. The new intercept is $\hat{\beta}_0+\sum_j\hat{\beta}_j\bar{x}_j=\bar{y}$.

Key result: When all regressors are centered and the regression still includes an intercept, the fitted values and slope estimates are unchanged. The intercept in the centered parameterization equals $\bar{y}$. Centering changes the reference point for the intercept, not the outcome predictions.

This result can also be understood by considering the set of possible fitted vectors. Subtracting a constant from a regressor does not change the span of the intercept column and that regressor: the original column is the centered column plus a multiple of the intercept column. Consequently, when all regressors are centered and the intercept remains in the model, the design matrices span the same column space. OLS therefore projects $y$ onto the same space and produces the same fitted vector. With full column rank, the slopes in the two parameterizations are the same, and the intercept adjusts as shown.

Worked Example: Centering Two Regressors

Consider four hypothetical observations with two regressors and outcomes as follows.

Observation	$x_1$	$x_2$	$y$
1	0	0	2
2	1	0	5
3	0	1	4
4	1	1	8

The sample means are $\bar{x}_1=0.5$, $\bar{x}_2=0.5$, and $\bar{y}=4.75$. The balanced design makes the two regressor columns orthogonal after centering. The slope for $x_1$ is the difference between the average outcomes at $x_1=1$ and $x_1=0$: $(5+8)/2-(2+4)/2=6.5-3=3.5$. Similarly, the slope for $x_2$ is $(4+8)/2-(2+5)/2=6-3.5=2.5$.

The original intercept is $$ \hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}_1-\hat{\beta}_2\bar{x}_2 =4.75-(3.5)(0.5)-(2.5)(0.5)=1.75. $$ The original fitted equation is therefore $\hat{y}=1.75+3.5x_1+2.5x_2$. Its fitted values are $1.75, 5.25, 4.25,$ and $7.75$; the residuals are $0.25,-0.25,-0.25,$ and $0.25$, which sum to zero.

After centering both regressors, the equation becomes $\hat{y}=4.75+3.5(x_1-0.5)+2.5(x_2-0.5)$. Its slopes and fitted values are unchanged, while its intercept is $4.75=\bar{y}$. This example illustrates a change of coordinates for the same fitted regression, not a change in the predictions.

What Changes If the Intercept Is Omitted?

Without an intercept, the vector of ones is not necessarily a column of the design matrix. The normal equations still make the residuals orthogonal to the included regressors, but they no longer require residuals to be orthogonal to the vector of ones. Therefore, their sum need not be zero. Centering regressors while omitting the intercept is also not the same operation as centering them while retaining it: it can change the set of possible fitted values and hence the estimated slopes.

Worked Example: Why an Intercept Cannot Be Dropped Casually

Take two hypothetical observations, $x=(1,2)$ and $y=(2,5)$. With an intercept, the fitted line passes through both points: $\bar{x}=1.5$, $\bar{y}=3.5$, $\hat{\beta}_1=(5-2)/(2-1)=3$, and $\hat{\beta}_0=3.5-(3)(1.5)=-1$. The fitted values are $2$ and $5$, so both residuals are zero.

If the intercept is omitted, OLS minimizes squared errors using $\hat{y}_i=\hat{\beta}_1x_i$, and the slope is $$ \hat{\beta}_1=\frac{\sum_i x_i y_i}{\sum_i x_i^2} =\frac{(1)(2)+(2)(5)}{1^2+2^2} =\frac{12}{5}=2.4. $$ The fitted values are $2.4$ and $4.8$; the residuals are $-0.4$ and $0.2$, whose sum is $-0.2$, not zero. The zero-sum property does not apply because the intercept column was omitted.

Now center $x$, giving $x^c=(-0.5,0.5)$, but still fit without an intercept. The slope becomes $$ \frac{\sum_i x_i^c y_i}{\sum_i (x_i^c)^2} =\frac{(-0.5)(2)+(0.5)(5)}{0.25+0.25} =\frac{1.5}{0.5}=3. $$ This is different from the uncentered no-intercept slope $2.4$. The example shows why the invariance result requires keeping the intercept in both regressions.

Common Mistakes and Practical Uses

Claiming residuals sum to zero in every regression. This follows for OLS when an intercept is included. Without one, the residuals need not sum to zero.
Assuming the original intercept equals the outcome mean. In an uncentered multiple regression, the intercept is $\bar{y}-\sum_j\hat{\beta}_j\bar{x}_j$. It equals $\bar{y}$ after all regressors are centered.
Thinking centering changes the fitted relationship. With an intercept retained, centering shifts the intercept and leaves slopes and fitted values unchanged. Dropping the intercept changes the model's fitted-value space.
Treating the zero-sum property as evidence of model quality. The property is an algebraic consequence of OLS with an intercept, not a diagnostic that establishes a correct specification or a causal relationship.

Centering can be useful when zero is an unhelpful reference point: it makes the intercept the fitted outcome at average regressor values. In a model with multiple regressors, this can make the intercept easier to describe without altering the OLS fit. The residual and fitted-mean identities are also useful checks on calculations, provided the regression includes an intercept.

Check Your Understanding

Use the definitions and results above to answer the following questions.

Which normal-equation condition implies that OLS residuals sum to zero when an intercept is included?
In a regression with two uncentered regressors, write the formula for the estimated intercept in terms of $\bar{y}$, the regressor means, and the estimated slopes.
After centering every regressor while retaining the intercept, what happens to the fitted values, slope estimates, and intercept?
Why does the zero-sum residual result not generally hold when the intercept is omitted?
What happens to the fitted-value space if regressors are centered but an intercept is still omitted?

Observation	\(x_1\)	\(x_2\)	\(y\)
1	0	0	2
2	1	0	5
3	0	1	4
4	1	1	8