Solving for the Least-Squares Slope and Intercept

From the Normal Equations to the Fitted Line

For paired observations, the least-squares line is the line that minimizes the sum of squared vertical differences between the observed responses and the fitted responses. Finding this line requires two coefficients: the slope, which determines how the fitted response changes with the predictor, and the intercept, which sets the fitted response when the predictor is zero.

The earlier material on deriving the least-squares normal equations establishes the equations that a minimizing line must satisfy. Here we solve those equations and connect their solution to means, centered sums, sample covariance, and sample variance. The focus is calculation and algebra; a probability model for the observations is not required to compute the least-squares coefficients.

Definition: Given observed pairs $(x_1,y_1),\ldots,(x_n,y_n)$, write a candidate line as $\hat{y}=a+bx$. The least-squares intercept $\hat{\beta}_0$ and slope $\hat{\beta}_1$ are the values of $a$ and $b$ that minimize $$ Q(a,b)=\sum_{i=1}^{n}(y_i-a-bx_i)^2. $$ The fitted line is $\hat{y}=\hat{\beta}_0+\hat{\beta}_1x$.

For a unique slope and intercept, the observed predictor values must not all be equal. If every $x_i$ has the same value, the data do not distinguish among slopes: multiple lines with different slopes can give the same fitted values at that one predictor value. No distributional assumption, such as normality, is needed for the algebraic least-squares fit.

Solving the Equations

The normal equations for minimizing $Q(a,b)$ are

\sum_{i=1}^{n} y_i=n\hat{\beta}_0+\hat{\beta}_1\sum_{i=1}^{n}x_i, \qquad \sum_{i=1}^{n}x_i y_i=\hat{\beta}_0\sum_{i=1}^{n}x_i+\hat{\beta}_1\sum_{i=1}^{n}x_i^2.

Define the sample means $\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i$ and $\bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i$. Also define the centered predictor sum of squares $S_{xx}$ and centered cross-product sum $S_{xy}$ by

S_{xx}=\sum_{i=1}^{n}(x_i-\bar{x})^2, \qquad S_{xy}=\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}).

The first normal equation immediately gives $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$. Substituting this expression into the second equation gives the slope in terms of centered quantities. Thus the practical solution is:

Theorem: If the predictor values are not all equal, the unique least-squares coefficients are $$ \hat{\beta}_1=\frac{S_{xy}}{S_{xx}}, \qquad \hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}. $$

Calculate the means.
Find $\bar{x}$ and $\bar{y}$ from the observed predictor and response values.

Calculate the centered sums.
Compute $S_{xx}$ and $S_{xy}$, keeping the paired $x_i$ and $y_i$ values together.

Find the slope, then the intercept.
Use $\hat{\beta}_1=S_{xy}/S_{xx}$, followed by $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$.

Check the result.
Substitute the coefficients into the normal equations or verify that the fitted value at $\bar{x}$ equals $\bar{y}$.

Why the Centered-Sum Formula Holds

The centered-sum expression is not a separate fitting rule; it is the result of solving the normal equations. From the first equation, $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$. Put this into the second equation:

\sum_{i=1}^{n}x_i y_i =(\bar{y}-\hat{\beta}_1\bar{x})\sum_{i=1}^{n}x_i +\hat{\beta}_1\sum_{i=1}^{n}x_i^2.

Since $\sum x_i=n\bar{x}$, rearranging yields

\hat{\beta}_1 = \frac{\sum_{i=1}^{n}x_i y_i-n\bar{x}\bar{y}} {\sum_{i=1}^{n}x_i^2-n\bar{x}^2}.

Expanding the definitions of the centered sums gives $S_{xy}=\sum x_i y_i-n\bar{x}\bar{y}$ and $S_{xx}=\sum x_i^2-n\bar{x}^2$, so this is exactly $\hat{\beta}_1=S_{xy}/S_{xx}$. The denominator is positive precisely when the predictor values are not all equal, which explains the condition for a unique slope.

There is also a useful covariance interpretation. If $s_{xy}=S_{xy}/(n-1)$ is the sample covariance and $s_x^2=S_{xx}/(n-1)$ is the sample variance of the predictor, then

\hat{\beta}_1=\frac{s_{xy}}{s_x^2}.

The factors $n-1$ cancel because both quantities use the same divisor. The same ratio results if both are defined using divisor $n$. A covariance calculated with one divisor must not be divided by a sum of squares using a different scale without accounting for that difference.

Worked Calculations

Worked Example: Solving the Normal Equations from Four Observations

In a hypothetical operations study, a supervisor records the number of hours of practice $x$ and a processing score $y$ for four trainees. The observed pairs are $(1,4),(2,5),(3,5),(4,10)$. Find the least-squares line by solving the normal equations, and check the result.

Method. First calculate the totals needed in the normal equations:

n=4,\quad \sum x_i=1+2+3+4=10,\quad \sum y_i=4+5+5+10=24, $$ $$ \sum x_i^2=1+4+9+16=30,\qquad \sum x_i y_i=4+10+15+40=69.

The normal equations are $4\hat{\beta}_0+10\hat{\beta}_1=24$ and $10\hat{\beta}_0+30\hat{\beta}_1=69$. Multiply the first equation by $2.5$, giving $10\hat{\beta}_0+25\hat{\beta}_1=60$. Subtract this from the second equation:

5\hat{\beta}_1=9,\qquad \hat{\beta}_1=1.8.

Substitute into the first equation: $4\hat{\beta}_0+10(1.8)=24$, so $4\hat{\beta}_0=6$ and $\hat{\beta}_0=1.5$. The fitted line is $\hat{y}=1.5+1.8x$.

Check. In the first normal equation, $4(1.5)+10(1.8)=6+18=24=\sum y_i$. In the second, $10(1.5)+30(1.8)=15+54=69=\sum x_i y_i$. Both equations hold. The fitted value at the mean predictor is also $1.5+1.8(2.5)=6$, which equals $\bar{y}=24/4=6$.

Common mistake avoided. The coefficient of $\hat{\beta}_1$ in the second equation is $\sum x_i^2=30$, not $(\sum x_i)^2=100$. Squaring the sum in place of summing squares changes the equation and gives an incorrect fit.

Worked Example: Computing the Fit from Centered Values

In a hypothetical study of a new work procedure, the predictor is training time in hours and the response is a daily output score. The five observed pairs are $(0,3),(2,4),(4,8),(6,7),(8,13)$. Find the least-squares slope and intercept using centered sums.

The means are $\bar{x}=(0+2+4+6+8)/5=4$ and $\bar{y}=(3+4+8+7+13)/5=35/5=7$. The centered values and products are:

\begin{array}{c|r|r|r} x_i & x_i-\bar{x} & y_i-\bar{y} & (x_i-\bar{x})(y_i-\bar{y})\\ 0&-4&-4&16\\ 2&-2&-3&6\\ 4&0&1&0\\ 6&2&0&0\\ 8&4&6&24 \end{array}

Therefore $S_{xy}=16+6+0+0+24=46$, while

S_{xx}=(-4)^2+(-2)^2+0^2+2^2+4^2 =16+4+0+4+16=40.

The slope is $\hat{\beta}_1=46/40=1.15$. The intercept is $\hat{\beta}_0=7-(1.15)(4)=7-4.6=2.4$. Thus the fitted line is $\hat{y}=2.4+1.15x$.

Check. The line gives $\hat{y}=2.4+1.15(4)=7$ at $x=\bar{x}=4$, matching $\bar{y}$. The raw totals are $\sum x_i=20$, $\sum y_i=35$, $\sum x_i^2=120$, and $\sum x_i y_i=186$. The normal-equation left and right sides are $35$ and $5(2.4)+1.15(20)=35$ for the first equation; for the second they are $186$ and $2.4(20)+1.15(120)=48+138=186$.

Common mistake avoided. The intercept is not the mean response, and it is not generally the response observed at the smallest $x$. It is $\bar{y}-\hat{\beta}_1\bar{x}=2.4$; the fitted line passes through the mean point $(4,7)$.

Worked Example: Using Sample Covariance and Sample Variance

A hypothetical analyst studies four equipment settings $x$ and resulting response measurements $y$, recording $(1,8),(3,5),(4,6),(6,1)$. Find the fitted line using sample covariance and sample variance, and explain why their denominators do not alter the slope.

The means are $\bar{x}=(1+3+4+6)/4=3.5$ and $\bar{y}=(8+5+6+1)/4=5$. The centered products are

(-2.5)(3)=-7.5,\quad (-0.5)(0)=0,\quad (0.5)(1)=0.5,\quad (2.5)(-4)=-10.

Thus $S_{xy}=-7.5+0+0.5-10=-17$. The predictor sum of squares is

S_{xx}=(-2.5)^2+(-0.5)^2+(0.5)^2+(2.5)^2 =6.25+0.25+0.25+6.25=13.

Since $n-1=3$, the sample covariance is $s_{xy}=-17/3$, and the sample variance is $s_x^2=13/3$. Therefore

\hat{\beta}_1=\frac{s_{xy}}{s_x^2} =\frac{-17/3}{13/3}=-\frac{17}{13}\approx -1.3077.

The intercept is

\hat{\beta}_0=5-\left(-\frac{17}{13}\right)(3.5) =5+\frac{119}{26} =\frac{249}{26}\approx 9.5769.

To four decimal places, the fitted line is $\hat{y}=9.5769-1.3077x$. Check: at $\bar{x}=3.5$, the exact fitted value is $249/26-(17/13)(7/2)=(249-119)/26=5=\bar{y}$.

Common mistake avoided. Dividing the sample covariance $-17/3$ by the unscaled sum $S_{xx}=13$ would give $-17/39$, which is wrong. Use either $S_{xy}/S_{xx}$, or divide the sample covariance by the sample variance; do not mix the two conventions.

Interpretation, Conditions, and Exam Traps

The formulas describe the least-squares line for the observed data. They do not, by themselves, establish that the response changes causally with the predictor, that a linear relationship is appropriate outside the observed range, or that a statistical model's assumptions hold. Those are separate questions from solving for the coefficients. The slope is measured in response units per predictor unit, while the intercept is measured in response units.

For calculation, the key condition is $S_{xx}>0$. If $S_{xx}=0$, then every predictor equals $\bar{x}$, the denominator in the slope formula is zero, and the slope cannot be uniquely determined from the observations. When $S_{xx}>0$, the fitted coefficients satisfy both normal equations and minimize the sum of squared residuals.

Common exam errors usually come from mixing equivalent formulas or losing a sign. Keep the following distinctions explicit:

Sum of squares versus squared sum. $S_{xx}=\sum x_i^2-n\bar{x}^2$, not $(\sum x_i)^2$.
Cross-product versus covariance. $S_{xy}$ is a sum; sample covariance divides it by $n-1$. The matching predictor sample variance also divides by $n-1$.
Intercept after slope. Calculate $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$ only after finding the slope, preserving the sign of $\hat{\beta}_1$.
Pairing matters. Each $x_i$ must remain paired with its corresponding $y_i$ when calculating $\sum x_i y_i$ or $S_{xy}$.

Key takeaway: Solve the slope first as the centered cross-product sum divided by the predictor sum of squares, then anchor the line at the sample means with $\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$. A reliable final check is $\hat{\beta}_0+\hat{\beta}_1\bar{x}=\bar{y}$.

Check Your Understanding

For each problem, calculate the requested least-squares coefficients and check your result using the indicated summary or centered quantities.

A small study records the pairs $(1,4),(2,7),(3,5),(4,10)$. Calculate $S_{xy}$, $S_{xx}$, the slope, and the intercept. (Answer: $S_{xy}=8$, $S_{xx}=5$, $\hat{\beta}_1=1.6$, $\hat{\beta}_0=2.5$.)
For the pairs $(2,11),(4,9),(6,8),(8,4)$, calculate the least-squares line from centered sums and verify that its value at $\bar{x}$ equals $\bar{y}$. (Answer: $\bar{x}=5$, $\bar{y}=8$, $S_{xy}=-22$, $S_{xx}=20$, $\hat{\beta}_1=-1.1$, $\hat{\beta}_0=13.5$; the fitted value at $x=5$ is $8$.)
A data set has $n=4$, $\sum x_i=12$, $\sum y_i=20$, $\sum x_i^2=44$, and $\sum x_i y_i=52$. Solve the normal equations and check both equations. (Answer: $\hat{\beta}_1=-1$, $\hat{\beta}_0=8$; $4(8)+12(-1)=20$ and $12(8)+44(-1)=52$.)
A sample has $\bar{x}=3$, $\bar{y}=10$, sample covariance $s_{xy}=-6$, and sample variance $s_x^2=4$. Find the least-squares line. Explain why using the sample covariance directly as the slope would be wrong. (Answer: $\hat{\beta}_1=-6/4=-1.5$, $\hat{\beta}_0=10-(-1.5)(3)=14.5$; the slope is a covariance-to-variance ratio, not the covariance alone.)
For observations with $n=5$, suppose $S_{xy}=12$ and $S_{xx}=8$, while $\bar{x}=2$ and $\bar{y}=6$. Calculate the coefficients and give the fitted value at the predictor mean. (Answer: $\hat{\beta}_1=12/8=1.5$, $\hat{\beta}_0=6-(1.5)(2)=3$; the fitted value at $x=2$ is $6$.)

What You'll Learn