A Complete Least-Squares Calculation from Raw Data

From Paired Observations to a Fitted Line

A regression output table can make a fitted model look as though it appeared all at once. In practice, the calculation begins with paired observations: each predictor value is matched to a response value. Working from those pairs helps distinguish the slope for a line with an intercept from a different calculation that forces the line through the origin. It also makes it possible to check fitted values, residuals, and the scale of the residual error.

This tutorial uses ordinary least squares (OLS) for simple linear regression. The goal is to calculate the estimates and residual standard error from a small data set, not to conduct a hypothesis test or construct an interval.

Model and definitions: For $n$ paired observations $(x_i,y_i)$, the simple linear regression model is $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$. The OLS estimates, denoted $\hat{\beta}_0$ and $\hat{\beta}_1$, minimize the sum of squared residuals. The fitted response is $\hat{y}_i=\hat{\beta}_0+\hat{\beta}_1x_i$, and the residual is $e_i=y_i-\hat{y}_i$. The residual sum of squares is $RSS=\sum_{i=1}^{n}e_i^2$.

For the usual interpretation of the residual standard error as an estimate of the error standard deviation, assume the errors have mean zero, are independent, and have a common variance $\sigma^2$, conditional on the observed predictor values. Normal errors are additionally needed for exact small-sample inference, but not to carry out the least-squares calculations below. The predictor values must not all be identical, and $n$ must exceed 2 to estimate the residual standard error.

The Calculation From Raw Data

First calculate the sample means and the centered sums of squares and cross-products:

\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i,\qquad \bar{y}=\frac{1}{n}\sum_{i=1}^{n}y_i,\qquad S_{xx}=\sum_{i=1}^{n}(x_i-\bar{x})^2,\qquad S_{xy}=\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}).

For a model that includes an intercept, the coefficient estimates are

\hat{\beta}_1=\frac{S_{xy}}{S_{xx}},\qquad \hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}.

The residual standard error is the square root of the residual sum of squares divided by the residual degrees of freedom:

s=\sqrt{\frac{RSS}{n-2}}.

There are $n-2$ residual degrees of freedom because fitting the intercept and slope uses two estimated parameters. The units of $s$ are the units of the response $y$, not the predictor $x$.

Why the coefficient formulas hold: Write the sum of squared residuals for candidate coefficients $b_0,b_1$ as $Q(b_0,b_1)=\sum_i(y_i-b_0-b_1x_i)^2$. At its minimum, setting the two derivatives to zero gives $\sum_i(y_i-b_0-b_1x_i)=0$ and $\sum_i x_i(y_i-b_0-b_1x_i)=0$. The first equation implies $b_0=\bar{y}-b_1\bar{x}$. Substituting this into the second gives $\sum_i(x_i-\bar{x})(y_i-\bar{y})-b_1\sum_i(x_i-\bar{x})^2=0$, so $b_1=S_{xy}/S_{xx}$. This also shows why the calculation uses centered values when the model includes an intercept.

Worked Calculations

Worked Example: A Complete Fit From Five Pairs

A hypothetical analyst records a predictor $x$ and response $y$ for five observations. The response and predictor are both measured in their stated data units. Calculate the least-squares line, all fitted values and residuals, and the residual standard error.

Observation	$x_i$	$y_i$
1	1	2
2	2	3
3	3	5
4	4	4
5	5	6

1. Calculate the means and centered sums. The totals are $\sum x_i=15$, $\sum y_i=20$, $\sum x_i^2=55$, and $\sum x_iy_i=69$. Therefore $\bar{x}=15/5=3$ and $\bar{y}=20/5=4$. Using the equivalent raw-total formulas,

S_{xx}=55-\frac{15^2}{5}=55-45=10,\qquad S_{xy}=69-\frac{(15)(20)}{5}=69-60=9.

2. Estimate the coefficients. The slope is $\hat{\beta}_1=9/10=0.9$. The intercept is $\hat{\beta}_0=4-(0.9)(3)=1.3$. Thus the fitted line is $\hat{y}=1.3+0.9x$.

3. Calculate fitted values and residuals. Substituting each $x_i$ into the fitted line gives:

$x_i$	$y_i$	$\hat{y}_i=1.3+0.9x_i$	$e_i=y_i-\hat{y}_i$	$e_i^2$
1	2	2.2	-0.2	0.04
2	3	3.1	-0.1	0.01
3	5	4.0	1.0	1.00
4	4	4.9	-0.9	0.81
5	6	5.8	0.2	0.04

4. Calculate the residual standard error. The residual sum of squares is $RSS=0.04+0.01+1.00+0.81+0.04=1.90$. With $n-2=5-2=3$ residual degrees of freedom, $s=\sqrt{1.90/3}=\sqrt{0.633333\ldots}=0.796$, rounded to three decimal places.

Check. The residuals sum to $-0.2-0.1+1.0-0.9+0.2=0$, and their predictor-weighted sum is $(1)(-0.2)+(2)(-0.1)+(3)(1.0)+(4)(-0.9)+(5)(0.2)=0$. These checks are consistent with the fitted intercept model. A common mistake is to use the uncentered ratio $\sum x_iy_i/\sum x_i^2=69/55\approx1.2545$ as the slope. That ratio is not the slope for this intercept model; the correct slope is $S_{xy}/S_{xx}=9/10=0.9$. The uncentered ratio comes from a different least-squares problem in which the line is constrained to pass through the origin.

Worked Example: Keeping Track of Response Units

In a hypothetical operations study, $x$ is staff inspection hours and $y$ is weekly cost in thousands of dollars. Four pairs are $(2,5),(4,7),(6,8),(8,12)$. Find the fitted line, residual standard error, and interpretation of the slope in dollars per hour.

1. Find the sample summaries. Here $\sum x_i=20$, $\sum y_i=32$, $\sum x_i^2=4+16+36+64=120$, and $\sum x_iy_i=10+28+48+96=182$. Thus $\bar{x}=20/4=5$, $\bar{y}=32/4=8$, $S_{xx}=120-20^2/4=120-100=20$, and $S_{xy}=182-(20)(32)/4=182-160=22$.

2. Fit the line. The slope is $22/20=1.1$ thousand dollars per inspection hour. The intercept is $8-(1.1)(5)=2.5$ thousand dollars. Therefore $\hat{y}=2.5+1.1x$, where fitted responses are in thousands of dollars.

3. Compute residuals and their squared sum. At $x=2,4,6,8$, the fitted values are $4.7,6.9,9.1,11.3$. Subtracting fitted from observed responses gives residuals $0.3,0.1,-1.1,0.7$, in thousands of dollars. Hence $RSS=0.3^2+0.1^2+(-1.1)^2+0.7^2=0.09+0.01+1.21+0.49=1.80$.

4. Calculate and interpret $s$. There are $4-2=2$ residual degrees of freedom, so $s=\sqrt{1.80/2}=\sqrt{0.9}=0.948683\ldots$ thousand dollars. Rounded to the nearest cent after conversion, this is $0.948683\ldots\times1000=\$948.68$. The slope is $1.1\times1000=\$1{,}100$ per additional inspection hour. A unit trap would be to report $s$ as $0.95$ dollars; the calculation was in thousands of dollars, so the conversion is essential.

Check. The residuals sum to $0.3+0.1-1.1+0.7=0$, and their products with the corresponding predictor values sum to $2(0.3)+4(0.1)+6(-1.1)+8(0.7)=0$. The residual standard error is in response units, as required.

Worked Example: Exact Residual Calculations With Six Observations

For a hypothetical data set, the six pairs are $(1,3),(2,5),(3,4),(4,6),(5,7),(6,8)$. Find the fitted line, residual sum of squares, and residual standard error. Keep enough precision to avoid rounding the residuals before squaring.

1. Calculate the centered sums. The totals are $\sum x_i=21$, $\sum y_i=33$, $\sum x_i^2=91$, and $\sum x_iy_i=132$. Thus $\bar{x}=21/6=3.5$, $\bar{y}=33/6=5.5$, $S_{xx}=91-21^2/6=91-73.5=17.5$, and $S_{xy}=132-(21)(33)/6=132-115.5=16.5$.

2. Estimate the coefficients. The slope is $16.5/17.5=33/35$. The intercept is $5.5-(33/35)(3.5)=5.5-3.3=2.2$. Therefore $\hat{y}=2.2+(33/35)x$.

3. Calculate fitted values and residuals. The fitted values, in observation order, are $110/35,143/35,176/35,209/35,242/35,275/35$, or approximately $3.142857,4.085714,5.028571,5.971429,6.914286,7.857143$. Subtracting these fitted values from the observed responses gives residuals $-5/35,32/35,-36/35,1/35,3/35,5/35$.

4. Calculate the error summary. Squaring the exact residuals gives $RSS=(25+1024+1296+1+9+25)/1225=2380/1225=68/35=1.942857\ldots$. There are $6-2=4$ residual degrees of freedom. Consequently $s=\sqrt{(68/35)/4}=\sqrt{17/35}=\sqrt{0.485714\ldots}=0.697$, rounded to three decimal places.

Check. The residual numerators sum to $-5+32-36+1+3+5=0$. Their predictor-weighted numerators sum to $-5+64-108+4+15+30=0$. Both checks hold. Rounding each residual to a few decimals before squaring can change the final RSS slightly; retain exact fractions or adequate precision until the final rounding.

Checks, Units, and Common Traps

A complete calculation is easier to audit when each stage is kept separate: raw totals, centered sums, coefficient estimates, fitted values, residuals, then $RSS$ and $s$. In particular, do not square a response or residual in a different unit from the one used for the other terms. If $y$ is measured in thousands of dollars, then residuals and $s$ are also measured in thousands of dollars; converting to dollars means multiplying by 1,000.

For a fitted line with an intercept, two useful arithmetic checks are $\sum_i e_i=0$ and $\sum_i x_ie_i=0$, up to rounding. A failure can indicate a subtraction error in a residual, a coefficient error, or use of a slope formula for a different model. Also remember that $RSS$ is a sum of squared residuals, while $s^2=RSS/(n-2)$ is the estimated error variance and $s$ is its square root. Dividing by $n$ or taking a square root before dividing gives a different, incorrect result for residual standard error.

These calculations provide a foundation for reading regression summaries: they show what the coefficient and error quantities represent and how their units arise. The numerical residual standard error summarizes the observed residual scatter around the fitted line, but by itself it does not establish that the linear model assumptions are appropriate.

Check Your Understanding

For each problem, calculate the requested quantities before checking the final answer.

For the pairs $(1,2),(2,3),(3,5)$, calculate the fitted intercept and slope for a model with an intercept. (Answer: $\hat{\beta}_0=1/3,\ \hat{\beta}_1=1.5$.)
For $(0,1),(2,4),(4,6)$, calculate the fitted response at $x=2$. (Answer: $11/3$.)
For $(1,2),(2,4),(3,5),(4,7)$, calculate $RSS$ and the residual standard error. (Answer: $RSS=0.20,\ s=\sqrt{0.10}\approx0.3162$, in response units.)
A hypothetical response, measured in dollars, is observed with predictor values in hours: $(2,100),(4,120),(6,130),(8,170)$. Calculate the slope, $RSS$, and residual standard error. (Answer: $\hat{\beta}_1=11$ dollars per hour, $RSS=180$ dollars squared, $s=\sqrt{90}\approx9.4868$ dollars.)
For $(1,3),(2,5),(3,4),(4,8),(5,10)$, calculate the fitted line and residual standard error. (Answer: $\hat{y}=0.9+1.7x,\ s=\sqrt{1.7}\approx1.3038$, in response units.)

Observation	\(x_i\)	\(y_i\)
1	1	2
2	2	3
3	3	5
4	4	4
5	5	6

What You'll Learn