From Raw Observations to a Fitted Line
A simple regression fit can be calculated directly from paired observations. The essential discipline is to keep the predictor values, response values, fitted values, and residuals distinct throughout the arithmetic. A small error in a mean or cross-product can change every coefficient that follows.
Suppose a hypothetical study records \(n\) pairs \((x_i,y_i)\), where \(x_i\) is a predictor value and \(y_i\) is its corresponding response. A straight-line fit assigns the value \(\hat{y}_i=b_0+b_1x_i\) to observation \(i\). The least-squares coefficients are those that minimize the sum of the squared differences between observed and fitted responses.
The calculation below concerns the least-squares fit to the observed pairs; it does not require a probability distribution or normality assumption. To obtain a unique slope and intercept, however, the predictor values must not all be identical. Equivalently, \(S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2\) must be positive. If all predictors are the same, the data cannot distinguish a slope from a compensating change in the intercept.
The Calculation Formulas
First compute the sample means \(\bar{x}=\sum x_i/n\) and \(\bar{y}=\sum y_i/n\). Then form the centered sums \(S_{xx}\) and \(S_{xy}\), where \(S_{xy}=\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})\). Provided \(S_{xx}>0\), the least-squares coefficients are:
The slope is the centered cross-product divided by the centered sum of predictor squares. The intercept then makes the fitted line pass through \((\bar{x},\bar{y})\). Once the coefficients are known, calculate each fitted value, subtract it from the observed response to obtain the signed residual, and square and sum the residuals for \(RSS\).
A useful arithmetic alternative is to calculate the centered sums from raw sums:
These expressions give the same centered sums. They can be efficient when a table of \(x_i\), \(y_i\), \(x_i^2\), and \(x_iy_i\) is easy to construct. They also provide an independent check on a calculation using deviations from the means.
Why the Slope Formula Holds
For any proposed line, the quantity being minimized is \(Q(b_0,b_1)=\sum [y_i-(b_0+b_1x_i)]^2\). Differentiating with respect to the two coefficients and setting the derivatives equal to zero gives the least-squares normal equations:
The first equation gives \(\sum y_i-nb_0-b_1\sum x_i=0\), so \(b_0=\bar{y}-b_1\bar{x}\). Substitute this expression into the second equation. The resulting equation is \(\sum (x_i-\bar{x})(y_i-\bar{y})=b_1\sum (x_i-\bar{x})^2\). If \(S_{xx}>0\), division gives \(b_1=S_{xy}/S_{xx}\), and substitution gives the intercept formula. Thus the formulas follow from minimizing the squared vertical deviations, rather than from an arbitrary rule.
Worked Calculations
Worked Example: Fitting a Line to Training-Hour Data
In a hypothetical training study, four participants report hours of practice \(x\), and an assessment records a score \(y\). The observed pairs are \((1,4),(2,5),(4,8),(5,8)\). Calculate the least-squares coefficients, all fitted values and residuals, and \(RSS\).
Method. Use centered sums, then calculate the fitted responses and signed residuals. No distributional assumption is needed for this arithmetic fit.
The means are \(\bar{x}=(1+2+4+5)/4=12/4=3\) and \(\bar{y}=(4+5+8+8)/4=25/4=6.25\). The centered calculations are:
Thus \(S_{xy}=4.50+1.25+1.75+3.50=11\). The squared predictor deviations sum to \(S_{xx}=(-2)^2+(-1)^2+1^2+2^2=4+1+1+4=10\). Therefore \(b_1=11/10=1.1\), and \(b_0=6.25-(1.1)(3)=6.25-3.30=2.95\). The fitted line is \(\hat{y}=2.95+1.1x\).
At \(x=1\), \(\hat{y}=2.95+1.1=4.05\), so \(e=4-4.05=-0.05\). At \(x=2\), \(\hat{y}=2.95+2.2=5.15\), so \(e=5-5.15=-0.15\). At \(x=4\), \(\hat{y}=2.95+4.4=7.35\), so \(e=8-7.35=0.65\). At \(x=5\), \(\hat{y}=2.95+5.5=8.45\), so \(e=8-8.45=-0.45\).
The residual sum of squares is \(RSS=(-0.05)^2+(-0.15)^2+(0.65)^2+(-0.45)^2=0.0025+0.0225+0.4225+0.2025=0.6500\). As a check, the signed residuals sum to \(-0.05-0.15+0.65-0.45=0\), consistent with the intercept fit. Common mistake avoided: using the fitted value minus the observed value would reverse every residual sign. Here residuals are defined as observed minus fitted; the squared sum would be unchanged by reversing all signs, but the reported residuals would be wrong.
Worked Example: Using Raw Sums and Finding One Residual
A hypothetical analyst records four paired measurements \((0,2),(2,3),(4,7),(6,8)\). Use raw sums to find the fitted line, then calculate the residual for the observation with \(x=2\) and the full \(RSS\).
Method. Calculate \(S_{xx}\) and \(S_{xy}\) from raw sums. This avoids calculating each deviation individually, but the sums and correction terms must use the same \(n=4\) observations.
Here \(\sum x_i=0+2+4+6=12\), \(\sum y_i=2+3+7+8=20\), \(\sum x_i^2=0+4+16+36=56\), and \(\sum x_iy_i=0+6+28+48=82\). The means are \(\bar{x}=12/4=3\) and \(\bar{y}=20/4=5\). Therefore:
The slope is \(b_1=22/20=1.1\), and the intercept is \(b_0=5-(1.1)(3)=1.7\). Thus \(\hat{y}=1.7+1.1x\). For \(x=2\), the fitted value is \(1.7+1.1(2)=3.9\); the observed response is 3, so the residual is \(3-3.9=-0.9\).
For completeness, at \(x=0\) the fitted value is \(1.7\) and the residual is \(2-1.7=0.3\). At \(x=4\), the fitted value is \(1.7+4.4=6.1\) and the residual is \(7-6.1=0.9\). At \(x=6\), the fitted value is \(1.7+6.6=8.3\) and the residual is \(8-8.3=-0.3\). Hence \(RSS=(0.3)^2+(-0.9)^2+(0.9)^2+(-0.3)^2=0.09+0.81+0.81+0.09=1.80\).
The answer is checked by the centered sums: deviations in \(x\) are \(-3,-1,1,3\), and deviations in \(y\) are \(-3,-2,2,3\). Their squared \(x\)-deviations sum to \(20\), and their cross-products sum to \(9+2+2+9=22\), matching the raw-sum method. Common mistake avoided: the correction in \(S_{xy}\) is \((\sum x_i)(\sum y_i)/n=240/4=60\), not a correction using a squared sum or an incorrect sample size.
Worked Example: A Complete Fit with a Small Residual Sum
A hypothetical equipment study pairs four operating settings \(x\) with recorded output \(y\): \((1,2),(2,4),(3,5),(4,7)\). Calculate the line and tabulate the fitted values and residuals. Report \(RSS\) to four decimal places.
Method. Find the means and centered sums first; retain exact values until the final sum of squared residuals. The means are \(\bar{x}=(1+2+3+4)/4=2.5\) and \(\bar{y}=(2+4+5+7)/4=18/4=4.5\).
The predictor deviations are \(-1.5,-0.5,0.5,1.5\), so \(S_{xx}=2.25+0.25+0.25+2.25=5\). The response deviations are \(-2.5,-0.5,0.5,2.5\). Therefore \(S_{xy}=(-1.5)(-2.5)+(-0.5)(-0.5)+(0.5)(0.5)+(1.5)(2.5)=3.75+0.25+0.25+3.75=8\).
The slope is \(b_1=8/5=1.6\), and the intercept is \(b_0=4.5-(1.6)(2.5)=4.5-4=0.5\). The fitted line is \(\hat{y}=0.5+1.6x\). For \(x=1\), the fit is \(2.1\) and the residual is \(2-2.1=-0.1\). For \(x=2\), the fit is \(3.7\) and the residual is \(4-3.7=0.3\). For \(x=3\), the fit is \(5.3\) and the residual is \(5-5.3=-0.3\). For \(x=4\), the fit is \(6.9\) and the residual is \(7-6.9=0.1\).
Thus \(RSS=(-0.1)^2+(0.3)^2+(-0.3)^2+(0.1)^2=0.01+0.09+0.09+0.01=0.20\), or \(0.2000\) to four decimal places. The residuals sum to \(-0.1+0.3-0.3+0.1=0\), and the fitted line at \(\bar{x}=2.5\) gives \(0.5+1.6(2.5)=4.5=\bar{y}\). Both checks agree with the calculated coefficients.
Common Arithmetic Traps
A hand calculation is easiest to audit when every intermediate quantity is visible. The following errors are especially consequential:
- Pairing values incorrectly. Each \(x_i\) must remain paired with the \(y_i\) observed alongside it. Reordering one column without applying the same reordering to the other changes the cross-product sum and therefore the slope.
- Mixing centered and raw-sum formulas. In \(S_{xy}\), either sum \((x_i-\bar{x})(y_i-\bar{y})\), or use \(\sum x_iy_i-(\sum x_i)(\sum y_i)/n\). Do not subtract the raw-sum correction from an already centered sum.
- Reversing the residual convention. This tutorial uses \(e_i=y_i-\hat{y}_i\). State the convention and apply it consistently. Squaring hides a sign error in \(RSS\), so a correct \(RSS\) alone does not verify the residual column.
- Rounding coefficients too early. If a slope or intercept is not terminating, retain sufficient precision during fitted-value calculations and round only the requested final result. Early rounding can noticeably affect a small \(RSS\).
- Dividing by \(n\) in the slope formula. The slope is \(S_{xy}/S_{xx}\); both centered sums already use the observed data. Their ratio is not divided by \(n\) again.
This hand workflow is aligned with the SOA VEE Mathematical Statistics syllabus because it connects least-squares coefficients to the actual paired observations and squared fitting errors. When \(S_{xx}=0\), do not apply the slope formula: all observed predictor values are identical, so only the fitted response at that predictor value is identified. The calculations also describe fit to the observed data; they do not by themselves establish a causal relationship or validate a regression model.
Check Your Understanding
For each problem, calculate the requested quantities from the paired data. Use residual \(e_i=y_i-\hat{y}_i\).
- A hypothetical study records \((1,2),(2,4),(3,5)\). Find \(b_0\), \(b_1\), the residual at \(x=2\), and \(RSS\). (Answer: \(b_0=2/3,\ b_1=1.5\), residual at \(x=2\) is \(1/3\), and \(RSS=1/6\).)
- For the observations \((0,3),(1,4),(3,5),(4,8)\), calculate the least-squares line using raw sums, then find every residual and \(RSS\). (Answer: \(\hat{y}=2.8+1.1x\); residuals \(0.2,0.1,-1.1,0.8\); \(RSS=1.90\).)
- A hypothetical calibration has pairs \((2,5),(4,6),(6,8),(8,9)\). Find the line, the fitted value at \(x=6\), its residual, and the total \(RSS\). (Answer: \(\hat{y}=3.5+0.7x\); fitted value \(7.7\); residual \(0.3\); \(RSS=0.20\).)
- Three observations have predictor \(x=2\) each and responses \(1,3,4\). Find the minimum \(RSS\), and explain why the slope is not uniquely determined. (Answer: \(RSS=14/3\); the fitted value at \(x=2\) is \(8/3\), but infinitely many lines with \(b_0+2b_1=8/3\) give those same fitted values.)
- For \((1,2),(3,3),(5,7),(7,8)\), compute the least-squares line, the residual at \(x=5\), and \(RSS\). (Answer: \(\hat{y}=0.6+1.1x\); residual at \(x=5\) is \(0.9\); \(RSS=1.80\).)