Why Centered Sums Are Useful
In simple linear regression, the least-squares estimates describe the line that minimizes the sum of squared differences between observed responses and the line’s fitted values. When the data are given as paired observations, the slope and intercept can be computed efficiently from two summaries: the variation in the predictor around its mean and the paired variation of predictor and response around their means.
The key is to preserve the pairing between each predictor value and its response. Separately sorting or summing the two variables can destroy the information needed for the slope. The calculations below use the usual simple linear regression setup: \(n\) observed pairs \((x_i,y_i)\), with an intercept included in the fitted line. The least-squares calculations themselves do not require a probability distribution for the errors; no error distribution or parameters are assumed here.
The Least-Squares Estimates
Write the fitted line as \(\hat{y}=b_0+b_1x\), where \(b_0\) is the estimated intercept and \(b_1\) is the estimated slope. The intercept is included in the model: the line is not being forced through the origin.
The condition \(S_{xx}>0\) means the observed predictor values are not all the same. If they are all equal, changing the slope can be offset by changing the intercept while leaving the fitted value at that one predictor value unchanged. There is then no unique least-squares slope.
Why the Formulas Hold
For any proposed coefficients \(b_0\) and \(b_1\), the sum of squared residuals is \(Q(b_0,b_1)=\sum_{i=1}^{n}(y_i-b_0-b_1x_i)^2\). The least-squares estimates minimize this quantity. At a minimum, its derivatives with respect to the two coefficients are zero.
The first equation gives \(\sum y_i-nb_0-b_1\sum x_i=0\). Dividing by \(n\) and rearranging yields \(b_0=\bar{y}-b_1\bar{x}\). Substitute this expression into the second equation. Since \(y_i-b_0-b_1x_i=(y_i-\bar{y})-b_1(x_i-\bar{x})\), the resulting equation is
Thus \(S_{xy}-b_1S_{xx}=0\). When \(S_{xx}>0\), solving gives \(b_1=S_{xy}/S_{xx}\), and substituting this slope into \(b_0=\bar{y}-b_1\bar{x}\) gives the intercept formula. This derivation also shows why both sums must use deviations from their respective means.
A Reliable Calculation Process
For a short data set, make a row for every paired observation and calculate the centered predictor and response values together. For larger sets, the same result can be obtained from raw sums using \(S_{xx}=\sum x_i^2-n\bar{x}^2\) and \(S_{xy}=\sum x_iy_i-n\bar{x}\bar{y}\). Those identities follow by expanding the centered definitions and using \(\sum x_i=n\bar{x}\) and \(\sum y_i=n\bar{y}\).
Calculate \(\bar{x}\) and \(\bar{y}\) from the paired observations or supplied summaries.
Find \(S_{xx}\) and \(S_{xy}\), keeping each \(x_i\) paired with its \(y_i\).
Use \(b_1=S_{xy}/S_{xx}\), then \(b_0=\bar{y}-b_1\bar{x}\), provided \(S_{xx}>0\).
Substitute \(\bar{x}\) into the fitted line. The result should be \(\bar{y}\), because \(b_0+b_1\bar{x}=\bar{y}\).
The final check is an algebraic consequence of the intercept formula. It is useful for catching arithmetic or substitution errors, but it does not by itself establish that a straight-line model is appropriate for a particular application.
Worked Example: Computing Both Estimates From Paired Data
A hypothetical analyst records a numeric predictor \(x\) and a continuous response \(y\) for four observations. The data are \((1,3),(2,5),(3,4),(4,8)\). Calculate the least-squares slope and intercept, then check the fitted line at the mean predictor. No probability distribution for the errors is specified or needed for this coefficient calculation.
Method. Calculate the means, form the centered sums, and apply the least-squares formulas. The common mistake to avoid is calculating the response deviations without matching them to the corresponding predictor deviations.
First, \(\bar{x}=(1+2+3+4)/4=10/4=2.5\), and \(\bar{y}=(3+5+4+8)/4=20/4=5\). The paired centered calculations are:
| \(x_i\) | \(y_i\) | \(x_i-\bar{x}\) | \(y_i-\bar{y}\) | \((x_i-\bar{x})^2\) | \((x_i-\bar{x})(y_i-\bar{y})\) |
|---|---|---|---|---|---|
| 1 | 3 | -1.5 | -2 | 2.25 | 3 |
| 2 | 5 | -0.5 | 0 | 0.25 | 0 |
| 3 | 4 | 0.5 | -1 | 0.25 | -0.5 |
| 4 | 8 | 1.5 | 3 | 2.25 | 4.5 |
| Sum | 5 | 7 |
Thus \(S_{xx}=2.25+0.25+0.25+2.25=5\), and \(S_{xy}=3+0-0.5+4.5=7\). The slope is \(b_1=7/5=1.4\). The intercept is \(b_0=5-(1.4)(2.5)=5-3.5=1.5\). The fitted line is \(\hat{y}=1.5+1.4x\).
For the check, at \(\bar{x}=2.5\), the fitted value is \(1.5+(1.4)(2.5)=1.5+3.5=5\), which equals \(\bar{y}\). The estimates are internally consistent.
Worked Example: Using Supplied Centered Sums
For a hypothetical set of observations, the summary statistics are \(n=6\), \(\bar{x}=8\), \(\bar{y}=21\), \(S_{xx}=32\), and \(S_{xy}=48\). Find the least-squares line and verify the fitted value at the mean predictor. No probability distribution is specified; the task is to calculate the least-squares coefficients from the summaries.
Method. Since \(S_{xx}=32>0\), the slope is uniquely determined. The common mistake to avoid is dividing the response mean by the predictor mean instead of using the centered cross-product divided by the centered predictor sum of squares.
Calculate the slope: \(b_1=S_{xy}/S_{xx}=48/32=1.5\). Then calculate the intercept: \(b_0=\bar{y}-b_1\bar{x}=21-(1.5)(8)=21-12=9\). Therefore the fitted line is \(\hat{y}=9+1.5x\).
At the mean predictor, the fitted value is \(9+(1.5)(8)=9+12=21\), equal to \(\bar{y}\). The slope is positive because the supplied centered cross-product is positive, while the intercept is the fitted value at \(x=0\) for this fitted line.
Worked Example: Converting Raw Sums and Avoiding a Through-Origin Error
A hypothetical data set contains three paired observations with \(\sum x_i=6\), \(\sum y_i=15\), \(\sum x_i^2=14\), and \(\sum x_iy_i=31\). Calculate the least-squares line with an intercept. No probability distribution is specified or needed for this calculation.
Method. Convert the raw sums to centered sums, then calculate the two coefficients. The common mistake is to use \(\sum x_iy_i/\sum x_i^2\) as the slope, which corresponds to a different fit constrained to pass through the origin.
The means are \(\bar{x}=6/3=2\) and \(\bar{y}=15/3=5\). The centered predictor sum of squares is \(S_{xx}=\sum x_i^2-n\bar{x}^2=14-3(2^2)=14-12=2\). The centered cross-product is \(S_{xy}=\sum x_iy_i-n\bar{x}\bar{y}=31-3(2)(5)=31-30=1\).
Since \(S_{xx}=2>0\), \(b_1=1/2=0.5\). Then \(b_0=5-(0.5)(2)=5-1=4\), so the fitted line is \(\hat{y}=4+0.5x\). At \(\bar{x}=2\), it gives \(4+(0.5)(2)=5=\bar{y}\).
The incorrect through-origin calculation would give \(31/14\approx2.2143\) as the slope. That calculation uses raw sums and omits the intercept adjustment; it does not produce the least-squares slope for a line with an intercept. The corrected centered-sum calculation gives \(0.5\).
Common Mistakes and Interpretation
The slope is a ratio of centered quantities, not a ratio of raw totals. The intercept then depends on both the slope and the two sample means. In particular, calculating \(b_0\) as \(\bar{y}\), or treating \(S_{xy}/S_{xx}\) as the intercept, confuses distinct roles in the fitted line.
- Dropping the pairing: Each product \((x_i-\bar{x})(y_i-\bar{y})\) must use values from the same observation.
- Using the wrong denominator: The denominator is \(S_{xx}\), the sum of squared centered predictor values, not the number of observations or \(\sum x_i^2\) by itself.
- Forgetting the intercept adjustment: Once the slope is found, use \(b_0=\bar{y}-b_1\bar{x}\).
- Ignoring the zero-variation case: If every \(x_i\) is identical, \(S_{xx}=0\), so the slope formula cannot be used.
These calculations produce coefficient estimates for the observed data. Whether the fitted relationship is useful, how uncertain the estimates are, and what assumptions are appropriate for inference are separate questions. For the coefficient calculation here, the essential requirements are paired observations and, for a unique slope, variation among the predictor values.
Check Your Understanding
For each problem, calculate the requested least-squares coefficients or centered sums. Each data set is hypothetical; no error distribution is specified or needed for these coefficient calculations.
- A data summary gives \(\bar{x}=3\), \(\bar{y}=11\), \(S_{xx}=8\), and \(S_{xy}=12\). Find the slope and intercept, and check the fitted value at \(\bar{x}\). (Answer: \(b_1=1.5,\ b_0=6.5\); the fitted value at \(x=3\) is \(11\).)
- For a set of observations, \(\bar{x}=10\), \(\bar{y}=4\), \(S_{xx}=20\), and \(S_{xy}=-30\). Calculate the fitted line and verify its value at the mean predictor. (Answer: \(\hat{y}=19-1.5x\); at \(x=10\), \(\hat{y}=4\).)
- The paired data are \((1,4),(2,2),(3,6)\). Calculate \(\bar{x}\), \(\bar{y}\), \(S_{xx}\), and \(S_{xy}\), then find the least-squares line. (Answer: \(\bar{x}=2,\ \bar{y}=4,\ S_{xx}=2,\ S_{xy}=2;\ \hat{y}=2+x\).)
- For \(n=3\), suppose \(\sum x_i=6\), \(\sum y_i=15\), \(\sum x_i^2=14\), and \(\sum x_iy_i=31\). Convert to centered sums and find the fitted line with an intercept. (Answer: \(\bar{x}=2,\ \bar{y}=5,\ S_{xx}=2,\ S_{xy}=1;\ \hat{y}=4+0.5x\).)
- Every observed predictor value in a data set equals \(7\), while the response values vary. What is \(S_{xx}\), and can the least-squares slope with an intercept be uniquely calculated? (Answer: \(S_{xx}=0\); no unique slope can be calculated.)