Two Least-Squares Models, Two Different Constraints
A line constrained to pass through the origin may seem like a minor variation on ordinary simple linear regression. It is not: fixing the intercept at zero changes the optimization problem, generally changes the fitted slope, and removes some familiar residual identities. The constraint should represent a defensible feature of the relationship, not just a preference for a simpler-looking equation.
For observed pairs \((x_i,y_i)\), \(i=1,\ldots,n\), ordinary least squares with an estimated intercept minimizes the sum of squared residuals over both coefficients. The through-origin fit minimizes the same quantity but restricts the intercept to zero. These are algebraic least-squares calculations; the formulas for the minimizing coefficients do not require a probability distribution for the observations. For statistical interpretation, one may separately posit a regression model such as \(Y_i=\beta_0+\beta_1x_i+\varepsilon_i\), with fixed predictor values and errors having mean zero and constant variance. A through-origin model additionally asserts \(\beta_0=0\); if that assertion is not appropriate, the constraint can distort the fitted relationship.
How the Slope Changes
Write the residual sum of squares for a through-origin candidate slope \(b\) as \(Q(b)=\sum_{i=1}^n(y_i-bx_i)^2\). Differentiating this quadratic with respect to \(b\) gives the minimizing slope, provided \(\sum x_i^2>0\).
The numerator and denominator are uncentered sums: they are not generally the covariance-like sum \(\sum(x_i-\bar{x})(y_i-\bar{y})\) and the centered sum \(\sum(x_i-\bar{x})^2\) used for the ordinary slope. With an estimated intercept, the coefficients are
when the centered predictor sum of squares is positive. The intercept model can choose a vertical shift as well as a slope; the origin model cannot. The two slopes therefore answer different optimization questions, and they need not be close.
Worked Example: Comparing Both Fits for Three Calibration Points
A hypothetical device is calibrated at predictor settings \(x=1,2,3\), producing measurements \(y=2,3,5\). Find both least-squares lines, calculate their residuals, and check the residual identities relevant to each fit. Treat these as fixed observed data; no distributional assumption is needed for the calculations.
Method. First compute the ordinary fit using centered sums, then compute the constrained slope using uncentered sums. This comparison avoids assuming that a through-origin fit must use the ordinary slope.
For the ordinary fit, \(\bar{x}=(1+2+3)/3=2\) and \(\bar{y}=(2+3+5)/3=10/3\). The centered predictor sum is \(\sum(x_i-\bar{x})^2=1+0+1=2\). The centered cross-product sum is \(\sum(x_i-\bar{x})(y_i-\bar{y})=4/3+0+5/3=3\). Thus \(b_1=3/2\) and \(b_0=10/3-(3/2)(2)=1/3\), giving \(\hat{y}=1/3+(3/2)x\).
The ordinary fitted values are \(11/6,10/3,29/6\). Subtracting them from the observations gives residuals \(1/6,-1/3,1/6\). Their sum is \(1/6-1/3+1/6=0\), and their predictor-weighted sum is \(1(1/6)+2(-1/3)+3(1/6)=0\).
For the origin fit, \(\sum x_i y_i=1(2)+2(3)+3(5)=23\) and \(\sum x_i^2=1+4+9=14\). Therefore \(b_{\mathrm{O}}=23/14\). The fitted values are \(23/14,46/14,69/14\), so the residuals are \(5/14,-4/14,1/14\). Their sum is \(2/14=1/7\), not zero. But their predictor-weighted sum is \(5/14-8/14+3/14=0\).
Check. The ordinary line passes through \((\bar{x},\bar{y})=(2,10/3)\), while the constrained line passes through \((0,0)\). The calculations confirm that the origin-fit residuals need not sum to zero, although their predictor-weighted sum is zero.
Which Residual Identities Still Hold?
The normal equation for the ordinary fit includes one condition for each fitted coefficient. Differentiating the residual sum of squares with respect to the intercept gives \(\sum e_i=0\); differentiating with respect to the slope gives \(\sum x_i e_i=0\). The constrained fit has only one free coefficient, its slope, so only the second condition is imposed.
This also explains why orthogonality to fitted values survives. In the through-origin model, fitted values are multiples of the predictor vector, and residuals are orthogonal to that vector. But a vector of ones is not among the fitted directions. There is no least-squares condition requiring the residuals to be orthogonal to that constant vector, so the residual sum is generally nonzero. Consequently, the fitted line is not generally required to pass through the data centroid \((\bar{x},\bar{y})\).
There is a corresponding sum-of-squares identity, but it is naturally uncentered. Since \(y_i=\hat{y}_i+e_i\) and \(\sum \hat{y}_i e_i=0\), expansion gives
By contrast, the familiar decomposition of variation about \(\bar{y}\) uses centered quantities and the residual-sum-zero property of the fit with an intercept. It does not automatically carry over to a line through the origin. In particular, inserting the origin-fit residual sum of squares into \(1-\mathrm{SSE}/\sum(y_i-\bar{y})^2\) can produce a negative value. That result signals that the constrained fit can perform worse than predicting the sample mean under this centered comparison; it is not a contradiction of least squares, which optimized a different objective.
Decide whether the model estimates an intercept or fixes it at zero. Do not infer the constraint merely from a data pattern.
For a free intercept, center \(x\) and \(y\); for an origin fit, use \(\sum x_i y_i/\sum x_i^2\).
Both fits have \(\sum x_i e_i=0\). Only the fit with an estimated intercept guarantees \(\sum e_i=0\).
Model Comparison and a Common Exam Trap
The model with an estimated intercept contains every line through the origin as a special case, by setting \(b_0=0\). Thus it has at least as many choices over which to minimize squared residuals. Its minimum residual sum of squares cannot be greater than that of the constrained fit on the same data. This comparison does not establish that the intercept model is scientifically preferable; it only describes the two optimization problems.
Worked Example: A Constraint That Substantially Raises the Residual Sum
A hypothetical sensor produces readings \(y=10,11,10\) at settings \(x=1,2,3\). Calculate the two fitted lines and residual sums of squares, then evaluate the usual centered expression \(1-\mathrm{SSE}/\sum(y_i-\bar{y})^2\) for the origin fit. No probability model is needed.
Method. Use centered sums for the unrestricted fit and uncentered sums for the constrained fit. The common mistake is to assume that the origin fit minimizes residuals relative to the sample mean.
Here \(\bar{x}=2\), \(\bar{y}=31/3\), \(\sum(x_i-\bar{x})^2=2\), and \(\sum(x_i-\bar{x})(y_i-\bar{y})=(-1)(-1/3)+0(2/3)+(1)(-1/3)=0\). Therefore \(b_1=0\), \(b_0=31/3\), and the ordinary fitted values are all \(31/3\). The residuals are \(-1/3,2/3,-1/3\), giving \(\mathrm{SSE}=1/9+4/9+1/9=2/3\).
For the origin fit, \(\sum x_i y_i=10+22+30=62\) and \(\sum x_i^2=14\), so \(b_{\mathrm{O}}=31/7\). The fitted values are \(31/7,62/7,93/7\), and the residuals are \(39/7,15/7,-23/7\). Thus \(\mathrm{SSE}=(39^2+15^2+23^2)/49=(1521+225+529)/49=2275/49=325/7\).
The centered total sum of squares is \(2/3\), so the stated centered expression for the origin fit is \(1-(325/7)/(2/3)=1-975/14=-961/14\), approximately \(-68.64\). It is negative because this constrained fit has a much larger SSE than the sample-mean prediction benchmark. For comparison, the ordinary fit has SSE \(2/3\), equal to the centered total sum of squares here.
Check. The origin fit's SSE, \(325/7\), exceeds the unrestricted fit's \(2/3\), as the nested-model argument requires. The negative centered value is not a computational error; it warns against treating the usual centered R-squared interpretation as automatic for a through-origin fit.
A useful quick check is to calculate the residual sum directly from the fitted slope: \(\sum e_i=\sum y_i-b_{\mathrm{O}}\sum x_i\). It will usually not be zero. In contrast, \(\sum x_i e_i=0\) should hold, apart from small rounding differences. If the predictor values are all zero, the denominator in the slope formula is zero and no unique slope is determined by this calculation; the formula applies when \(\sum x_i^2>0\).
Worked Example: Checking the Residual Conditions Without an Intercept
For a hypothetical process, predictor settings are \(x=1,2,4\) and measured responses are \(y=3,2,5\). Fit the line through the origin and check both the residual sum and predictor-weighted residual sum. Treat the observations as fixed; no distributional assumption is needed.
Method. Calculate the uncentered slope, form every residual, and check each sum. This avoids applying the intercept-model residual-sum identity by habit.
The numerator is \(\sum x_i y_i=3+4+20=27\), and the denominator is \(\sum x_i^2=1+4+16=21\). Hence \(b_{\mathrm{O}}=27/21=9/7\). The fitted values are \(9/7,18/7,36/7\); the residuals are \(12/7,-4/7,-1/7\).
The residual sum is \(12/7-4/7-1/7=1\), so it is not zero. The predictor-weighted sum is \(1(12/7)+2(-4/7)+4(-1/7)=12/7-8/7-4/7=0\). As an additional check, \(\sum x_i y_i-b_{\mathrm{O}}\sum x_i^2=27-(9/7)(21)=0\), exactly the normal equation for the fitted slope.
Check. Both direct checks agree: the residual sum is \(1\), while the predictor-weighted residual sum is \(0\). The line is the least-squares fit through the origin, but it does not satisfy every residual identity of a fit with a free intercept.
Practical Checklist
Before interpreting a through-origin regression, check that zero on the predictor scale is meaningful and that a zero predictor should imply a zero response under the intended relationship. A mere absence of observed data near zero does not establish that condition. Also remember that changing the units of the predictor or response changes the numerical coefficient in the corresponding way; it does not turn an unjustified zero-intercept restriction into a justified one.
- For the through-origin fit, use \(b_{\mathrm{O}}=\sum x_i y_i/\sum x_i^2\), not the centered ordinary slope.
- Expect \(\sum x_i e_i=0\) and \(\sum\hat{y}_i e_i=0\) for the fitted origin model.
- Do not assume \(\sum e_i=0\) or that the fitted line passes through \((\bar{x},\bar{y})\).
- Distinguish the uncentered identity \(\sum y_i^2=\sum\hat{y}_i^2+\sum e_i^2\) from the centered decomposition used for an intercept fit.
- When using a centered R-squared expression for a constrained fit, recognize that it can be negative and does not have the automatic interpretation of the ordinary fit's R-squared.
Check Your Understanding
For each question, show the coefficient and residual checks before comparing with the answers.
- For \(x=(1,2)\) and \(y=(3,4)\), find the through-origin slope and the sum of its residuals. (Answer: \(b_{\mathrm{O}}=11/5\); \(\sum e_i=2/5\).)
- For \(x=(1,2,3)\) and \(y=(2,4,5)\), find both the through-origin slope and the ordinary intercept and slope. (Answer: \(b_{\mathrm{O}}=25/14\); ordinary line \(\hat{y}=2/3+(3/2)x\).)
- For \(x=(1,2,3)\) and \(y=(1,3,4)\), find the through-origin slope, the residual sum, and the predictor-weighted residual sum. (Answer: \(b_{\mathrm{O}}=19/14\); \(\sum e_i=-1/7\); \(\sum x_i e_i=0\).)
- For \(x=(1,2)\) and \(y=(4,4)\), calculate the origin-fit SSE and the ordinary-fit SSE. Is the centered total sum of squares positive, and can the centered R-squared expression be evaluated? (Answer: origin-fit SSE \(=16/5\); ordinary-fit SSE \(=0\); centered total sum of squares \(=0\), so the centered expression is undefined.)
- For \(x=(1,2,4)\) and \(y=(3,2,5)\), calculate the through-origin slope, residual sum, and SSE. (Answer: \(b_{\mathrm{O}}=9/7\); \(\sum e_i=1\); \(\mathrm{SSE}=23/7\).)