From Raw Measurements to a Fit
A regression calculation can be numerically correct and still answer the wrong question if the data have been mishandled first. An observation’s predictor and response belong together; sorting one column on its own breaks that pairing. Missing measurements require an explicit rule, and a change of units or scale must be reflected in the model. After fitting, residuals provide checks on both the arithmetic and the pattern left unexplained by the model.
We will use a small dataset with one predictor \(x\) and one measured response \(y\). Each row represents one observation. The regression model used below has an intercept and a linear predictor. Its fitted response is a vector of predicted values, and its residual vector is the observed response minus the fitted response.
The norm here is the Euclidean norm: for a vector \(\mathbf{v}\), \(\|\mathbf{v}\|^2\) is the sum of the squares of its entries. In a line fit with predictor values \(z_i\), the \(i\)th row of the design matrix is \((1,z_i)\), and the coefficient vector contains the intercept and slope. A fit is least squares because it minimizes the sum of squared residuals, not because it necessarily passes through every observation.
Order and Missing Measurements Before Calculating
Consider records entered in an untidy order. The symbol “missing” means no response was recorded; it is not a numerical value.
| Record | \(x\) | \(y\) |
|---|---|---|
| C | 4 | 5 |
| A | 1 | 2 |
| E | 5 | 4 |
| B | 2 | 3 |
| D | 3 | missing |
Worked Example: Sort Records and Apply a Missing-Value Rule
Sort by \(x\), then use a complete-case rule for a regression of \(y\) on \(x\): retain a record only when both values are observed.
First sort the records by their \(x\)-values, moving each entire row so that its record label and response stay with it. The sorted order is A, B, D, C, E. Record D has an observed predictor but no response, so it cannot contribute a paired predictor-response observation to this fit. Under the stated rule, remove D. The analysis table is therefore:
| Record | \(x\) | \(y\) |
|---|---|---|
| A | 1 | 2 |
| B | 2 | 3 |
| C | 4 | 5 |
| E | 5 | 4 |
The retained predictor vector is \((1,2,4,5)^\top\), and its response vector is \((2,3,5,4)^\top\), in that same order. The complete-case rule is a choice about which observations enter the calculation, not a claim that the omitted response should have any particular value. If missingness is related to the unobserved response, excluding incomplete records can distort the pattern in the retained data. A different analysis may need a justified imputation or missing-data method; inserting an invented value without stating a rule is not a valid repair.
Reordering every row together changes only the order in which observations are written. It does not change the underlying least-squares fit.
Proof. A simultaneous row permutation rearranges the entries of \(\mathbf{A}\mathbf{x}-\mathbf{b}\) in the same way for every \(\mathbf{x}\). Its squared norm is the sum of the squares of those entries, and rearranging a finite sum does not change its value. Thus the objective \(\|\mathbf{A}\mathbf{x}-\mathbf{b}\|^2\) has exactly the same value for every coefficient vector before and after the permutation. The vectors minimizing that objective are therefore the same. \(\square\)
Scale the Predictor Without Losing Its Meaning
Centering and scaling can make predictor values easier to work with and can clarify the units of a coefficient. For a predictor \(x\), a common transformation is \(z=(x-c)/s\), where \(c\) is a chosen center and \(s\ne0\) is a chosen scale. If the fitted model in scaled coordinates is \(\widehat y=\alpha+\gamma z\), substituting the definition of \(z\) gives an equation in the original predictor units. The fitted predictions should not change merely because the predictor has been represented on a different scale.
Worked Example: Convert a Scaled Equation Back to Original Units
Suppose a measurement is recorded in hundreds of units, with \(x=1,2,4,5\), and the responses are \(2,3,5,4\). Center at \(c=3\) and scale by \(s=2\). Then \(z=(x-3)/2\). Compute the transformed predictor values in the same order as the observations:
If a fit in the scaled variable has equation \(\widehat y=\frac72+\frac65z\), substitute \(z=(x-3)/2\):
The scaled slope \(\frac65\) is the change in predicted response for a one-unit increase in \(z\), which is a two-unit increase in \(x\). Thus the slope per unit of \(x\) is \(\frac65\div2=\frac35\), as the converted equation shows. For a check, at \(x=4\), the scaled equation gives \(\frac72+\frac65(\frac12)=\frac{41}{10}\). The converted equation gives \(\frac{17}{10}+\frac35(4)=\frac{41}{10}\) as well.
Centering and scaling do not fix incorrect pairings or missing-data decisions. They change the numerical representation of a predictor, so keep the center and scale with the analysis. If a coefficient is reported in the original units, convert the fitted equation back as shown rather than interpreting a scaled slope as a per-unit-\(x\) effect.
Fit the Line and Verify Its Residuals
We now fit the complete-case observations from the first example. Use the scaled predictor \(z=(x-3)/2\), and retain the response order \(2,3,5,4\). The design matrix and response vector are
The columns of \(\mathbf{A}\) represent the intercept and scaled predictor. A coefficient vector \(\mathbf{x}=(\alpha,\gamma)^\top\) gives fitted values \(\alpha+\gamma z_i\). For this small fit, the normal equations are \(\mathbf{A}^\top\mathbf{A}\widehat{\mathbf{x}}=\mathbf{A}^\top\mathbf{b}\).
Worked Example: Least-Squares Fit with Residual Checks
Calculate each side of the normal equations. The sum of the scaled predictor values is \(-1-\frac12+\frac12+1=0\), and their squared values sum to \(1+\frac14+\frac14+1=\frac52\). Therefore,
The response sum is \(2+3+5+4=14\). The sum of scaled predictor times response is \((-1)(2)+(-\frac12)(3)+(\frac12)(5)+(1)(4)=-2-\frac32+\frac52+4=3\). Hence
The first equation is \(4\alpha=14\), so \(\alpha=\frac72\). The second is \(\frac52\gamma=3\), so \(\gamma=3\cdot\frac25=\frac65\). The fitted equation is \(\widehat y=\frac72+\frac65z\).
Substitute each scaled predictor value to calculate the fitted response:
Subtract the fitted responses from the observations to obtain the residuals:
Checks: Adding fitted values and residuals recovers the observed responses: \(\frac{23}{10}-\frac3{10}=2\), \(\frac{29}{10}+\frac1{10}=3\), \(\frac{41}{10}+\frac9{10}=5\), and \(\frac{47}{10}-\frac7{10}=4\). Also, the residual sum is \(({-3}+1+9-7)/10=0\), and their scaled-predictor weighted sum is \((-1)(-\frac3{10})+(-\frac12)(\frac1{10})+(\frac12)(\frac9{10})+(1)(-\frac7{10})=0\). These checks agree with the normal equations. The sum of squared residuals is \((9+1+81+49)/100=\frac75\).
For a model with an intercept and a single predictor, zero residual sum means the residuals are orthogonal to the constant column, and zero predictor-weighted residual sum means they are orthogonal to the predictor column. These are checks on the least-squares calculation. They do not establish that a line is an appropriate scientific model: residuals should also be inspected for patterns, unusual observations, and changes in spread, with care not to draw strong conclusions from a very small dataset.
Proof. Write \(\mathbf{r}=\mathbf{b}-\mathbf{A}\widehat{\mathbf{x}}\). First suppose \(\widehat{\mathbf{x}}\) minimizes the squared norm. For any \(\mathbf{h}\in\mathbb{R}^p\) and real \(t\), the vector \(\widehat{\mathbf{x}}+t\mathbf{h}\) is an allowed candidate, so minimality gives \(\|\mathbf{r}-t\mathbf{A}\mathbf{h}\|^2-\|\mathbf{r}\|^2\ge0\). Expanding the square gives \(-2t\mathbf{r}^\top\mathbf{A}\mathbf{h}+t^2\|\mathbf{A}\mathbf{h}\|^2\ge0\) for every real \(t\). If \(\mathbf{r}^\top\mathbf{A}\mathbf{h}\ne0\), a sufficiently small \(t\) with the same sign as that quantity makes the linear term negative with magnitude greater than the quadratic term, contradicting the inequality. Thus \(\mathbf{r}^\top\mathbf{A}\mathbf{h}=0\) for every \(\mathbf{h}\). Taking \(\mathbf{h}\) to be each standard basis vector of \(\mathbb{R}^p\) shows \(\mathbf{A}^\top\mathbf{r}=\mathbf{0}\), or equivalently \(\mathbf{A}^\top(\mathbf{A}\widehat{\mathbf{x}}-\mathbf{b})=\mathbf{0}\).
Conversely, suppose \(\mathbf{A}^\top(\mathbf{A}\widehat{\mathbf{x}}-\mathbf{b})=\mathbf{0}\), so \(\mathbf{A}^\top\mathbf{r}=\mathbf{0}\). For any candidate \(\widehat{\mathbf{x}}+\mathbf{h}\), its residual is \(\mathbf{r}-\mathbf{A}\mathbf{h}\). Expanding and using \(\mathbf{r}^\top\mathbf{A}\mathbf{h}=(\mathbf{A}^\top\mathbf{r})^\top\mathbf{h}=0\) gives \(\|\mathbf{r}-\mathbf{A}\mathbf{h}\|^2=\|\mathbf{r}\|^2+\|\mathbf{A}\mathbf{h}\|^2\ge\|\mathbf{r}\|^2\). Therefore no candidate has a smaller squared residual norm, and \(\widehat{\mathbf{x}}\) is a least-squares minimizer. \(\square\)
Common Mistake: Sorting Columns Independently
Sorting the predictor and response columns separately can create pairs that were never measured. In the retained records, sorting \(x\) gives \(1,2,4,5\), while sorting the responses separately gives \(2,3,4,5\). Pairing these sorted lists would produce \((1,2),(2,3),(4,4),(5,5)\), rather than the actual pairs \((1,2),(2,3),(4,5),(5,4)\). The incorrect data have means \(\overline{x}=3\) and \(\overline{y}=\frac72\). Their centered cross-product sum is \(3+\frac12+\frac12+3=7\), while the centered predictor square sum is \(4+1+1+4=10\). The resulting slope would be \(\frac7{10}\), not the correct slope \(\frac35\) from the actual observations. The arithmetic is consistent with the wrong pairings; the correction is to move complete rows together, never to sort columns independently.
A dependable workflow keeps each decision visible: preserve observation pairings, state how incomplete records are treated, record any scaling transformation, solve the least-squares problem, and compare the fitted values and residuals with the data. Each stage can be checked before proceeding, which makes it easier to find whether a surprising result comes from the model or from preparing the measurements.
Check Your Understanding
Use exact arithmetic, keep observation pairings intact, and state the rule or calculation behind each answer.
- For records \((x,y)=(3,7),(1,2),(2,\text{missing})\), list the sorted records and the complete cases for a regression of \(y\) on \(x\).
- If \(z=(x-4)/2\) and a fitted equation is \(\widehat y=5-3z\), express it in terms of \(x\).
- For \(\mathbf{A}=\begin{bmatrix}1&-1\\1&0\\1&1\end{bmatrix}\) and \(\mathbf{b}=\begin{bmatrix}1\\2\\5\end{bmatrix}\), calculate the normal equations and the least-squares coefficients.
- For a fitted model with an intercept, what sum involving the residuals should be zero at a least-squares solution? Explain why.
- Describe a concrete error caused by sorting response values separately from their predictor values.
