From Paired Observations to a Line Model
Suppose we have \(n\) paired observations \((x_i,y_i)\), where each \(x_i\) is an input and each \(y_i\) is its observed response. A line model assigns the prediction \(a+bx_i\) to the input \(x_i\). The scalar \(a\) is the intercept, and \(b\) is the slope. In a best-fit line problem, the observations generally do not all lie on one line, so we seek values of \(a\) and \(b\) whose predictions are collectively close to the observed responses.
The design matrix records how each parameter contributes to every prediction. Its first column consists of ones because the intercept contributes \(a\) at every input; its second column lists the inputs because the slope contributes \(bx_i\) at observation \(i\). Throughout, the data are real and are listed in the same order in every vector and matrix.
Here \(\mathbf{A}\) has \(n\) rows and two columns, while \(\mathbf{c}\) has two entries. Consequently, \(\mathbf{A}\mathbf{c}\) and \(\mathbf{y}\) both have \(n\) entries. The residual \(\mathbf{y}-\mathbf{A}\mathbf{c}\) is the observed response minus the prediction at each input. Minimizing its Euclidean norm is equivalent to minimizing the sum of the squared residuals, because the square root is increasing on nonnegative numbers.
Why These Two Columns Produce the Predictions
Proof. The \(i\)-th row of \(\mathbf{A}\) is \(\begin{bmatrix}1&x_i\end{bmatrix}\). By the definition of matrix-vector multiplication, its product with \(\mathbf{c}=\begin{bmatrix}a\\b\end{bmatrix}\) is \(1\cdot a+x_i\cdot b=a+bx_i\). This holds for each index \(i=1,\dots,n\), so the entries of \(\mathbf{A}\mathbf{c}\) are precisely all the line predictions, in the order of the observations. \(\square\)
Proof. By the preceding proposition, the \(i\)-th entry of \(\mathbf{A}\mathbf{c}\) is \(a+bx_i\). Therefore the \(i\)-th entry of \(\mathbf{y}-\mathbf{A}\mathbf{c}\) is \(y_i-(a+bx_i)\). The squared Euclidean norm of a real column vector is the sum of the squares of its entries. Applying that definition to the residual vector gives the displayed equality. \(\square\)
These propositions justify the setup: minimizing the residual norm in the matrix problem is exactly the same task as choosing an intercept and slope to minimize the sum of squared errors over the data. The matrix does not itself supply a solution; it expresses all predictions and errors in a compact form that can be analyzed by least-squares methods.
Constructing and Reading the Design Matrix
Worked Example: Four Observations
Use the observations \((-1,1)\), \((0,2)\), \((2,2)\), and \((3,5)\) to set up the line-fitting problem. Do not solve for the best-fit coefficients.
Reading the inputs in order gives \(x_1=-1\), \(x_2=0\), \(x_3=2\), and \(x_4=3\). The first column contains four ones, and the second column contains those four inputs. The observed responses form the right-hand vector:
Multiplying row by row gives
Thus the residual vector and the objective to minimize are
Check: the first prediction is \(a+b(-1)=a-b\), and the fourth is \(a+b(3)=a+3b\), matching the first and fourth entries of the matrix product. Each residual entry is its observed response minus that prediction.
Keeping the observations in a consistent order matters. The \(i\)-th row of \(\mathbf{A}\), the \(i\)-th entry of \(\mathbf{y}\), and the \(i\)-th prediction must all refer to the same paired observation. Reordering the data is fine if the corresponding rows of \(\mathbf{A}\) and entries of \(\mathbf{y}\) are reordered together.
Worked Example: Repeated Inputs with Different Responses
Consider \((0,1)\), \((0,3)\), and \((1,2)\). Set up the least-squares problem and determine whether one line can pass through all three observations exactly.
The inputs give two identical first-column entries in the first two rows, while the responses remain distinct:
The residual and squared error are therefore
An exact fit would require every residual to be zero. The first two equations would then require \(a=1\) and \(a=3\) simultaneously, which is impossible. So no line passes through all three observations exactly; the least-squares formulation still makes sense and seeks parameters minimizing the displayed squared error.
Check: at the repeated input \(x=0\), both predictions are \(a+b\cdot0=a\), so the first two entries of \(\mathbf{A}\mathbf{c}\) must agree. The conflicting observed responses \(1\) and \(3\) confirm that both cannot be fitted exactly.
When the Intercept and Slope Are Distinguishable
The two columns of the design matrix are the all-ones vector and the input vector. If all inputs are identical, these columns are multiples of one another, and different intercept-slope pairs can produce exactly the same predictions. If at least two inputs differ, the columns are independent. This gives a simple data-based test for full column rank, meaning that the two columns are linearly independent.
Proof. Write the columns as \(\mathbf{u}=(1,\dots,1)^{\top}\) and \(\mathbf{x}=(x_1,\dots,x_n)^{\top}\). By the definition of linear independence, they are independent exactly when the equation \(c_1\mathbf{u}+c_2\mathbf{x}=\mathbf{0}\) has only \(c_1=c_2=0\).
First suppose the inputs are not all equal. Then there are indices \(i,j\) with \(x_i\ne x_j\). If \(c_1\mathbf{u}+c_2\mathbf{x}=\mathbf{0}\), its \(i\)-th and \(j\)-th entries give \(c_1+c_2x_i=0\) and \(c_1+c_2x_j=0\). Subtracting the second equation from the first yields \(c_2(x_i-x_j)=0\). Since \(x_i-x_j\ne0\), it follows that \(c_2=0\); substituting into either equation gives \(c_1=0\). Thus the columns are independent.
Conversely, suppose all inputs have the same value \(d\). Then \(\mathbf{x}=d\mathbf{u}\), so \(-d\mathbf{u}+\mathbf{x}=\mathbf{0}\). The coefficients \((-d,1)\) are not both zero because the second coefficient is \(1\). This is a nontrivial linear relation, so the columns are dependent. The two implications prove the stated equivalence. \(\square\)
When the inputs are not all equal, the design matrix has full column rank. The earlier uniqueness result for least-squares problems then guarantees a unique least-squares coefficient vector. When all inputs are equal, the design matrix is rank deficient: it cannot distinguish changes in \(a\) that are offset by changes in \(b\). This is a statement about whether the data identify the two parameters separately, not about whether a least-squares solution exists.
Worked Example: Identical Inputs and Dependent Columns
Take the observations \((2,1)\), \((2,2)\), and \((2,4)\). Show directly why the design matrix cannot distinguish all intercept-slope pairs.
All three inputs equal \(2\), so
For any coefficient vector \(\mathbf{c}=\begin{bmatrix}a\\b\end{bmatrix}\),
For example, \(\begin{bmatrix}1\\0\end{bmatrix}\) and \(\begin{bmatrix}-1\\1\end{bmatrix}\) both give the prediction vector \(\begin{bmatrix}1\\1\\1\end{bmatrix}\), since \(1+2(0)=1\) and \(-1+2(1)=1\). More generally, changing \((a,b)\) to \((a-2t,b+t)\) leaves \(a+2b\) unchanged for every real \(t\). The column dependence is also visible in \(\mathbf{x}=2\mathbf{u}\), or equivalently \(-2\mathbf{u}+\mathbf{x}=\mathbf{0}\).
Check: multiplying the two coefficient vectors by \(\mathbf{A}\) gives, respectively, three copies of \(1+2(0)=1\) and three copies of \(-1+2(1)=1\). The distinct coefficient pairs produce identical predictions, as claimed.
Common Setup Errors
Omitting the column of ones changes the model. Suppose the observations are \((1,3)\) and \((2,5)\). A tempting but incorrect setup is \(\begin{bmatrix}1\\2\end{bmatrix}b\), which predicts \(b\) and \(2b\). It has no parameter for an intercept, so it represents the restricted model \(y=bx\), not the general line \(y=a+bx\). The correct setup is
At \(a=1\) and \(b=2\), the predictions are \(1+2=3\) and \(1+4=5\), matching the responses. The missing-ones setup cannot represent this line: its equations would require both \(b=3\) and \(2b=5\), which no single \(b\) satisfies.
Do not put the observations into the wrong roles. The inputs belong in the second column of \(\mathbf{A}\), while observed responses belong in \(\mathbf{y}\). The matrix describes how parameters generate predictions; the response vector records the target values. Keeping those roles distinct makes the residual \(\mathbf{y}-\mathbf{A}\mathbf{c}\) meaningful.
The design-matrix formulation separates the data from the unknown coefficients. Once the rows and response vector are set up correctly, the familiar least-squares problem applies directly. The key checks are dimensional and interpretive: there must be one row per observation, two columns for an intercept and slope, and the \(i\)-th prediction must be \(a+bx_i\).
Check Your Understanding
For each question, identify the intercept and slope roles and verify dimensions before making a conclusion.
- Write the design matrix, response vector, and coefficient vector for \((1,4)\), \((3,5)\), and \((5,9)\).
- For inputs \(x_1=0\), \(x_2=2\), and \(x_3=4\), write the prediction vector and residual vector in terms of \(a\), \(b\), and the responses \(y_1,y_2,y_3\).
- Explain why the design matrix for three observations with \(x_1=x_2=x_3=-1\) has dependent columns.
- State the sum-of-squared-errors expression represented by \(\|\mathbf{y}-\mathbf{A}\mathbf{c}\|^2\) for four paired observations.
- For the setup \(\mathbf{A}=\begin{bmatrix}1&0\\1&2\\1&4\end{bmatrix}\), determine whether its columns are independent and justify your answer using the input values.
