From Observations to a Matrix
Many applications begin with a collection of records: one record for each person, time point, location, or other observational unit. Before applying matrix calculations, the records need a consistent arrangement. A matrix makes that arrangement explicit, but it does not decide what the rows and columns mean. Those meanings must be specified and kept fixed throughout the calculation.
Suppose there are \(n\) observations and \(p\) measured numerical variables. Use one row per observation and one column per variable. Write the resulting data matrix as \(\mathbf{X}\in\mathbb{R}^{n\times p}\). Its entry \(x_{ij}\) is the recorded value of variable \(j\) for observation \(i\). If each observation also has one numerical response value, place those values in a separate column vector \(\mathbf{y}\in\mathbb{R}^n\), with entry \(y_i\) belonging to the same observation as row \(i\) of \(\mathbf{X}\).
The dimensions record an important part of the data structure: \(\mathbf{X}\) has \(n\) rows and \(p\) columns, while \(\mathbf{y}\) has \(n\) entries. This shared row count is not just a formatting preference. A later calculation involving \(\mathbf{X}\) and \(\mathbf{y}\) pairs values from the same observation. A mismatch in row order can therefore change the calculation, even when the two objects separately contain all the right values.
For example, if the columns of \(\mathbf{X}\) are “hours studied” and “practice problems completed,” then the first row must give both values for the same observation. The first entry of \(\mathbf{y}\) must be the response for that same observation. A column heading, a unit, and a row identifier help make these meanings auditable. An identifier can be stored alongside the data for reference, but it is not automatically a numerical variable to include in matrix arithmetic.
Building and Checking the Arrangement
Decide what one row represents, such as one participant or one day. Do not mix different kinds of units in the same row sequence without a clear reason.
Assign each measured variable one column and record its units. Keep this column order fixed when interpreting the entries.
For every row \(i\), place the associated response in \(y_i\). Use identifiers to check this pairing when records are sorted or filtered.
Confirm that \(\mathbf{X}\) has \(n\) rows and \(p\) columns, and that \(\mathbf{y}\) has \(n\) entries. Ordinary matrix multiplication requires numerical entries; labels and blanks need explicit treatment before such calculations.
Numerical entries in different columns may have different units. The matrix records those values; it does not make them comparable or put them on a common scale. Changing units or rescaling variables is a separate operation. In particular, do not alter entries while organizing the data unless the change is deliberate and documented.
Worked Example: Three Observations and Two Variables
Three observations have values for variables \(x_1\) and \(x_2\), together with a response \(y\): \((1,2;4)\), \((2,1;3)\), and \((3,0;2)\). Arrange the records as \(\mathbf{X}\) and \(\mathbf{y}\), and compute two products that commonly occur in later linear-model calculations.
Keep the observations in the listed order and the variables in the order \(x_1,x_2\). Then
The dimensions are \(3\times2\) for \(\mathbf{X}\) and \(3\times1\) for \(\mathbf{y}\), so \(\mathbf{X}^{\top}\mathbf{X}\) and \(\mathbf{X}^{\top}\mathbf{y}\) are defined. Compute each entry by multiplying corresponding entries and adding:
Similarly,
Check: The first component of \(\mathbf{X}^{\top}\mathbf{y}\) uses the first variable values paired with their own responses: \(1(4)+2(3)+3(2)=16\). The second uses \(2(4)+1(3)+0(2)=11\). Both products use all three matched rows, and the dimensions of the results are \(2\times2\) and \(2\times1\), respectively.
Why Consistent Row Order Matters
Changing the order of observations is harmless if every object indexed by observation is reordered in exactly the same way. This can be stated precisely. A permutation matrix is a square matrix obtained by reordering the rows of an identity matrix. Multiplying on the left by a permutation matrix reorders the rows of a matrix or the entries of a column vector. Its rows are distinct standard basis rows, so multiplying it by its transpose gives the identity.
Proof. Because \(\mathbf{P}\) reorders the standard basis rows, each row and column of \(\mathbf{P}\) contains one \(1\) and otherwise zeros. Its transpose reverses that reordering, so \(\mathbf{P}^{\top}\mathbf{P}=\mathbf{I}_n\). Using the definitions of \(\widetilde{\mathbf{X}}\) and \(\widetilde{\mathbf{y}}\), and the rule \((\mathbf{A}\mathbf{B})^{\top}=\mathbf{B}^{\top}\mathbf{A}^{\top}\), we obtain \(\widetilde{\mathbf{X}}^{\top}\widetilde{\mathbf{X}}=(\mathbf{P}\mathbf{X})^{\top}(\mathbf{P}\mathbf{X})=\mathbf{X}^{\top}\mathbf{P}^{\top}\mathbf{P}\mathbf{X}=\mathbf{X}^{\top}\mathbf{X}\). Likewise, \(\widetilde{\mathbf{X}}^{\top}\widetilde{\mathbf{y}}=(\mathbf{P}\mathbf{X})^{\top}(\mathbf{P}\mathbf{y})=\mathbf{X}^{\top}\mathbf{P}^{\top}\mathbf{P}\mathbf{y}=\mathbf{X}^{\top}\mathbf{y}\). Thus simultaneous reordering preserves both products. \(\square\)
Worked Example: Reordering All Records Together
Use the data from the previous example, but list observations in the order third, first, second. The corresponding permutation matrix and reordered objects are
Compute the products directly to check the theorem:
Check: These are exactly the products from the original ordering. In each reordered row, the response stayed attached to the observation whose variables appear in that row.
The theorem does not say that the order of \(\mathbf{X}\) can be changed independently of the order of \(\mathbf{y}\). For instance, pairing the first row of the original \(\mathbf{X}\) with the third response changes which observations are associated. Such a mismatch is not a harmless reordering: it changes the data being represented.
Missing Entries and Usable Rows
A missing value is not the number zero. Zero is a measured numerical value that participates in addition and multiplication; a missing entry has no numerical value to use in those operations. Therefore a table with blanks or missing-value markers is not yet an ordinary real matrix for calculations such as \(\mathbf{X}^{\top}\mathbf{X}\). Replacing a blank by zero without justification changes the recorded information and generally changes the products.
One transparent way to form a numerical matrix is to identify a set of rows for which all required variables and the response are observed, then retain those same rows in both objects. This is often called a complete-case restriction for the variables being used. It gives aligned numerical inputs, but it may discard information; it does not by itself establish that the retained observations are representative of all observations.
Proof. Let \(I=(i_1,\ldots,i_k)\) be the retained indices in their chosen order. The \(j\)th entry of \(\mathbf{X}_I^{\top}\mathbf{y}_I\) is, by the definition of matrix multiplication, \(\sum_{\ell=1}^{k}x_{i_\ell j}y_{i_\ell}\). Each term uses the value of variable \(j\) and the response from the same original observation \(i_\ell\), and every retained observation contributes exactly once. No omitted index contributes a term. \(\square\)
Worked Example: Restricting to Complete Observations
Suppose the records are \((x_1,x_2;y)=(1,2;5)\), \((2,\text{missing};6)\), \((3,1;7)\), and \((4,0;\text{missing})\). If both variables and the response are required, only observations 1 and 3 are complete. Retain those rows in their original order:
Now calculate:
Check: The first component \(26\) is \(1(5)+3(7)\), and the second is \(2(5)+1(7)\); both use only observations 1 and 3. The incomplete second and fourth records contribute no terms to these restricted products.
Different calculations may require different sets of observed values. If one product is computed from one subset of rows and another product from a different subset, they do not describe the same collection of observations. Any such choice should be explicit. Methods for addressing missingness beyond restricting to complete rows are a separate statistical question; the matrix notation alone does not determine a missing value.
Common Pitfall: Deleting a Missing Value and Shifting the Pairing
Consider one variable with observed values \(x=(1,\text{missing},3)^{\top}\) and responses \(\mathbf{y}=(10,20,30)^{\top}\). The complete rows are 1 and 3, so the matched vectors are \(\mathbf{x}_I=(1,3)^{\top}\) and \(\mathbf{y}_I=(10,30)^{\top}\). Their product is \(1(10)+3(30)=100\).
A wrong calculation deletes the missing value from \(\mathbf{x}\) but leaves the full response vector unchanged. It then pairs \((1,3)\) with \((10,20)\), producing \(1(10)+3(20)=70\). This is not a harmless shortcut: it pairs the third observation’s \(x\)-value with the second observation’s response. The correction is to apply the same row selection to both objects, giving \((1,3)\) and \((10,30)\), or to use another explicitly defined method that handles missing entries.
Keep the row identity, variable order, response order, and missing-value treatment visible. Once those choices are consistent, matrix dimensions and products become reliable summaries of the observations actually being used.
Check Your Understanding
For each question, state which observations and variables are being used before carrying out any calculation.
- A data matrix has \(8\) observations and \(3\) variables. What are its dimensions, and how many entries must the matching response vector have?
- Explain why sorting the rows of \(\mathbf{X}\) requires the same reordering of \(\mathbf{y}\).
- If a response is missing for one record, explain why inserting zero is not the same as omitting that observation.
- For \(\mathbf{X}=\begin{bmatrix}1&2\\2&0\end{bmatrix}\) and \(\mathbf{y}=\begin{bmatrix}3\\4\end{bmatrix}\), calculate \(\mathbf{X}^{\top}\mathbf{y}\).
- In the example with \(x=(1,\text{missing},3)^{\top}\), what response entries must accompany the retained \(x\)-values?
