Why the Matrix Product Gives the Fitted Value

From a Regression Equation to a Matrix Product

A fitted value can be calculated in the familiar way: start with an intercept, multiply each predictor value by its coefficient estimate, and add the results. A design matrix packages the predictor values into a row, while a coefficient vector packages the corresponding estimates into a column. The product of those two objects is a compact way to write the same calculation.

Understanding why the product works is useful when reading regression notation or checking a hand calculation. The key is that the multiplication pairs entries in the same positions: the first row entry multiplies the first coefficient, the second row entry multiplies the second coefficient, and so on. The resulting sum is the intercept-plus-terms expression, not a different prediction rule.

The Model and Its Ingredients

Consider a regression model with \(p\) predictor columns. For record \(i\), let \(y_i\) be the observed outcome and let \(x_{ij}\) be the value in predictor column \(j\). In an additive model with an intercept, the model is

Here \(\beta_0,\beta_1,\ldots,\beta_p\) are the model coefficients, and \(\varepsilon_i\) is the error for record \(i\). A standard mean-model condition is that the errors have conditional mean zero given the predictor values: \(E(\varepsilon_i\mid x_{i1},\ldots,x_{ip})=0\). This condition gives the model's conditional mean the intercept-plus-terms form. Other assumptions may be needed for particular statistical procedures, but they are not needed to verify the multiplication that calculates a fitted value.

After estimating the coefficients, write their estimates as \(\hat{\beta}_0,\hat{\beta}_1,\ldots,\hat{\beta}_p\). The fitted value for record \(i\), written \(\hat{y}_i\), is the value obtained by inserting its predictor values and the estimated coefficients into the regression equation:

The fitted value is not the observed outcome \(y_i\). The observed outcome may differ from the fitted value; their difference is the residual. The calculation below concerns only how the fitted value is written as a matrix product.

The superscript \(\mathsf{T}\) indicates a transpose: it makes the displayed sequence of entries a row rather than a column. There are \(p+1\) entries in the design row because it includes one intercept entry as well as the \(p\) predictor entries. The coefficient vector also has \(p+1\) entries, in the corresponding order. Thus the product is defined: a \(1\)-by-\((p+1)\) row multiplied by a \((p+1)\)-by-\(1\) column gives a \(1\)-by-\(1\) number.

Why the Product Reproduces the Equation

For a row vector multiplied by a column vector, the rule is to multiply entries in matching positions and add those products. Applying that rule to the design row and estimated coefficient vector gives

The first equality applies the definition of the design row and coefficient vector. The second follows from the row-by-column multiplication rule. In the third, the intercept term simplifies because \(1\hat{\beta}_0=\hat{\beta}_0\); the other products are written in the same order as the regression equation. The result is exactly the definition of the fitted value.

This also explains why an intercept model includes a 1 in every design row. That entry ensures the intercept estimate is included in the product. Without it, there would be no product term for \(\hat{\beta}_0\). The row entries need not all be measurements in their original form: a predictor column can be an indicator or another constructed term. Whatever its meaning, its entry must be paired with the coefficient estimate for that column.

Indicator Columns Also Follow the Same Rule

An indicator column records membership in a group, often using 1 for one group and 0 for the other. Its coefficient estimate is multiplied by that 0-or-1 entry just like any other coefficient estimate is multiplied by its predictor value. If the entry is 0, that term contributes zero; if it is 1, the coefficient estimate is included in the sum. The multiplication rule does not change for a categorical predictor represented by indicator columns.

Order Is Part of the Calculation

A matrix product pairs positions, not labels. If a design row lists the predictor columns in one order but the coefficient vector lists their estimates in another, the calculation can be wrong even if every number appears somewhere in the two objects. In practice, the order is determined by the model's columns. Write it down before multiplying, especially when a model has several numeric predictors or indicator columns.

For a categorical predictor represented by more than one indicator, each indicator has its own design-row entry and corresponding coefficient. A zero in one indicator column does not remove the other columns from the product; it simply makes that particular product zero. The intercept remains a separate first term in an intercept model.

What the Product Does—and Does Not—Depend On

The identity \(\hat{y}_i=\mathbf{x}_i^\mathsf{T}\hat{\boldsymbol{\beta}}\) is an algebraic statement about a specified design row and its corresponding coefficient estimates. It does not require the errors to be normally distributed, independent, or to have constant variance. Such conditions can matter when estimating coefficients or drawing conclusions about them, but the row-by-column multiplication itself follows from the definition of matrix multiplication.

The product also does not decide which columns belong in the model, how categorical variables are coded, or whether the model is appropriate for a particular scientific question. It applies once the model terms, their order, the record's values, and the coefficient estimates are specified. The result is a fitted outcome on the outcome's scale; it is not automatically an observed outcome or a causal effect.

Common Mistake: Multiplying the Wrong Positions

A common error is to put the predictor values into a row in one order and the coefficients into a vector in another, then multiply matching positions anyway. For example, if a model's columns are age followed by an indicator, the row \((1,\text{age},D)\) must pair with \((\hat{\beta}_0,\hat{\beta}_{\text{age}},\hat{\beta}_D)^\mathsf{T}\). Pairing the age entry with the indicator coefficient changes the model calculation.

The correction is to align the row and vector by model term before doing arithmetic. Check both the number of entries and their order, then expand the product as a sum. If that expanded sum does not match the intercept-plus-terms equation, the row or coefficient vector has been arranged incorrectly.