Multiple Regression Model and Notation

Why Multiple Regression Needs Careful Notation

In simple regression, one predictor is used to describe the conditional mean of a response. Multiple regression extends that setup to two or more predictors. The extension is compact to write, but it creates several quantities that must not be confused: a response variable, an observed response, predictor variables, their observed values, an unknown conditional mean, and a fitted value calculated from estimated coefficients.

For example, a hypothetical analyst might study repair time using both machine age and recent workload. Each machine supplies one observation, with one recorded repair time and a recorded value for each predictor. A regression model describes how the response’s conditional mean depends on those predictor values; it does not say that every observed response equals that mean.

Definition: For $n$ observations and $p$ predictors, the multiple linear regression model is $$ Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\cdots+\beta_px_{ip}+\varepsilon_i,\qquad i=1,\ldots,n. $$ Here $Y_i$ is the response for observation $i$; $x_{ij}$ is the observed value of predictor $j$ for observation $i$; $\beta_0,\ldots,\beta_p$ are unknown population coefficients; and $\varepsilon_i$ is the error for observation $i$. The response and predictor variables describe the study, while $Y_i$ and $x_{ij}$ are their particular observed values.

The subscript $i$ identifies an observation, or case. The subscript $j$ identifies a predictor. Thus $x_{24}$, when $p$ is at least 4, means the value of predictor 4 for observation 2. It is not a coefficient or a separate response. The model has $p$ predictors and $p+1$ coefficients because the intercept $\beta_0$ is included.

The coefficients are population-level parameters in the model. When estimated from data, they are written $\hat{\beta}_0,\hat{\beta}_1,\ldots,\hat{\beta}_p$. A hat indicates an estimate; it does not turn an estimated coefficient into the unknown population parameter itself. The model can include an intercept as written here. A model specified without an intercept is a different model and should be stated explicitly.

Conditional Mean and Error Assumption

The equation separates each response into a systematic part, expressed using the predictors and coefficients, and an error term. To interpret the systematic part as a conditional mean, we require the error to have mean zero given the predictor values. In notation, let $\mathbf{x}_i=(x_{i1},\ldots,x_{ip})$ denote the observed predictor values for case $i$. The condition is $E(\varepsilon_i\mid\mathbf{x}_i)=0$.

Result: Under the multiple regression model and the condition $E(\varepsilon_i\mid\mathbf{x}_i)=0$, the population conditional mean response at the predictor values for observation $i$ is $$ E(Y_i\mid\mathbf{x}_i)=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}. $$ This conditional mean is a population quantity. It is not necessarily equal to the response actually observed for that case.

The equality follows by taking a conditional expectation of both sides of the model. The coefficients and the observed predictor values are fixed when conditioning on $\mathbf{x}_i$; the conditional expectation of the error is zero. Therefore,

\begin{aligned} E(Y_i\mid\mathbf{x}_i) &=E\left(\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}+\varepsilon_i\mid\mathbf{x}_i\right)\\ &=\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}+E(\varepsilon_i\mid\mathbf{x}_i)\\ &=\beta_0+\sum_{j=1}^{p}\beta_jx_{ij}. \end{aligned}

This short derivation also explains why the error term matters: without the zero conditional mean condition, the right-hand side of the regression equation need not equal the conditional mean. The equation alone does not require a normal distribution for errors. Independence, constant error variance, and normality are additional assumptions used for particular properties or inferential procedures; they should not be silently inferred just from the phrase “multiple linear regression.”

Worked Example: Writing a Model and Finding a Conditional Mean

A hypothetical study records repair time $Y$, in hours, for each machine. Its two predictors are machine age $x_1$, in years, and workload $x_2$, in hundreds of operating hours over the prior month. Suppose the stated model has $\beta_0=4$, $\beta_1=0.6$, and $\beta_2=0.08$. For one machine, $x_1=5$, $x_2=20$, and the observed repair time is $Y=9.4$ hours. Write the model, calculate its conditional mean at these predictor values, and find the observed deviation from that mean. No interest rate or mortality assumption applies. No error distribution is specified or needed; use the zero conditional mean assumption.

Method. There are two predictors, so $p=2$, and each observation has an intercept plus two predictor terms:

Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i.

Substitute the stated coefficient values and the machine’s predictor values into the systematic part:

\begin{aligned} E(Y\mid x_1=5,x_2=20) &=4+0.6(5)+0.08(20)\\ &=4+3+1.6\\ &=8.6\text{ hours}. \end{aligned}

The observed response is $9.4$ hours, so its deviation from the conditional mean is $9.4-8.6=0.8$ hours. In this model that difference is the observed error, $\varepsilon=0.8$. Check: $8.6+0.8=9.4$, which recovers the observed response. The common mistake is to report $9.4$ as the conditional mean; it is the observation, while $8.6$ is the model’s mean at the given predictor values.

Estimated Conditional Means and Fitted Values

In applications, the population coefficients are generally unknown. After estimating them, the corresponding fitted value for observation $i$ is

\hat{Y}_i=\hat{\beta}_0+\hat{\beta}_1x_{i1}+\cdots+\hat{\beta}_px_{ip}.

The fitted value $\hat{Y}_i$ is an estimate of the conditional mean at observation $i$’s predictor values. It is calculated by using the observed predictor values with estimated coefficients. The response $Y_i$, by contrast, is the observed outcome. Their difference, $e_i=Y_i-\hat{Y}_i$, is the residual. A residual is calculated after fitting; it is not the unknown model error $\varepsilon_i$, though it is used as an estimate of that error.

The same predictor values give the same fitted value when the same coefficient estimates are used. If two observations share those values but have different observed responses, their fitted values coincide while their residuals differ. This is one reason to keep “fitted mean” and “observed response” distinct.

Worked Example: Two Observations With the Same Predictors

A hypothetical model describes repair time in hours using machine age $x_1$ in years and workload $x_2$ in hundreds of operating hours. The estimated coefficients are $\hat{\beta}_0=4$, $\hat{\beta}_1=0.6$, and $\hat{\beta}_2=0.08$. Two machines each have $x_1=5$ and $x_2=20$; their observed repair times are 9.4 and 7.9 hours. Find each fitted value and residual. No interest rate or mortality assumption applies. No error distribution is specified or needed for these calculations.

Step 1: Calculate the fitted value at the shared predictor values.

\begin{aligned} \hat{Y} &=4+0.6(5)+0.08(20)\\ &=4+3+1.6\\ &=8.6\text{ hours}. \end{aligned}

Step 2: Calculate each residual as observed response minus fitted value. For the first machine, $e_1=9.4-8.6=0.8$ hours. For the second machine, $e_2=7.9-8.6=-0.7$ hours.

Check. Adding each residual back to the fitted value recovers its observation: $8.6+0.8=9.4$ and $8.6-0.7=7.9$. The common mistake is to change the fitted value for the second machine simply because its observed outcome differs. The fitted value is determined by the predictor values and estimated coefficients; the observed outcome determines the residual.

Reading the Indices and the Number of Quantities

The number of observations and the number of predictors answer different questions. If the data contain $n$ rows, there are $n$ response observations, $Y_1,\ldots,Y_n$. If each row has $p$ predictors, there are $p$ predictor variables, each observed $n$ times. The coefficient list contains $p+1$ parameters when the intercept is included. For example, 60 observations with 3 predictors involve 60 response values, 3 predictor variables, and 4 population coefficients.

The terms “predictor” and “predictor value” should also be kept separate. “Machine age” is a predictor variable; “5 years for machine 1” is an observed predictor value. Likewise, “repair time” is the response variable; “9.4 hours for machine 1” is an observed response. Describing the model in context should identify both the variable and its units where relevant.

Worked Example: Counting Predictors and Evaluating a Three-Predictor Model

A hypothetical model relates claim-handling time $Y$, measured in days, to a complexity score $x_1$, document count $x_2$, and specialist-review indicator $x_3$, where $x_3=1$ means a specialist reviewed the claim and $x_3=0$ means otherwise. The stated coefficients are $\beta_0=2$, $\beta_1=1.2$, $\beta_2=0.5$, and $\beta_3=-0.4$. For one claim, $x_1=3$, $x_2=4$, $x_3=2$ is not a valid indicator value; instead, the recorded value is $x_3=1$. Its observed handling time is 7.5 days. Identify $p$, state the coefficient count, and calculate the conditional mean and observed deviation. No interest rate or mortality assumption applies. No error distribution is specified or needed; use the zero conditional mean assumption.

The model has three predictors, so $p=3$, and it has four coefficients including the intercept. The model is

Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3}+\varepsilon_i.

The indicator must be 0 or 1; use the stated valid value $x_3=1$. Substitute all values:

\begin{aligned} E(Y\mid x_1=3,x_2=4,x_3=1) &=2+1.2(3)+0.5(4)-0.4(1)\\ &=2+3.6+2-0.4\\ &=7.2\text{ days}. \end{aligned}

The observed deviation is $7.5-7.2=0.3$ days. Check: $7.2+0.3=7.5$. A common trap is to include the indicator as a fourth predictor because its value is 1. It is one of the three predictors; the intercept is a coefficient, not a predictor.

Common Mistakes and Exam Traps

Confusing $n$ with $p$. $n$ counts observations; $p$ counts predictors. Adding observations does not create additional predictor variables.
Omitting the intercept unintentionally. The standard form shown here includes $\beta_0$. Dropping it changes the model and must be specified.
Calling the observed response the fitted mean. $Y_i$ is observed; $\hat{Y}_i$ is calculated from estimated coefficients and predictor values.
Reading a subscript incorrectly. $x_{ij}$ is predictor $j$ for observation $i$, not coefficient $j$. The corresponding coefficient is $\beta_j$.
Assuming a normal error distribution just from the model equation. The conditional mean model and the distributional assumptions used for inference are distinct statements. State any needed distributional assumptions separately.

A reliable reading order is to identify the response, count the predictors, interpret the two subscripts, and then ask whether a quantity is a population parameter, an estimate, an observation, or a fitted value. This discipline prevents arithmetic that is correct but answers the wrong question. These distinctions are aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Check Your Understanding

For each problem, identify the requested quantities carefully before calculating.

A hypothetical model for daily energy use has response $Y$ and predictors $x_1$, building area, and $x_2$, outdoor temperature. Write its equation for $n$ observations, name the number of coefficients when an intercept is included, and explain what $x_{34}$ would mean if a fourth predictor were added. (Answer: $Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i$; 3 coefficients; with a fourth predictor, $x_{34}$ is predictor 4’s value for observation 3.)
Suppose $Y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\varepsilon_i$, $\beta_0=6$, $\beta_1=2$, $\beta_2=-0.5$, and the predictor values are 3 and 4. Under zero conditional mean error, calculate the conditional mean. (Answer: $6+2(3)-0.5(4)=10$.)
For the fitted model $\hat{Y}=5+1.5x_1-2x_2$, an observation has $x_1=4$, $x_2=1$, and $Y=12$. Calculate its fitted value and residual, using observed minus fitted for the residual. (Answer: $\hat{Y}=5+1.5(4)-2(1)=9$; $e=12-9=3$.)
A data set has 80 observations and 5 predictors, with an intercept. How many response values, predictor variables, and regression coefficients are involved? (Answer: 80 response values, 5 predictor variables, and 6 coefficients.)
Two observations share identical predictor values in a fitted model. Their observed responses are 14 and 11, and their shared fitted value is 12. Find both residuals and state which quantity is shared. (Answer: residuals are $14-12=2$ and $11-12=-1$; the fitted value is shared.)

What You'll Learn