The Simple Regression Equation

From a Relationship Plot to an Equation

A scatterplot can suggest that a quantitative response tends to increase or decrease as a quantitative predictor changes. A simple linear regression model expresses that pattern with an equation. The equation describes a population relationship; it does not require every observed response to lie exactly on a straight line. Individual responses may vary around the linear pattern.

For example, a hypothetical analyst might examine study time and quiz scores. The predictor is study time, and the response is quiz score. A regression equation can describe how the mean score changes with study time, while allowing individual students to score above or below that mean. The equation describes an association; by itself, it does not show that changing study time causes a change in score.

What the Model Says

The equation separates the response into two parts: a systematic component, \(\beta_0+\beta_1x_i\), and an error component, \(\varepsilon_i\). For any predictor value \(x\), the systematic component gives the model's mean response at that value. The error represents an individual response's deviation from that mean.

For the basic model, assume that the mean response is linear in the predictor, that errors have mean zero at each predictor value, and that their variance is the same across predictor values. We also assume errors from different observations are independent, conditional on their predictor values. In symbols, the mean and variance conditions are

These are modeling assumptions, not facts guaranteed by writing down an equation. The linear-mean assumption says the expected response follows a straight-line pattern. The equal-variance assumption says responses have the same amount of vertical scatter around that mean at different predictor values. The independence assumption concerns the errors for different observations. Normal errors are not needed just to define or interpret this equation.

The intercept and slope have units determined by the variables. The intercept is measured in response units. The slope is measured in response units per predictor unit. If the predictor is measured in hours and the response in points, the intercept is in points and the slope is in points per hour. The intercept may have little practical meaning if a predictor value of zero is impossible or far outside the observed range, even though it remains part of the equation.

The Model Equation and the Estimated Regression Line

The model equation uses population parameters \(\beta_0\) and \(\beta_1\), which describe the population relationship and are generally unknown. A sample of observations is used to estimate them. Denote the resulting estimates by \(b_0\) and \(b_1\). The estimated regression line is

Here \(\hat{y}_i\), read “y-hat,” is the fitted value for observation \(i\): the value predicted by the estimated line at predictor value \(x_i\). The symbols \(b_0\) and \(b_1\) are sample estimates; they are not the population parameters themselves. The model equation describes a population mean plus random error, while the estimated line gives a single prediction for each predictor value using estimates from a sample.

The estimated line is not automatically a statement about the true population relationship. It is an estimate based on observed data. Different samples can produce different \(b_0\) and \(b_1\), and therefore different fitted lines. The rule used to choose those estimates is a separate question.

Model Errors and Sample Differences

The model error \(\varepsilon_i\) is the difference between an observed response and the population mean response at that observation's predictor value. Since the population parameters are unknown, the model error is not usually available as a directly calculated quantity. In contrast, the fitted value uses estimated coefficients. The difference between an observed response and its fitted value is a sample quantity, the residual.

These distinctions matter because the two lines involved are different: the population mean line uses \(\beta_0\) and \(\beta_1\), while the estimated line uses \(b_0\) and \(b_1\). Therefore, a model error and a residual need not have the same value. A residual reflects both individual variation around the population mean and the fact that the population line has been estimated.

Justifying the Mean and Slope Interpretations

The slope interpretation follows directly by comparing the model mean at two predictor values. If the predictor changes from \(x\) to \(x+1\), the model gives the mean responses \(\beta_0+\beta_1x\) and \(\beta_0+\beta_1(x+1)\). Their difference is

The intercept interpretation follows by setting \(x=0\): \(\beta_0+\beta_1(0)=\beta_0\). Both interpretations concern the mean response described by the model. They do not claim that an individual response must change by exactly the slope for a one-unit increase, because the individual error may also differ.

Common Mistakes and Why the Distinctions Matter

• Calling the estimated line the population model. The model contains unknown parameters \(\beta_0,\beta_1\) and errors. The estimated line uses the sample-based estimates \(b_0,b_1\) and supplies fitted values.
• Interpreting the slope as an individual change. The slope describes a change in the model's mean response per predictor unit, not a guaranteed change for every individual.
• Forgetting the units. A slope is measured in response units per predictor unit, not just “units.” State both variables' units when interpreting it.
• Treating the intercept as automatically meaningful. It is the model's mean response at predictor value zero. If zero is outside a sensible or observed range, that interpretation may not be useful in the application.
• Confusing error and residual. The model error is defined relative to the population mean line; the residual is calculated relative to the estimated line. They are not generally equal.
• Reading an association as a cause. A regression equation describes a relationship in the data under its assumptions. The equation alone does not establish that changing the predictor causes a response change.