Response Variance Versus Error Variance

Why Two Variances Appear in Regression

A response can vary for two distinct reasons. Individuals with different predictor values may have different expected responses, and individuals with the same predictor value may differ from that expected response. Regression distinguishes these sources: the first is variation in the conditional mean, and the second is conditional error variation.

This distinction matters whenever a question asks how dispersed the response is in a population. The variance of the response across that population generally includes both sources. By contrast, the regression error variance describes variation around the conditional mean at a given predictor value. The two quantities answer different questions and are not generally equal.

Definition: For a response $Y$ and predictor $X$, the conditional mean is $E[Y\mid X]$, and the conditional variance is $\operatorname{Var}(Y\mid X)$. The marginal variance $\operatorname{Var}(Y)$ describes variation in $Y$ across the population, without fixing $X$.

The Law of Total Variance

The law of total variance separates the overall response variance into two nonnegative components. The first measures how much the conditional mean changes across predictor values. The second averages the response variance remaining within predictor values.

Theorem: Provided $Y$ has finite variance, $$ \operatorname{Var}(Y) = \operatorname{Var}\bigl(E[Y\mid X]\bigr) + E\bigl[\operatorname{Var}(Y\mid X)\bigr]. $$ The first term is the variance of the conditional means across the distribution of $X$. The second is the expected conditional variance, averaged over that same distribution.

A short derivation follows by writing $m(X)=E[Y\mid X]$. Add and subtract $m(X)$ inside the centered response:

Y-E[Y] = \bigl(Y-m(X)\bigr) + \bigl(m(X)-E[Y]\bigr).

Squaring and taking expectations gives the variance of $Y$. The cross-product has expectation zero because $E[Y-m(X)\mid X]=0$. Specifically,

E\left[(Y-m(X))(m(X)-E[Y])\right] = E\left[(m(X)-E[Y])E[Y-m(X)\mid X]\right] =0.

The expected square of the first term is $E[\operatorname{Var}(Y\mid X)]$, while the expected square of the second is $\operatorname{Var}(m(X))$. Adding those terms proves the identity. The decomposition requires a finite response variance; it does not require a linear conditional mean or normally distributed errors.

Connection to the Simple Linear Regression Model

In the simple linear regression model, write

Y=\beta_0+\beta_1X+\varepsilon, \qquad E[\varepsilon\mid X]=0.

These conditions imply $E[Y\mid X]=\beta_0+\beta_1X$. If the model is also homoscedastic, meaning $\operatorname{Var}(\varepsilon\mid X)=\sigma^2$ for every relevant $X$, then $\operatorname{Var}(Y\mid X)=\sigma^2$. When $X$ is a random variable with finite variance, the law of total variance becomes

\operatorname{Var}(Y) = \beta_1^2\operatorname{Var}(X)+\sigma^2.

The intercept does not appear because adding a constant shifts every conditional mean equally and does not change its variance. The slope and the spread of $X$ determine the variation in conditional means; $\sigma^2$ describes the average squared spread around those means.

Important distinction: At a fixed predictor value $x$, the homoscedastic model gives $\operatorname{Var}(Y\mid X=x)=\sigma^2$. Across a population in which $X$ varies, $\operatorname{Var}(Y)=\beta_1^2\operatorname{Var}(X)+\sigma^2$. The latter is at least as large as the former, and it is strictly larger when $\beta_1\ne 0$ and $\operatorname{Var}(X)>0$.

If the conditional error variance changes with the predictor, write $\operatorname{Var}(\varepsilon\mid X)=\sigma^2(X)$. The corresponding decomposition is

\operatorname{Var}(Y) = \beta_1^2\operatorname{Var}(X) + E[\sigma^2(X)].

Here the second component is an average over the predictor distribution, not a single common error variance. In either version, if the predictor is treated as fixed at one value, there is no variation in the conditional mean across predictor values to include in that conditional variance. The decomposition concerns variation across a distribution of $X$, or across predictor values viewed as a population.

Worked Example: Homoscedastic Errors with Three Predictor Values

In a hypothetical study of annual equipment-maintenance cost, let $X$ be a site’s activity level, measured in units, with $P(X=0)=0.25$, $P(X=1)=0.50$, and $P(X=2)=0.25$. Suppose the conditional mean cost, in dollars, is $E[Y\mid X]=100+20X$, and the conditional error variance is $400$ dollars squared at every predictor value. Find the marginal response variance and standard deviation. Assume the stated conditional mean and variance describe the full population model.

Method. Apply the law of total variance. The common conditional variance is the error-variance component; variation in the conditional means is the other component.

First calculate the mean and variance of $X$:

E[X]=0(0.25)+1(0.50)+2(0.25)=1, $$ $$ E[X^2]=0^2(0.25)+1^2(0.50)+2^2(0.25)=1.50, \qquad \operatorname{Var}(X)=1.50-1^2=0.50.

The conditional mean is $100+20X$, so its variance is

\operatorname{Var}(E[Y\mid X]) = \operatorname{Var}(100+20X) = 20^2(0.50)=200.

The average conditional variance is $400$, since it is the same at every value of $X$. Therefore,

\operatorname{Var}(Y)=200+400=600\text{ dollars}^2, \qquad \operatorname{SD}(Y)=\sqrt{600}\approx 24.49\text{ dollars}.

The check is direct: the conditional means at $X=0,1,2$ are $100,120,140$, and their population mean is $120$. Their variance is $0.25(100-120)^2+0.50(120-120)^2+0.25(140-120)^2=200$. Adding the conditional variance $400$ again gives $600$. The standard deviation is rounded to two decimal places. A common mistake would be to report $400$ as the response variance; that omits the $200$ dollars-squared contribution from differences in the expected costs across activity levels.

When Conditional Variances Differ

The decomposition remains valid when the conditional variance is not constant. The second term then weights each conditional variance by how common its predictor values are. It is not the variance at an arbitrarily selected value of $X$, nor is it necessarily an unweighted average of variances for groups of unequal sizes.

Worked Example: A Predictor with Unequal Error Variances

A hypothetical portfolio segment has a response $Y$, measured in thousands of dollars, and a binary predictor $X$. Suppose $P(X=0)=0.25$, $P(X=1)=0.75$, with $E[Y\mid X=0]=8$, $E[Y\mid X=1]=14$, $\operatorname{Var}(Y\mid X=0)=4$, and $\operatorname{Var}(Y\mid X=1)=16$, in thousands of dollars squared. Find the marginal mean, both variance components, and the marginal response variance.

Method. Weight conditional means and variances using the given probabilities, then apply the law of total variance. The error variance is not constant in this model.

The marginal mean is

$$ E[Y]=0.25(8)+0.75(14)=2+10.5=12.5. $$

The conditional-mean component is

\operatorname{Var}(E[Y\mid X]) = 0.25(8-12.5)^2+0.75(14-12.5)^2 $$ $$ =0.25(20.25)+0.75(2.25) =5.0625+1.6875=6.75.

The expected conditional variance is a probability-weighted average:

E[\operatorname{Var}(Y\mid X)] = 0.25(4)+0.75(16)=1+12=13.

Thus the marginal response variance is $6.75+13=19.75$ thousands of dollars squared. As a check, $E[Y^2\mid X=0]=8^2+4=68$ and $E[Y^2\mid X=1]=14^2+16=212$. Hence $E[Y^2]=0.25(68)+0.75(212)=176$, and $\operatorname{Var}(Y)=176-(12.5)^2=176-156.25=19.75$. A common mistake is to average the two conditional variances equally, obtaining $10$; the groups are not equally probable, so the correct average is $13$.

Interpreting the Components

The first component, $\operatorname{Var}(E[Y\mid X])$, measures how much the model’s expected response changes as the predictor varies. It is zero if the conditional mean is constant across the predictor distribution. It can be positive even when every response at a given $X$ is close to its conditional mean.

The second component, $E[\operatorname{Var}(Y\mid X)]$, averages the remaining conditional variation. In a homoscedastic regression model it equals $\sigma^2$. With nonconstant conditional variance it averages the different variances, accounting for how often each predictor value occurs. Neither component alone is the full marginal response variance unless the other component is zero.

This identity is a variance accounting rule, not a causal conclusion. A nonzero variation in conditional means describes an association across predictor values under the stated model. The decomposition alone does not establish why those means differ.

Worked Example: Comparing Error Variation with Total Variation

For a hypothetical population, $X$ equals $1$ or $3$, each with probability $0.50$. Suppose $E[Y\mid X]=50+4X$, and the conditional variances are $9$ when $X=1$ and $25$ when $X=3$. Find the marginal response variance and the proportion of it attributable to variation in conditional means.

Method. Compute the variance of the two conditional means, then compute the probability-weighted average of the conditional variances. The conditional variances differ, so do not substitute either one as the overall error component.

The conditional means are $50+4(1)=54$ and $50+4(3)=62$. Their overall mean is $0.50(54)+0.50(62)=58$. Therefore,

\operatorname{Var}(E[Y\mid X]) = 0.50(54-58)^2+0.50(62-58)^2 =0.50(16)+0.50(16)=16.

The average conditional variance is

E[\operatorname{Var}(Y\mid X)] = 0.50(9)+0.50(25)=4.5+12.5=17.

Thus $\operatorname{Var}(Y)=16+17=33$. The share attributable to variation in conditional means is $16/33\approx 0.4848$, or $48.48\%$ to two decimal places. The remaining $17/33\approx 51.52\%$ is the average conditional-variance component. The check $48.48\%+51.52\%=100.00\%$ confirms the partition. A common mistake is to call $9$ or $25$ the response variance; each is conditional on one predictor value, while the marginal variance is $33$.

Common Exam Traps

Equating response variance and error variance. This omits variation in conditional means. Under random $X$ and homoscedasticity, use $\operatorname{Var}(Y)=\beta_1^2\operatorname{Var}(X)+\sigma^2$, not just $\sigma^2$.
Using a conditional variance at one predictor value. With heteroskedasticity, average the conditional variances over the distribution of $X$.
Forgetting group probabilities. The expected conditional variance is weighted by probabilities, not necessarily computed as a simple average of listed groups.
Confusing conditional and marginal questions. A question about responses at a fixed $x$ concerns $\operatorname{Var}(Y\mid X=x)$; one about the whole population concerns $\operatorname{Var}(Y)$.

Key takeaway: Response variance across a population equals variation in the conditional means plus average variation around those means. In a homoscedastic linear model with random $X$, this is $\beta_1^2\operatorname{Var}(X)+\sigma^2$; error variance alone is only the within-predictor component.

Check Your Understanding

For each problem, use the law of total variance and keep the conditional-mean and conditional-variance components separate.

A hypothetical response has $X\sim\operatorname{Bernoulli}(0.4)$, $E[Y\mid X]=30+5X$, and $\operatorname{Var}(Y\mid X)=9$ for both values of $X$. Find $\operatorname{Var}(E[Y\mid X])$ and $\operatorname{Var}(Y)$. (Answer: $6$ and $15$, respectively.)
A predictor takes values $-1$ and $1$ with equal probabilities. The conditional mean is $12+3X$, and the conditional variances at $-1$ and $1$ are $4$ and $12$. Find the marginal response variance. (Answer: $17$.)
A predictor equals $0$ with probability $0.75$ and $2$ with probability $0.25$. The conditional mean is $20+8X$, and the conditional variances at $0$ and $2$ are $5$ and $13$. Find both variance components and the marginal response variance. (Answer: conditional-mean component $48$, average conditional variance $7$, and marginal variance $55$.)
A hypothetical response has $X\sim\operatorname{Bernoulli}(0.2)$, conditional means $10$ when $X=0$ and $25$ when $X=1$, and conditional variance $6$ at both values. Find the marginal variance and the fraction attributable to variation in conditional means. (Answer: marginal variance $42$; fraction $36/42=6/7$, or approximately $0.8571$.)

What You'll Learn