Exact Sampling Distributions Of Regression Coefficients

Why Regression Coefficients Have Sampling Distributions

A least-squares slope and intercept are calculated from observed data, so their values change from one sample to another. In regression inference, the predictor values can be treated as fixed while the responses vary because of random errors. This makes it possible to describe how the estimated coefficients vary across repeated samples with the same predictor values.

The focus here is the exact conditional sampling distribution of the least-squares slope and intercept. “Conditional” means that the predictor values are held fixed. The normal distributions derived below require normally distributed errors; they are not merely large-sample approximations.

Model and assumptions: For fixed predictor values $x_1,\ldots,x_n$, suppose $Y_i=\beta_0+\beta_1x_i+\varepsilon_i$, where the errors $\varepsilon_1,\ldots,\varepsilon_n$ are independent and each has distribution $N(0,\sigma^2)$. The parameters $\beta_0,\beta_1$ and $\sigma^2$ are the same for all observations. Assume $n\geq 2$ and $S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2>0$, so the slope is identifiable. The predictor values are fixed, not random.

Write $b_0$ and $b_1$ for the least-squares estimates of the intercept and slope. Using the usual simple-regression least-squares formulas, $b_1=\sum (x_i-\bar{x})Y_i/S_{xx}$ and $b_0=\bar{Y}-b_1\bar{x}$. The exact sampling distributions follow by expressing these estimates as linear combinations of the random responses.

Exact Distributions of the Slope and Intercept

Theorem: Under the fixed-design model and assumptions above, conditional on $x_1,\ldots,x_n$, $$ b_1\mid x_1,\ldots,x_n\sim N\left(\beta_1,\frac{\sigma^2}{S_{xx}}\right), \qquad b_0\mid x_1,\ldots,x_n\sim N\left(\beta_0,\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{S_{xx}}\right)\right). $$ These are marginal distributions: each describes one coefficient by itself. The variance parameter is the true error variance $\sigma^2$, not an estimate substituted for it.

The conditional means show that both least-squares coefficients are unbiased under this model: their expected values, given the predictor values, are the corresponding regression parameters. The variances describe their sampling uncertainty. In particular, the slope variance depends on the spread $S_{xx}$ of the fixed predictor values. The intercept variance also depends on $\bar{x}$, because the intercept represents the fitted response at $x=0$.

Why the Formulas Hold

Substitute the model equation into the least-squares formula for $b_1$. Since $\sum (x_i-\bar{x})=0$, the intercept term drops out:

b_1 =\frac{\sum (x_i-\bar{x})(\beta_0+\beta_1x_i+\varepsilon_i)}{S_{xx}} =\beta_1+\frac{\sum (x_i-\bar{x})\varepsilon_i}{S_{xx}}.

The errors have conditional means zero, so $E[b_1\mid x_1,\ldots,x_n]=\beta_1$. Independence gives the variance of the weighted error sum as the sum of the individual variances:

\operatorname{Var}(b_1\mid x_1,\ldots,x_n) =\frac{\sum (x_i-\bar{x})^2\sigma^2}{S_{xx}^2} =\frac{\sigma^2}{S_{xx}}.

Because $b_1$ is a linear combination of independent normal errors plus a constant, it is normally distributed. This establishes the slope’s exact conditional distribution, not only its mean and variance.

For the intercept, use $b_0=\bar{Y}-\bar{x}b_1$. It is a linear combination of the responses, and hence a linear combination of normal errors plus a constant. More explicitly, its error coefficients are $c_i=1/n-\bar{x}(x_i-\bar{x})/S_{xx}$, so $b_0=\beta_0+\sum c_i\varepsilon_i$. The coefficients sum to one in the response representation, giving the stated mean. For the variance, expand the sum of squared error coefficients. The cross-term is zero because $\sum(x_i-\bar{x})=0$:

\sum_{i=1}^n c_i^2 =\sum_{i=1}^n\left(\frac{1}{n}-\frac{\bar{x}(x_i-\bar{x})}{S_{xx}}\right)^2 =\frac{1}{n}+\frac{\bar{x}^2}{S_{xx}}.

Multiplying this sum by the common error variance $\sigma^2$ gives the intercept variance. Normality follows because $b_0$ is also a linear combination of independent normal errors. This argument does not require the observed responses to equal their fitted values; it describes the repeated-sample variation under the model.

Using the Sampling Distributions

For probability calculations, first find $\bar{x}$ and $S_{xx}$, then substitute the model parameters and error variance into the appropriate distribution. If a coefficient $B$ has distribution $N(\mu,\tau^2)$, its standard deviation is $\tau$, the square root of the variance. Standardizing gives $Z=(B-\mu)/\tau\sim N(0,1)$. Keep the variance and standard deviation distinct: the second parameter in $N(\mu,\tau^2)$ is the variance.

Worked Example: Finding Both Coefficient Distributions

Suppose four fixed predictor values are $0,1,2,3$, and the response model is $Y_i=2+3x_i+\varepsilon_i$, with independent errors $\varepsilon_i\sim N(0,4)$. Find the exact conditional distributions of $b_0$ and $b_1$. Method: calculate the design summaries and substitute into the theorem. The calculation avoids treating the fitted coefficients as fixed across repeated samples.

Step 1: Find the predictor mean and spread. Here $n=4$, and $\bar{x}=(0+1+2+3)/4=1.5$. Therefore $S_{xx}=(0-1.5)^2+(1-1.5)^2+(2-1.5)^2+(3-1.5)^2=2.25+0.25+0.25+2.25=5$.

Step 2: Find the slope distribution. The model has $\beta_1=3$ and $\sigma^2=4$, so $\operatorname{Var}(b_1\mid x)=4/5=0.8$. Thus $b_1\mid x\sim N(3,0.8)$, with standard deviation $\sqrt{0.8}\approx0.8944$.

Step 3: Find the intercept distribution. The model has $\beta_0=2$, and $\operatorname{Var}(b_0\mid x)=4(1/4+1.5^2/5)=4(0.25+0.45)=2.8$. Thus $b_0\mid x\sim N(2,2.8)$, with standard deviation $\sqrt{2.8}\approx1.6733$.

Check. Both distributions have means equal to the stated model coefficients. The slope variance is positive, and the intercept variance includes both $1/n$ and the nonzero-centroid contribution $\bar{x}^2/S_{xx}$.

Worked Example: Calculating a Slope Probability

A hypothetical study uses fixed predictor values $1,2,3,4$. Its model is $Y_i=6-x_i+\varepsilon_i$, where the errors are independent $N(0,5)$. Find $P(b_1>0\mid x_1,\ldots,x_4)$. Method: obtain the exact slope distribution and standardize the event. The common mistake to avoid is using the intercept’s variance for the slope.

Step 1: Calculate $S_{xx}$. The predictor mean is $\bar{x}=(1+2+3+4)/4=2.5$. Hence $S_{xx}=(-1.5)^2+(-0.5)^2+(0.5)^2+(1.5)^2=2.25+0.25+0.25+2.25=5$.

Step 2: Write the slope distribution. Here $\beta_1=-1$ and $\sigma^2=5$, so $\operatorname{Var}(b_1\mid x)=5/5=1$. Therefore $b_1\mid x\sim N(-1,1)$, with standard deviation $1$.

Step 3: Standardize the required event. $P(b_1>0\mid x)=P((b_1-(-1))/1>(0-(-1))/1)=P(Z>1)$. From the standard normal distribution, $P(Z>1)=0.1587$ to four decimal places.

Check. The threshold zero is one standard deviation above the mean $-1$, so a probability below $0.5$ is consistent with the distribution’s location.

How Predictor Location Affects Intercept Uncertainty

The slope measures the response change per unit change in $x$, so adding the same constant to every predictor value leaves $S_{xx}$ unchanged and does not alter the slope variance. The intercept, however, is the response at $x=0$. If the observed predictors are far from zero, estimating that response requires extrapolation, reflected in the additional $\bar{x}^2/S_{xx}$ term in its variance.

Worked Example: Comparing Centered and Shifted Predictors

Consider the same hypothetical response model $Y_i=4+0.6x_i+\varepsilon_i$ with independent errors $\varepsilon_i\sim N(0,9)$. Compare coefficient variances for the fixed predictor sets $\{-2,-1,0,1,2\}$ and $\{8,9,10,11,12\}$. Method: calculate the mean and spread for each design. This avoids assuming that shifting the predictor changes both coefficient variances in the same way.

Centered design. For $\{-2,-1,0,1,2\}$, $n=5$, $\bar{x}=0$, and $S_{xx}=4+1+0+1+4=10$. Therefore $\operatorname{Var}(b_1\mid x)=9/10=0.9$ and $\operatorname{Var}(b_0\mid x)=9(1/5+0^2/10)=9(0.2)=1.8$. The distributions are $b_1\mid x\sim N(0.6,0.9)$ and $b_0\mid x\sim N(4,1.8)$.

Shifted design. For $\{8,9,10,11,12\}$, $n=5$, $\bar{x}=10$, and $S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=10$. Thus $\operatorname{Var}(b_1\mid x)=9/10=0.9$, while $\operatorname{Var}(b_0\mid x)=9(1/5+10^2/10)=9(0.2+10)=91.8$. The distributions are $b_1\mid x\sim N(0.6,0.9)$ and $b_0\mid x\sim N(4,91.8)$.

Check. The shift leaves $S_{xx}$, and therefore the slope variance, unchanged. It makes the intercept variance much larger because $x=0$ lies far from the observed predictor values.

Conditions and Common Exam Traps

The normal sampling distributions are exact only under the stated model, conditional on the fixed predictor values. If the errors are independent with mean zero and common variance but are not normal, the coefficient mean and variance formulas still hold, but exact normality does not follow. A normal approximation might be considered in an appropriate large-sample setting, but that is a different claim. If error variances differ or errors are dependent, the variance formulas derived here generally do not apply as written.

Using a variance as a standard deviation. For $N(\mu,\tau^2)$, standardize by $\tau$, not $\tau^2$.
Omitting the intercept’s location term. The intercept variance is $\sigma^2(1/n+\bar{x}^2/S_{xx})$, not just $\sigma^2/n$ unless $\bar{x}=0$.
Using the wrong spread. The slope variance uses $S_{xx}=\sum(x_i-\bar{x})^2$, not $\sum x_i^2$ in general.
Assuming exact normality without normal errors. Normal errors are a model assumption for these exact distributions, not a consequence of least squares alone.
Forgetting the fixed-design condition. These statements describe coefficient distributions conditional on the stated predictor values; they do not require a probability model for the predictors.

Key takeaway: With fixed predictor values and independent $N(0,\sigma^2)$ errors, the least-squares slope and intercept are exact normal random variables. Their means are $\beta_1$ and $\beta_0$; their variances are $\sigma^2/S_{xx}$ and $\sigma^2(1/n+\bar{x}^2/S_{xx})$, respectively.

Check Your Understanding

For each problem, condition on the stated fixed predictor values. Use the exact distributions and keep the variance as the second parameter of a normal distribution.

A model has $Y_i=1+3x_i+\varepsilon_i$, with independent errors $N(0,4)$, and fixed predictors $0,2,4$. Find the conditional distributions of $b_0$ and $b_1$. (Answer: $b_1\mid x\sim N(3,0.5)$, $b_0\mid x\sim N(1,10/3)$.)
For fixed predictors $2,4,6,8$, suppose $Y_i=5+0.4x_i+\varepsilon_i$, with independent errors $N(0,5)$. Find $P(b_1>1.4\mid x)$. (Answer: $0.0228$.)
For fixed predictors $1,2,3,4,5$, the model is $Y_i=-2+1.5x_i+\varepsilon_i$, with independent errors $N(0,4)$. Give both coefficient distributions. (Answer: $b_1\mid x\sim N(1.5,0.4)$, $b_0\mid x\sim N(-2,4.4)$.)
Suppose errors are independent with mean zero and common variance $\sigma^2$, but are not normally distributed. Which statements remain justified: the coefficient mean and variance formulas, or the exact normal distributions? (Answer: the mean and variance formulas remain justified; exact normality is not.)
For $n=5$ and $\sigma^2=4$, compare the coefficient variances for fixed predictors $0,1,2,3,4$ and $5,6,7,8,9$. (Answer: slope variance $0.4$ for each; intercept variances $2.4$ and $20.4$, respectively.)

What You'll Learn