Why Regression Inference Needs Error Assumptions
Least-squares regression produces coefficient estimates from observed responses and predictors. To describe how those estimates would vary across repeated samples, we must specify what is random and how the errors behave. The central perspective here is conditional on the observed predictor values: treat the design matrix as fixed, and let the responses vary because of random errors.
These assumptions are distinct from assumptions about the distribution of the predictors. Predictors do not need to be normally distributed for the conditional sampling results below. They do need to provide enough independent information to estimate the chosen coefficients.
In a simple linear regression with an intercept, each row of \(X\) is \((1,x_i)\), and \(\beta=(\beta_0,\beta_1)^\mathsf{T}\). In multiple regression, rows can also contain several predictors, indicators, or interaction terms. The same matrix results apply as long as \(X\) has full column rank.
Assumptions and What They Support
The standard conditional model for coefficient inference assumes that the errors have conditional mean zero and a common variance, and are uncorrelated across observations. The normal-error version adds a distributional assumption.
- Zero conditional mean: \(E[\varepsilon\mid X]=0\).
- Constant conditional variance: \(\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I_n\), where \(\sigma^2>0\) and \(I_n\) is the \(n\) by \(n\) identity matrix. This states both equal error variances and zero covariances between distinct errors.
- Normal errors, when exact normal sampling distributions are required: \(\varepsilon\mid X\sim N(0,\sigma^2 I_n)\).
Zero conditional mean makes least squares unbiased for \(\beta\). Constant variance and zero error covariances give the usual coefficient covariance matrix. Normality is a stronger assumption: it gives an exact multivariate normal sampling distribution in finite samples. Without normality, zero mean and constant variance still give the same expectation and covariance, but do not by themselves make the estimator normally distributed.
If error variances differ or errors are correlated, the zero-mean assumption can still ensure unbiasedness, but the usual covariance formula generally no longer applies. For a general conditional covariance matrix \(\Omega=\operatorname{Var}(\varepsilon\mid X)\), the covariance of the least-squares estimator is \((X^\mathsf{T}X)^{-1}X^\mathsf{T}\Omega X(X^\mathsf{T}X)^{-1}\). This formula describes the variability of ordinary least squares under that error structure; it does not assert that ordinary least squares is the most efficient estimator.
Deriving the Conditional Sampling Results
The results follow directly by substituting the regression model into the least-squares formula. This derivation also clarifies why the predictor values are conditioned on: once \(X\) is fixed, the estimator is a linear transformation of the errors.
\(\hat{\beta}=(X^\mathsf{T}X)^{-1}X^\mathsf{T}(X\beta+\varepsilon)=\beta+(X^\mathsf{T}X)^{-1}X^\mathsf{T}\varepsilon\), since \((X^\mathsf{T}X)^{-1}X^\mathsf{T}X=I_p\).
\(E[\hat{\beta}\mid X]=\beta+(X^\mathsf{T}X)^{-1}X^\mathsf{T}E[\varepsilon\mid X]=\beta\), using the zero conditional mean assumption.
For any fixed matrix \(A\), \(\operatorname{Var}(A\varepsilon\mid X)=A\operatorname{Var}(\varepsilon\mid X)A^\mathsf{T}\). Taking \(A=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\) gives the general covariance formula. Under \(\operatorname{Var}(\varepsilon\mid X)=\sigma^2 I_n\), it simplifies to \(\sigma^2(X^\mathsf{T}X)^{-1}\).
A fixed linear transformation of a multivariate normal vector is multivariate normal. Thus, if \(\varepsilon\mid X\sim N(0,\sigma^2 I_n)\), then \(\hat{\beta}\mid X\sim N_p\bigl(\beta,\sigma^2(X^\mathsf{T}X)^{-1}\bigr)\).
Under the normal-error assumption, the first formula is the covariance matrix of an exact conditional multivariate normal distribution. Its diagonal entries are the variances of the individual coefficient estimates; its off-diagonal entries are covariances between estimates. The estimates need not be independent merely because the errors are independent: their covariance also depends on the geometry of the predictors in \(X\).
Worked Example: Conditional Distribution With Equal Error Variance
In a hypothetical portfolio study, a response is modeled by a simple linear regression on four fixed predictor values \(x=0,1,2,3\). Suppose the true coefficients are \(\beta_0=10\) and \(\beta_1=2\), and the errors, conditional on these predictors, are independent \(N(0,4)\). Find the conditional distribution, expectation, and covariance matrix of the intercept and slope estimates.
Method. Use the normal-error result with the design matrix \(X\) having rows \((1,x_i)\). The given error distribution has mean zero, common variance \(\sigma^2=4\), and independent normal errors, so all assumptions for the exact conditional normal distribution hold.
First compute the predictor mean and sum of squared deviations: \(\bar{x}=(0+1+2+3)/4=6/4=3/2\), and \(S_{xx}=\sum_i(x_i-\bar{x})^2=9/4+1/4+1/4+9/4=20/4=5\). For an intercept-and-slope model, the inverse cross-product matrix is \((X^\mathsf{T}X)^{-1}=\begin{pmatrix}1/4+\bar{x}^2/S_{xx}&-\bar{x}/S_{xx}\\-\bar{x}/S_{xx}&1/S_{xx}\end{pmatrix}\).
Substitute \(\bar{x}=3/2\) and \(S_{xx}=5\): \(1/4+(3/2)^2/5=1/4+9/20=5/20+9/20=14/20=7/10\); the off-diagonal entry is \(-(3/2)/5=-3/10\); and the slope entry is \(1/5\). Therefore \((X^\mathsf{T}X)^{-1}=\begin{pmatrix}7/10&-3/10\\-3/10&1/5\end{pmatrix}\).
Multiplying by \(\sigma^2=4\) gives \(\operatorname{Var}(\hat{\beta}\mid X)=\begin{pmatrix}14/5&-6/5\\-6/5&4/5\end{pmatrix}\). The conditional expectation is \((10,2)^\mathsf{T}\), so the full result is \(\hat{\beta}\mid X\sim N_2\left(\begin{pmatrix}10\\2\end{pmatrix},\begin{pmatrix}14/5&-6/5\\-6/5&4/5\end{pmatrix}\right)\).
Check. The slope variance is \(4/S_{xx}=4/5\); the intercept-slope covariance is \(-\bar{x}(4/S_{xx})=-(3/2)(4/5)=-6/5\). Both agree with the matrix. The negative covariance is plausible because, for a fixed fitted line around the observed predictor center, a larger intercept estimate tends to pair with a smaller slope estimate.
Common mistake avoided. The variance matrix is not just \((X^\mathsf{T}X)^{-1}\); it must be multiplied by the error variance \(4\). Also, the second parameter in \(N_2\) is the covariance matrix, not a vector of standard deviations.
Unequal Variances Change the Covariance Formula
When errors are independent but their variances differ, their covariance matrix is diagonal but not \(\sigma^2 I_n\). The least-squares estimate can remain conditionally unbiased when the errors have mean zero, but its covariance must account for each observation's variance. The next calculation illustrates why substituting an average variance into the equal-variance formula can give a wrong answer.
Worked Example: Unequal Error Variances in a Three-Point Regression
Suppose a hypothetical model has fixed predictor values \(x=0,1,2\), with independent normal errors having variances \(1,1,4\), respectively. The true coefficients are \(\beta_0=6\) and \(\beta_1=3\). Calculate the conditional expectation and covariance matrix of the least-squares intercept and slope estimates.
Method. The errors have conditional mean zero, so the estimator is unbiased. They are independent and normal, but their variances are unequal; therefore use the general covariance formula with \(\Omega=\operatorname{diag}(1,1,4)\). The error distribution is \(\varepsilon\mid X\sim N(0,\Omega)\).
With rows \((1,0),(1,1),(1,2)\), the design cross-product is \(X^\mathsf{T}X=\begin{pmatrix}3&3\\3&5\end{pmatrix}\), whose inverse is \(A=(X^\mathsf{T}X)^{-1}=\frac16\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\). The weighted cross-product is \(X^\mathsf{T}\Omega X=\begin{pmatrix}1+1+4&0+1+8\\0+1+8&0+1+16\end{pmatrix}=\begin{pmatrix}6&9\\9&17\end{pmatrix}\).
Now multiply in order. First, \((X^\mathsf{T}\Omega X)A=\frac16\begin{pmatrix}6&9\\9&17\end{pmatrix}\begin{pmatrix}5&-3\\-3&3\end{pmatrix}=\begin{pmatrix}1/2&3/2\\-1&4\end{pmatrix}\). Then \(A(X^\mathsf{T}\Omega X)A=\frac16\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\begin{pmatrix}1/2&3/2\\-1&4\end{pmatrix}=\begin{pmatrix}11/12&-3/4\\-3/4&5/4\end{pmatrix}\).
Therefore \(E[\hat{\beta}\mid X]=(6,3)^\mathsf{T}\), and the conditional covariance matrix is \(\begin{pmatrix}11/12&-3/4\\-3/4&5/4\end{pmatrix}\). Because the errors are normal and \(\hat{\beta}\) is a linear transformation of them, the estimator has an exact conditional bivariate normal distribution with this mean and covariance matrix.
Check. The covariance matrix is symmetric, as a covariance matrix must be. Its diagonal entries are positive, and its determinant is \((11/12)(5/4)-(3/4)^2=55/48-27/48=28/48=7/12>0\), consistent with a positive-definite covariance matrix.
Common mistake avoided. Applying \(\sigma^2(X^\mathsf{T}X)^{-1}\) assumes one common variance for every error. Here the variances are \(1,1,4\), so that assumption fails; the calculation must retain \(\Omega\) on both sides of the design matrix.
What Changes Without Normal Errors?
Normality is not needed for the conditional expectation or covariance derivations. It is needed for the exact finite-sample normal sampling distribution. A useful way to see the distinction is to use errors that are independent and centered but take only two possible values.
Worked Example: Correct Moments but a Non-Normal Sampling Distribution
Let the fixed predictor values be \(x=0,1,2\), and let the true coefficients be \(\beta_0=5\) and \(\beta_1=2\). Suppose the errors are independent, with each error equal to \(+1\) or \(-1\) with probability \(1/2\). Find the conditional expectation and covariance matrix of the least-squares estimates, and decide whether their joint distribution is normal.
Method. Each error has mean \(0\) and variance \(1\): \(E[\varepsilon_i]=(+1)(1/2)+(-1)(1/2)=0\), and \(E[\varepsilon_i^2]=1\). The errors are independent, so \(\operatorname{Var}(\varepsilon\mid X)=I_3\). Use the equal-variance moment formula, but do not use the normal-distribution conclusion.
For this design, \((X^\mathsf{T}X)^{-1}=\frac16\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\). Thus \(E[\hat{\beta}\mid X]=(5,2)^\mathsf{T}\) and \(\operatorname{Var}(\hat{\beta}\mid X)=\frac16\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\).
To check the form of the sampling distribution, the estimation error is \(\hat{\beta}-\beta=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\varepsilon\). The intercept and slope error weights on \((\varepsilon_0,\varepsilon_1,\varepsilon_2)\) are respectively \((5/6,1/3,-1/6)\) and \((-1/2,0,1/2)\). For example, the slope error can take only \(-1,0,1\), with probabilities \(1/4,1/2,1/4\), since it equals \((-\varepsilon_0+\varepsilon_2)/2\). A normal variable does not have this finite set of possible values. Consequently the coefficient vector is not jointly normal.
Check. The slope variance is \((1/4)(1)+(1/4)(1)=1/2\), matching the lower-right matrix entry \(3/6\). The intercept variance is \(25/36+1/9+1/36=25/36+4/36+1/36=30/36=5/6\), matching \(5/6\). The covariance is \((5/6)(-1/2)+(1/3)(0)+(-1/6)(1/2)=-5/12-1/12=-1/2\), matching \(-3/6\).
Common mistake avoided. Correctly calculating the covariance does not establish normality. Here the zero-mean, equal-variance conditions give the same moment formulas, while the discrete error distribution prevents the exact normal conclusion.
Conditions, Scope, and Common Traps
The rank condition matters because \((X^\mathsf{T}X)^{-1}\) exists only when the columns of \(X\) are linearly independent. If the design is rank deficient, the stated unique least-squares coefficient vector and its covariance formula do not apply in this form. The fitted values may still be identifiable, but the coefficient representation is not unique.
The conditional perspective also prevents several common misinterpretations:
- Predictors need not be normal. The results condition on their observed values. A binary indicator or a fixed interaction column is compatible with these formulas.
- Zero mean and normality are different assumptions. Zero conditional mean supports unbiasedness; normal errors support the exact normal sampling distribution.
- Equal variance is not automatic. If the error variances differ, the usual \(\sigma^2(X^\mathsf{T}X)^{-1}\) covariance is not justified.
- Uncorrelated errors are not the same as independent errors. Independence is a sufficient condition for zero covariance, but the covariance calculations require the specified covariance structure. The exact normal result uses a multivariate normal error assumption.
- Conditioning is not the same as claiming predictors are inherently fixed. Even if predictors were sampled, these formulas describe the distribution conditional on the realized design, subject to the model assumptions.
In applications, these assumptions are a framework for interpreting coefficient uncertainty, not a guarantee that a chosen model is appropriate. For example, an omitted systematic pattern could make \(E[\varepsilon\mid X]\ne0\), undermining unbiasedness. Dependence or changing error variance affects the covariance calculation. Those issues must be considered before relying on standard coefficient inference.
Check Your Understanding
For each problem, condition on the stated predictor values and identify which assumptions justify your result.
- A simple regression has fixed predictor values \(x=1,3\). Errors are independent \(N(0,9)\), and the true coefficient vector is \((4,2)^\mathsf{T}\). Find the conditional expectation and covariance matrix of \(\hat{\beta}\). (Answer: \(E[\hat{\beta}\mid X]=(4,2)^\mathsf{T}\), \(\operatorname{Var}(\hat{\beta}\mid X)=\begin{pmatrix}45/2&-9\\-9&9/2\end{pmatrix}\), with an exact bivariate normal distribution.)
- For a simple regression with fixed \(x=0,1,2\), errors are independent, have mean zero, and have variances \(2,2,2\). Find the conditional covariance matrix. (Answer: \(\begin{pmatrix}5/3&-1\\-1&1\end{pmatrix}\).)
- For fixed \(x=0,1,2\), errors are independent with mean zero and variance one, but are not assumed normal. What are the expectation and covariance of the coefficient estimates, and can exact normality be concluded? (Answer: \(E[\hat{\beta}\mid X]=\beta\), \(\operatorname{Var}(\hat{\beta}\mid X)=\frac16\begin{pmatrix}5&-3\\-3&3\end{pmatrix}\); exact normality cannot be concluded.)
- A regression design has full column rank, and \(E[\varepsilon\mid X]=0\), but the errors have correlated components with covariance \(\Omega\ne\sigma^2I_n\). State the conditional expectation and covariance of the least-squares estimate. (Answer: \(E[\hat{\beta}\mid X]=\beta\), \(\operatorname{Var}(\hat{\beta}\mid X)=(X^\mathsf{T}X)^{-1}X^\mathsf{T}\Omega X(X^\mathsf{T}X)^{-1}\).)
- Suppose two predictor columns in a proposed design are exact multiples of each other. Which condition required for the standard coefficient sampling formulas fails, and what consequence follows? (Answer: Full column rank fails, so \(X^\mathsf{T}X\) is not invertible and the stated unique coefficient estimator and covariance formula do not apply.)