Least Squares Risk Under Gaussian Errors

Why the Design Matters for Risk

In a linear model, ordinary least squares (OLS) estimates a vector of regression coefficients from a response vector. Its errors are not summarized by a single variance: uncertainty in each coefficient, and the resulting uncertainty in fitted means, depend on the design matrix. A risk calculation makes that dependence explicit and distinguishes two targets that are easy to confuse: the regression mean and a future noisy response.

Throughout, treat the design matrix as fixed. Let there be \(n\) observations and \(p\) coefficients, with \(n \ge p\). Assume the columns of the design matrix are linearly independent. The model is

Here \(\mathbf Y\) is the \(n\)-vector of responses, \(\mathbf X\) is the fixed \(n\times p\) design matrix, and \(\boldsymbol\beta\) is the unknown coefficient vector. The error assumption says that the errors have mean zero, common variance \(\sigma^2\), and zero covariance between distinct observations; their joint distribution is multivariate normal. Full column rank ensures that \(\mathbf X^\mathsf T\mathbf X\) is invertible.

OLS Bias and Covariance

The least-squares estimator minimizes \(\|\mathbf Y-\mathbf X\mathbf a\|^2\) over coefficient vectors \(\mathbf a\). Under full column rank, its unique solution is

The following result gives its distributional properties. These properties are conditional on the fixed design; no averaging over a random design is involved.

Proof. Put \(\mathbf A=(\mathbf X^\mathsf T\mathbf X)^{-1}\mathbf X^\mathsf T\). Since \(\mathbf A\mathbf X=\mathbf I_p\), substituting the model into the OLS formula gives

Taking expectations and using \(E[\boldsymbol\varepsilon]=\mathbf 0\) yields \(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\), so the bias is zero. Applying the covariance rule for a fixed linear transformation gives

Finally, \(\hat{\boldsymbol\beta}\) is a fixed linear transformation of a multivariate normal vector, plus the fixed vector \(\boldsymbol\beta\). Its mean and covariance, just calculated, therefore give the stated normal distribution. \(\square\)

Expected Coefficient and In-Sample Prediction Loss

Write \(\mathbf e=\hat{\boldsymbol\beta}-\boldsymbol\beta\). Because the estimator is unbiased, \(E[\mathbf e]=\mathbf 0\). For any zero-mean random vector with finite second moments, the expected squared Euclidean norm is the trace of its covariance matrix: \(\|\mathbf e\|^2=\operatorname{tr}(\mathbf e\mathbf e^\mathsf T)\), so taking expectations gives \(E[\|\mathbf e\|^2]=\operatorname{tr}(\operatorname{Cov}(\mathbf e))\). Applying this identity to the covariance above gives the coefficient risk.

The in-sample error in fitted regression means is \(\mathbf X\mathbf e\), so its squared norm is \(\mathbf e^\mathsf T\mathbf X^\mathsf T\mathbf X\mathbf e\). Taking expectations and using the trace identity for a quadratic form gives

Equivalently, the matrix \(\mathbf P_X=\mathbf X(\mathbf X^\mathsf T\mathbf X)^{-1}\mathbf X^\mathsf T\) is the projection matrix onto the column space of \(\mathbf X\), and its trace is \(p\). Thus the expected total squared error in fitted means is \(p\sigma^2\), and the average across the \(n\) design points is \(p\sigma^2/n\). This total risk does not depend on the particular full-rank design, although the coefficient risk generally does.

To see the distinction, the error in estimating the regression mean at \(\mathbf x_0\) is \(\mathbf x_0^\mathsf T(\hat{\boldsymbol\beta}-\boldsymbol\beta)\). Its mean is zero, and its variance is the displayed quadratic form. For a future response, the prediction error is \(\mathbf x_0^\mathsf T(\hat{\boldsymbol\beta}-\boldsymbol\beta)-\varepsilon_0\). Independence makes the covariance term zero, and the two variances add. The extra \(\sigma^2\) is irreducible response noise, not uncertainty in the fitted mean.

Worked Risk Calculations

Interpretation and a Common Pitfall

The coefficient-risk formula depends on \((\mathbf X^\mathsf T\mathbf X)^{-1}\): a design that provides less information in some coefficient direction can make squared coefficient error larger. The in-sample total prediction risk has a simpler form, \(p\sigma^2\), because the fitted values vary only in the \(p\)-dimensional column space of the design. At a particular new covariate vector, however, uncertainty depends on its position relative to the design, through \(\mathbf x_0^\mathsf T(\mathbf X^\mathsf T\mathbf X)^{-1}\mathbf x_0\).

Common mistake: treating residual error as prediction error for the regression mean. The residual vector is \(\mathbf Y-\mathbf X\hat{\boldsymbol\beta}\); it compares observed responses, which contain noise, with fitted means. It is not the vector \(\mathbf X\hat{\boldsymbol\beta}-\mathbf X\boldsymbol\beta\), because the true mean vector is not observed directly. Nor should the expected in-sample squared error \(p\sigma^2\) be used as the risk for predicting a new response at any chosen covariate. For a new response, add the independent noise variance; for a particular mean prediction, use the corresponding quadratic form.