Deriving the Standard Error of a Fitted Mean

Why a Fitted Mean Has a Standard Error

A regression equation gives a fitted conditional mean for a specified predictor profile. That fitted value is calculated using estimated coefficients, however, and those coefficients are uncertain. The standard error of the fitted mean summarizes how that coefficient uncertainty affects the estimated mean at the chosen profile. It does not include the additional variation of a new individual outcome around its conditional mean.

The uncertainty depends on the fitted coefficients together, not just on their separate standard errors. When coefficients are correlated, their covariance contributes to the uncertainty of the fitted mean. The derivation below makes that contribution explicit.

Model and Notation

Suppose the regression model has \(p\) coefficients, including an intercept when the model includes one. For observation \(i\), write the model as \(Y_i=\mathbf{x}_i^{T}\boldsymbol{\beta}+\varepsilon_i\). Here \(\mathbf{x}_i\) is the vector of predictor values for that observation, \(\boldsymbol{\beta}\) is the vector of population regression coefficients, and \(\varepsilon_i\) is the error. For a new predictor profile of interest, let \(\mathbf{x}_0\) be its predictor vector, written in the same order and coding as the model columns.

The fitted conditional mean at that profile is \(\hat{\mu}(\mathbf{x}_0)=\mathbf{x}_0^{T}\hat{\boldsymbol{\beta}}\), where \(\hat{\boldsymbol{\beta}}\) is the estimated coefficient vector. For example, an intercept-and-slope model has \(\mathbf{x}_0=(1,x_0)^T\), so the fitted mean is \(\hat{\beta}_0+x_0\hat{\beta}_1\).

Deriving the Variance Formula

The fitted mean is a linear combination of the estimated coefficients. The variance of a linear combination includes both the variance of each coefficient and the covariances between coefficients. For two estimated coefficients, the expansion is:

The factor of \(2\) appears because the cross-product occurs twice when the matrix product is expanded: once as the covariance of the first coefficient with the second, and once in the reverse order. More generally, let \(V=\operatorname{Cov}(\hat{\boldsymbol{\beta}})\). Expanding the variance of \(\mathbf{x}_0^T\hat{\boldsymbol{\beta}}\) gives the sum of all coefficient variance and covariance terms, which is exactly the quadratic form \(\mathbf{x}_0^T V\mathbf{x}_0\).

This result follows directly from the variance rule for a linear transformation. For a random vector \(Z\) with covariance matrix \(V\), the variance of \(a^TZ\) is \(a^TVa\). Take \(Z=\hat{\boldsymbol{\beta}}\) and \(a=\mathbf{x}_0\). The result is a variance, in squared response units; its square root is the standard error, in response units.

The Usual Covariance Estimate in Linear Regression

To connect the general result to ordinary least squares, write the observations together as \(\mathbf{Y}=X\boldsymbol{\beta}+\boldsymbol{\varepsilon}\). The columns of \(X\) are the model predictors, including a column of ones if there is an intercept. Assume the conditional mean is correctly represented by this model, \(X\) has full column rank, and, conditional on \(X\), the errors have mean zero and covariance matrix \(\sigma^2 I\). Thus the errors are uncorrelated and have a common variance. Treating \(X\) as fixed, the ordinary least-squares estimator is \(\hat{\boldsymbol{\beta}}=(X^TX)^{-1}X^T\mathbf{Y}\).

Under these assumptions, its covariance matrix follows by applying the covariance rule for a linear transformation. The calculation is:

In practice, \(\sigma^2\) is unknown. Under this model, its usual estimate is the residual mean square \(s^2=\mathrm{SSE}/(n-p)\), where \(\mathrm{SSE}\) is the sum of squared residuals and \(n-p\) is the residual degrees of freedom. Therefore the estimated coefficient covariance matrix and fitted-mean standard error are:

The variance derivation uses the stated mean and covariance assumptions; normal errors are not needed to derive this variance formula. If the model assumptions are not suitable, the usual covariance estimate may not describe coefficient uncertainty correctly. The coefficient covariance matrix should be taken from an analysis whose assumptions match the situation.

Using the Matrix Formula in Multiple Regression

The same calculation applies when a fitted mean uses several predictors. The vector \(\mathbf{x}_0\) must include every model column, with the correct coding. Each diagonal entry of \(\hat{V}\) contributes a coefficient variance, while every off-diagonal covariance contributes a cross-term. In general, a covariance can either increase or decrease the total variance, depending on its sign and on the entries of \(\mathbf{x}_0\).

Calculating From the Simple-Regression Covariance Estimate

In simple regression, the general matrix formula can be evaluated using familiar summaries. If \(x\) is the predictor, let \(\bar{x}\) be its sample mean and \(S_{xx}=\sum_{i=1}^n(x_i-\bar{x})^2\). With an intercept, the estimated coefficient covariance matrix is:

Multiplying on both sides by \((1,x_0)^T\) and its transpose gives an equivalent expression:

This identity is an algebraic rearrangement of the matrix formula: the terms involving \(\bar{x}\) combine with the \(x_0^2/S_{xx}\) term to form \((x_0-\bar{x})^2/S_{xx}\). The expression remains an estimated variance, so its square root is the estimated standard error.

Common Mistake: Dropping the Covariance Terms

A tempting but generally incorrect calculation is to combine coefficient standard errors as if the estimates were independent. For a two-coefficient fitted mean, this amounts to using only \(a^2\operatorname{Var}(\hat{\beta}_0)+b^2\operatorname{Var}(\hat{\beta}_1)\). The correct formula also contains \(2ab\operatorname{Cov}(\hat{\beta}_0,\hat{\beta}_1)\). Coefficient estimates from a regression are often correlated, so the covariance is not automatically zero.

Another error is to report the variance itself as the standard error. The matrix product \(\mathbf{x}_0^T\hat{V}\mathbf{x}_0\) has squared response units; take its nonnegative square root to obtain an \(SE\) in response units. Finally, this is uncertainty in an estimated conditional mean. It is not the standard deviation of individual outcomes at the profile, which also includes their outcome-to-outcome variation around that mean.