Weighted Least Squares With Known Variances

Why Weight Observations?

Ordinary least squares gives each observation the same role in minimizing squared residuals. That is appropriate when observations have the same error variance, but not when some measurements are known to be much less precise than others. If the variance pattern is known, weighted least squares (WLS) accounts for it by giving less weight to observations with greater variance.

The weights affect how the fitted line is chosen; they do not change what a regression coefficient means. For example, a slope still describes the change in the model’s mean response associated with a one-unit increase in the predictor, with other predictors held fixed. Its interpretation is an association in the fitted mean model, not by itself a causal effect.

The Model and the Weighting Rule

For \(n\) observations, write the linear regression model as \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}\). Here \(\mathbf{y}\) is the response vector, \(\mathbf{X}\) is the design matrix, and \(\boldsymbol{\beta}\) is the vector of regression coefficients. Assume the errors are independent, have mean zero, and have known variances \(\operatorname{Var}(\varepsilon_i)=\sigma_i^2\). Thus the variance may differ across observations. The columns of \(\mathbf{X}\) must be linearly independent so the coefficients can be estimated uniquely. Normal errors are an additional assumption when using the exact normal-based confidence intervals below.

The weighting rule follows from measuring residuals in units of their standard deviations. An observation with variance \(\sigma_i^2\) has standard deviation \(\sigma_i\), so its standardized residual is \(e_i/\sigma_i\), where \(e_i=y_i-\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta}\). Squaring that standardized residual gives \(e_i^2/\sigma_i^2\). WLS therefore minimizes

To obtain the estimator, differentiate this quadratic with respect to \(\boldsymbol{\beta}\) and set the derivative equal to zero:

Rearranging gives the weighted normal equations, \(\mathbf{X}^{\mathsf T}\mathbf{W}\mathbf{X}\hat{\boldsymbol{\beta}}=\mathbf{X}^{\mathsf T}\mathbf{W}\mathbf{y}\). The full-rank assumption makes \(\mathbf{X}^{\mathsf T}\mathbf{W}\mathbf{X}\) invertible, yielding the estimator in the definition. Equivalently, WLS is ordinary least squares applied to the response and design rows after each has been multiplied by \(\sqrt{w_i}=1/\sigma_i\).

The covariance result can be checked directly. The estimator is a linear transformation of \(\mathbf{y}\), so its covariance is

because \(\operatorname{Var}(\mathbf{y})=\mathbf{W}^{-1}\), and \(\mathbf{W}\mathbf{W}^{-1}\mathbf{W}=\mathbf{W}\). A coefficient’s standard error is the square root of the corresponding diagonal entry of this covariance matrix.

Weighted Regression With Known Absolute Variances

When the variances are known in their actual units, the covariance matrix of the coefficient estimates is \((\mathbf{X}^{\mathsf T}\mathbf{W}\mathbf{X})^{-1}\), with no additional scale multiplier. Under normal errors, a coefficient estimate divided by its standard error has a standard normal distribution when its hypothesized value is correct. This gives normal-based tests and confidence intervals.

Known Relative Variances and an Unknown Scale

Sometimes the variance pattern is known only up to a common positive scale. Write \(\operatorname{Var}(\varepsilon_i)=\sigma^2v_i\), where the positive relative variances \(v_i\) are known but \(\sigma^2\) is unknown. In that case use \(w_i=1/v_i\). The coefficient estimate has the same WLS formula, while its covariance is \(\sigma^2(\mathbf{X}^{\mathsf T}\mathbf{W}\mathbf{X})^{-1}\). Estimate the common scale by \(\hat{\sigma}^2=Q(\hat{\boldsymbol{\beta}})/(n-p)\), where \(p\) is the number of fitted coefficients and \(Q\) is the weighted sum of squared residuals. With normal errors, coefficient-based confidence intervals use a \(t\) distribution with \(n-p\) degrees of freedom. This distinction matters: known relative variances do not mean that the absolute error variances are known.

When WLS Helps—and a Common Mistake

WLS can improve precision when the variance pattern is correctly specified: observations with smaller error variance contain more information about the mean and receive greater weight. Its coefficient estimates and standard errors rely on the variance model as well as on the mean model. A wrong mean structure, such as an omitted predictor or curvature, is not repaired merely by changing weights. Likewise, WLS is not a general guarantee against all heteroskedasticity; the weights must represent the observation variances, or their relative pattern, appropriately.

• Mistake: weighting by the variance. If one observation has variance 4 times as large as another, assigning it 4 times the weight gives the less precise observation more influence. The correction is to use \(w_i=1/\sigma_i^2\), or \(1/v_i\) when variances share an unknown scale.
• Mistake: treating relative variances as absolute variances. If only \(v_i\) is known in \(\sigma^2v_i\), the common scale still has to be estimated. The coefficient covariance therefore includes \(\sigma^2\), estimated from weighted residuals.
• Mistake: interpreting the weighted slope as a different kind of effect. Weighting changes the fitted coefficient values and their precision, but the slope’s units and mean-model interpretation remain the change in fitted mean response per predictor unit.