Weighted Least Squares From Known Observation Variances

Why Known Unequal Variances Change the Fit

In a linear regression, observations can have different error variances. If those variances are known, an observation with a larger variance is less precise than one with a smaller variance. Weighted least squares (WLS) accounts for that difference by giving each observation weight in inverse proportion to its variance. This changes the fitting criterion, not the underlying regression equation: the goal remains to estimate the same coefficient vector.

The distinction matters in applications where observation-specific precision is supplied by the measurement process or the design. WLS is not simply a way to make residuals look more uniform, and the weights are not chosen by inspecting which observations have the largest residuals. The inverse-variance rule follows from the model’s error covariance.

Model and Assumptions

The diagonal covariance matrix is the key restriction here: the error variance may differ by observation, but there is no covariance between distinct observations. No normality assumption is required to define or calculate WLS, or to obtain its conditional covariance matrix. If the errors are also conditionally normal, the same criterion is the Gaussian likelihood criterion for \(\beta\). The variances must be treated as known for the results below; replacing them with estimates changes the problem.

Let \(D=\operatorname{diag}(\sigma_1,\ldots,\sigma_n)\). Since its diagonal entries are positive, \(D\) is invertible. Multiplying the regression by \(D^{-1}\) divides each observation by its error standard deviation. In the transformed model the error covariance is the identity matrix:

This transformation is called whitening. It turns the unequal-variance regression into a regression with equal transformed error variances. Ordinary least squares on the transformed variables is therefore the route to WLS.

Deriving the Inverse-Variance Weights

Write \(\hat{\beta}_{WLS}\) for the WLS estimator. Ordinary least squares on the whitened equation minimizes the sum of squared transformed residuals. For a candidate coefficient vector \(b\), the transformed residual for observation \(i\) is \((y_i-x_i'b)/\sigma_i\), where \(x_i'\) is row \(i\) of \(X\). Therefore the criterion is:

To derive the estimator, differentiate the matrix criterion with respect to \(b\) and set the derivative to zero. Since \(W\) is symmetric, the resulting normal equations and solution are:

The inverse exists because \(X\) has full column rank and \(W\) is positive definite: for any nonzero \(a\), \(a'X'WXa=(Xa)'W(Xa)>0\). This also ensures the criterion has a unique minimum. Multiplying all weights by the same positive constant does not change the estimator: the common factor appears in both \(X'WX\) and \(X'Wy\) and cancels. But the relative weights matter. One must not confuse the regression weights \(1/\sigma_i^2\) with the divisors \(1/\sigma_i\) used to transform individual observations.

Properties of the Estimator

Because \(\hat{\beta}_{WLS}\) is a linear function of \(y\), its expectation and conditional covariance follow directly from the model:

For the covariance calculation, set \(A=(X'WX)^{-1}X'W\), so that \(\hat{\beta}_{WLS}=Ay\). Then \(\operatorname{Var}(Ay\mid X)=A\Omega A'\). Substituting \(W=\Omega^{-1}\), and using the symmetry of \(W\), gives:

because \(W\Omega W=W\). Thus, when the variances are known, this covariance matrix does not require an estimate of a common error variance. Under the stated zero-mean and covariance assumptions, WLS is the best linear unbiased estimator: among linear unbiased estimators, it has the smallest covariance matrix in the positive-semidefinite ordering. This is the GLS efficiency result for the diagonal covariance matrix \( \Omega \).

The covariance result does not by itself assert that the coefficient estimates are normally distributed. Conditional normality of the errors gives normal coefficient estimates and supports exact normal-theory inference using the known covariance matrix. Without that distributional assumption, unbiasedness and the covariance expression still hold, but exact finite-sample normal inference does not follow automatically.

Weighted Regression With Unequal Known Variances

For a regression with an intercept and one predictor, the WLS criterion is the sum of each squared residual multiplied by its inverse error variance. In the next example, the observations with larger known variances are deliberately given less influence. The variances are part of the hypothetical setup, not estimates obtained from this small data set.

Checking the Whitening Calculation

The weighted and whitened formulations are two expressions of the same calculation. Dividing each outcome and each design row by \(\sigma_i\) produces ordinary least squares on transformed data. The example below checks this equivalence using the fitted regression just calculated.

Common Mistakes and Practical Limits

• Using variance instead of inverse variance. Giving more weight to larger variances reverses the precision adjustment. The objective uses \(w_i=1/\sigma_i^2\), while whitening divides by \(\sigma_i\).
• Assuming WLS repairs a misspecified mean model. WLS changes how observations contribute to fitting; it does not fix omitted variables, incorrect functional form, or a nonzero conditional error mean.
• Treating estimated variances as known. The calculations here rely on observation variances supplied as known. Estimating a variance function and constructing feasible weights requires additional methods and uncertainty considerations.
• Assuming unequal variances alone imply independent errors. The diagonal covariance model also rules out cross-observation covariance. If errors are correlated, inverse variances alone do not describe the full covariance adjustment.
• Believing any positive weights have the same justification. A common positive rescaling leaves the coefficient estimate unchanged, but changing relative weights changes the estimator. The inverse-variance choice is justified by the stated covariance model.

Known-variance WLS is useful when the precision differences are substantively supported and the covariance assumptions are credible. Its value is not that it guarantees a better-looking regression, but that it uses the supplied error information to obtain the appropriate linear unbiased estimator under the model.