Feasible Weights From Estimated Variance Functions

Why Estimate the Weights?

Weighted least squares (WLS) uses different weights when observations have different error variances. If the variances are known, the usual choice is to weight observation \(i\) by the reciprocal of its variance. In many applications, however, the variance pattern is unknown. Regression residuals can provide information about that pattern, allowing us to estimate the variances and then carry out feasible weighted least squares (FWLS).

The word “feasible” matters: the procedure replaces unknown variance parameters with estimates. It is not generally identical to WLS using known variances, especially in a finite sample. A useful analysis therefore explains both how the weights were constructed and what assumptions support their use.

This setup assumes that the conditional mean is correctly specified and that observations have no conditional error covariance with one another. It also requires positive fitted variances. The predictors \(z_i\) may be columns of \(X\), other observed covariates, or functions of observed covariates. For example, a log-linear variance function is \(\sigma_i^2=\exp(z_i'\gamma)\); the exponential guarantees positive variances for every finite \(\gamma\).

Estimating a Variance Function From Residuals

A common starting point is an ordinary least-squares (OLS) regression of \(y\) on \(X\). Let \(\widehat{u}_i\) denote its residuals. Residual magnitudes contain information about error dispersion, but individual squared residuals are noisy: even when the variance is constant, some will be large and others small. A variance-function estimate therefore models a systematic pattern in residual squares rather than treating each square as the variance for that observation.

One approach is to group observations with similar variance predictors, calculate a residual-based variance estimate in each group, and fit a parametric function to those estimates. Another is to fit a specified model directly to squared residuals or to use a likelihood or quasi-likelihood criterion. The details depend on the model and available data. With a log-linear variance model, a common grouped approach regresses the logarithm of each group’s residual variance estimate on the group’s variance predictors. Adequate observations per group and a suitable model for how variance changes with those predictors are important.

From Estimated Variances to FWLS

Once the variance parameters have been estimated, form \(\widehat{\Omega}=\operatorname{diag}(\widehat{\sigma}_1^2,\ldots,\widehat{\sigma}_n^2)\). The FWLS estimator minimizes the weighted residual sum of squares. If \(X'\widehat{\Omega}^{-1}X\) is invertible, its formula is

Equivalently, it minimizes \(\sum_{i=1}^n (y_i-x_i'\beta)^2/\widehat{\sigma}_i^2\). An observation with a larger fitted error variance receives less weight because its outcome is expected to be less precise. Multiplying every weight by the same positive constant does not change the coefficient estimate: that constant multiplies the whole objective function but does not change its minimizer.

Why Consistency Can Survive Estimated Weights

Correct variance weights can improve precision, but consistency of the coefficient estimator does not generally require the variance function to be correct. The key mean condition is \(E(u\mid X)=0\). For any positive weights that are functions of observed covariates, the weighted population moment is then zero:

In a sample, the corresponding weighted normal equations are \(n^{-1}\sum_i \widehat{w}_i x_i(y_i-x_i'\widehat{\beta})=0\), where \(\widehat{w}_i=1/\widehat{\sigma}_i^2\). If estimated weights converge to positive, finite weights determined by the observed covariates, the weighted design matrix has a nonsingular limit, and the usual sampling regularity conditions hold, the weighted estimator converges to \(\beta\). In particular, a misspecified variance model can still yield consistent coefficient estimates under exogeneity and appropriate control of the weights. It may, however, fail to deliver the efficiency of correctly weighted estimation.

This reasoning also indicates why the mean assumption cannot be skipped. If the conditional mean is misspecified or \(E(u\mid X)\ne0\), weighting changes which observations influence the fit and does not generally repair the resulting bias. Nor is consistency guaranteed if fitted variances approach zero or become arbitrarily large in a way that makes the weights unstable, or if the weighted design loses rank.

Efficiency, Limitations, and a Common Pitfall

With a correctly specified conditional covariance and exogenous errors, GLS is the best linear unbiased estimator under the Gauss–Markov conditions. FWLS aims to approach that benchmark by estimating the covariance pattern. When the variance function is consistently estimated, fitted variances remain well behaved, and the design is sufficiently informative, the coefficient estimator can have the same large-sample first-order behavior as GLS with the true variances. This is an asymptotic claim, not a promise of improved precision in every finite sample.

If the variance function is wrong, positive weights based on observed covariates can still preserve consistency under the conditional mean assumption, but generally do not attain GLS efficiency. If weights are estimated from the same regression residuals, their estimation can also affect finite-sample behavior. Inference should not be treated as automatically exact merely because a weighted regression was run; its validity depends on the sampling and variance assumptions and on how uncertainty in the fitted variance model is handled.

Common pitfall: “The residuals are larger for some observations, so I can use their inverse squared residuals as weights.” This confuses noisy individual residual squares with underlying conditional variances. A single residual may be large by chance, and using its reciprocal can produce an extreme weight. Instead, estimate a structured variance function from a sufficiently informative collection of residuals, check that fitted variances are positive and plausible, and state the assumptions that support the chosen weighting model.