Estimated Weights and Feasible Weighted Least Squares

From Known Variances to Estimated Weights

Weighted least squares is most straightforward when the error variance for each observation is known. In practice, those variances are rarely supplied. Instead, an analyst may see that residuals tend to be small in one part of the predictor range and large in another, then estimate a variance pattern from the data. A weighted regression using those estimated variances is called feasible weighted least squares (FWLS, also called feasible generalized least squares).

The word “feasible” matters: the weights are not known in advance but are estimated. This can improve precision when the variance pattern is well estimated, but it also introduces uncertainty. If the estimated pattern is inaccurate, the weighted fit may not have the efficiency advantage expected from correct weights. Its uncertainty calculations should not be treated as automatically reliable just because the regression was weighted.

Definition: Suppose the mean model is \(Y_i=\mathbf{x}_i^\mathsf{T}\boldsymbol{\beta}+\varepsilon_i\), where \(\mathbf{x}_i\) is the predictor vector for observation \(i\), and the errors have mean zero and variances \(\operatorname{Var}(\varepsilon_i)=\sigma_i^2\). If the variances are unknown, FWLS first estimates them as \(\hat{\sigma}_i^2\), then uses weights \(w_i=1/\hat{\sigma}_i^2\) to fit the mean model by minimizing the weighted sum of squared residuals.

The model assumes that the conditional mean is correctly represented by the chosen regression model, that observations have independent errors, and that each error has mean zero and a finite, positive variance. The variance may differ across observations. A variance model adds a further working assumption: the pattern of variances can be estimated adequately from the available data. The weights are inverse variances, not inverse standard deviations. An observation estimated to have four times the error variance therefore receives one quarter the weight.

Estimating a Variance Pattern

A practical starting point is an ordinary least-squares fit of the mean model. Its residuals, \(\hat{\varepsilon}_i=Y_i-\hat{Y}_i\), provide information about the spread of errors. For example, observations can be grouped by a predictor and the average squared residual calculated within each group. Alternatively, a chosen variance function can be fitted to the squared residuals or their logarithms. Either approach is an estimate, not a direct observation of the true error variances.

This distinction is important because residuals come from a fitted model. Fitting the mean generally makes residuals smaller than the errors would be, and the extent of that effect can differ across observations. Residual squares can also be very noisy, especially when estimated separately for many small groups. A useful variance estimate should capture a plausible broad pattern without chasing random fluctuations. If the residual pattern is really caused by a misspecified mean—for example, unmodelled curvature—weighting alone will not correct it.

Worked Example: Turning Grouped Residuals Into Weights

Suppose a preliminary linear regression is fitted to hypothetical measurements. For illustration, the fitted residuals are grouped into three predictor ranges. The residuals are in response units, and the estimated variance pattern will be used to set relative weights.

Predictor range	Illustrative residuals	Mean squared residual
Low	-1, 1	\((1+1)/2=1\)
Middle	-2, 2	\((4+4)/2=4\)
High	-3, 3	\((9+9)/2=9\)

Treat these group means as rough estimates of the error variances. The resulting relative weights are their reciprocals:

\hat{\sigma}_{\text{low}}^2=1,\quad \hat{\sigma}_{\text{middle}}^2=4,\quad \hat{\sigma}_{\text{high}}^2=9 \qquad\Longrightarrow\qquad w_{\text{low}}=1,\quad w_{\text{middle}}=\frac14,\quad w_{\text{high}}=\frac19.

Thus, an observation in the high range receives one ninth the weight of an observation in the low range, because its estimated variance is nine times as large. These tiny groups are only a transparent arithmetic illustration: two residuals per range would not normally estimate a variance pattern reliably. In an actual analysis, the pattern should be estimated from adequate data and assessed for plausibility.

Only relative variances are needed to calculate the weighted least-squares coefficients. Multiplying every weight by the same positive constant leaves those coefficients unchanged. For example, weights \(1,\frac14,\frac19\) and \(9,\frac94,1\) produce the same coefficient estimates. A common scale is needed for the conventional model-based covariance estimate, but not for the coefficient fit itself.

Fitting the Feasible Weighted Regression

Let \(X\) be the design matrix, \(\mathbf{Y}\) the response vector, and \(W\) the diagonal matrix whose entries are the estimated weights. The FWLS estimate minimizes \((\mathbf{Y}-X\boldsymbol{\beta})^\mathsf{T}W(\mathbf{Y}-X\boldsymbol{\beta})\). Solving the resulting weighted normal equations gives the coefficient estimate.

\hat{\boldsymbol{\beta}}_{\text{FWLS}} =(X^\mathsf{T}WX)^{-1}X^\mathsf{T}W\mathbf{Y}.

The minimization explains the role of the weights: a large residual from an observation with high estimated variance contributes less to the objective than the same-sized residual from an observation with low estimated variance. This is appropriate only if the mean model is adequate and the relative variances are estimated usefully.

Worked Example: Calculating a Weighted Regression Line

Consider four hypothetical observations and relative weights estimated from a variance pattern. Fit a line to the data. The weights are \(1\) for the first two observations and \(\frac14\) for the last two.

\(x_i\)	\(y_i\)	\(w_i\)
0	1	1
1	2	1
2	2	\(\frac14\)
3	5	\(\frac14\)

For a weighted line, define \(S_w=\sum w_i\), \(S_x=\sum w_ix_i\), \(S_y=\sum w_iy_i\), \(S_{xx}=\sum w_ix_i^2\), and \(S_{xy}=\sum w_ix_iy_i\). Substituting the table values gives

\begin{aligned} S_w&=1+1+\frac14+\frac14=2.5,\\ S_x&=0+1+\frac12+\frac34=2.25,\\ S_y&=1+2+\frac12+\frac54=4.75,\\ S_{xx}&=0+1+1+\frac94=4.25,\\ S_{xy}&=0+2+1+\frac{15}{4}=6.75. \end{aligned}

The weighted slope and intercept are

\begin{aligned} \hat{\beta}_1 &=\frac{S_wS_{xy}-S_xS_y}{S_wS_{xx}-S_x^2} =\frac{(2.5)(6.75)-(2.25)(4.75)}{(2.5)(4.25)-(2.25)^2} =\frac{6.1875}{5.5625} \approx 1.112,\\ \hat{\beta}_0 &=\frac{S_y-\hat{\beta}_1S_x}{S_w} =\frac{4.75-(1.11236)(2.25)}{2.5} \approx 0.899. \end{aligned}

The fitted line is approximately \(\hat{Y}=0.899+1.112x\). The two observations assigned smaller weights have less influence on the fit than they would under equal weighting. This calculation establishes the weighted fit; it does not establish that the variance estimates are accurate or that the usual standard errors are appropriate.

Why Estimated Weights Change the Uncertainty Question

If the true variances were known up to a common scale, and the independent errors followed the specified variance pattern, a conventional covariance estimate for the weighted coefficients would be

\widehat{\operatorname{Cov}}_{\text{conventional}} (\hat{\boldsymbol{\beta}}) =\hat{\sigma}^2(X^\mathsf{T}WX)^{-1}.

Here \(\hat{\sigma}^2\) estimates the common scale after accounting for the fitted coefficients. With estimated weights, this formula is usually a working approximation: it acts as if the fitted weights were the correct relative variances. It does not, by itself, represent uncertainty from estimating the variance pattern. If that pattern is weakly estimated, based on a small sample, or selected after trying many alternatives, the conventional standard errors and tests may understate uncertainty.

A sandwich covariance estimate can provide a useful alternative when the chosen weights do not exactly describe the error variances. For the weighted fit, its HC0 form is

\widehat{\operatorname{Cov}}_{\text{HC0, weighted}} = (X^\mathsf{T}WX)^{-1} \left(\sum_{i=1}^{n} w_i^2\hat{\varepsilon}_i^2 \mathbf{x}_i\mathbf{x}_i^\mathsf{T}\right) (X^\mathsf{T}WX)^{-1}.

This calculation uses the observed weighted design and residual squares rather than assuming that the selected variance pattern is exactly right. It is still an approximation, and it does not automatically include the uncertainty from estimating or choosing the weights. In particular, robust standard errors do not repair a wrong mean model, dependence between observations, or severe small-sample limitations. HC1, HC2, or HC3 adjustments can be considered as appropriate; for a weighted fit, leverage is based on the weighted design.

Worked Example: Comparing Two Uncertainty Calculations

Consider a hypothetical intercept-only model with responses \(2,4,5\) and weights \(1,\frac14,\frac14\). Suppose these weights were estimated from a variance pattern, but that pattern may be imperfect. The weighted estimate of the common mean is

\hat{\beta}_0 =\frac{\sum w_i y_i}{\sum w_i} =\frac{2+1+\frac54}{1+\frac14+\frac14} =\frac{17}{6} \approx 2.833.

The residuals are \(-\frac56,\frac76,\frac{13}{6}\). For comparison, the conventional weighted estimate of a common scale, using \(n-p=3-1=2\) degrees of freedom, is

\hat{\sigma}^2 =\frac{\sum w_i\hat{\varepsilon}_i^2}{2} =\frac{\frac{25}{36}+\frac14\frac{49}{36}+\frac14\frac{169}{36}}{2} =\frac{53}{48} \approx 1.104.

Here \(X^\mathsf{T}WX=\sum w_i=1.5\). The conventional variance estimate for the intercept is therefore

\widehat{\operatorname{Var}}_{\text{conventional}}(\hat{\beta}_0) =\frac{53/48}{1.5} =\frac{53}{72} \approx 0.7361, \qquad SE_{\text{conventional}}\approx \sqrt{0.7361}=0.8580.

For HC0, the middle term in the sandwich formula is the scalar \(\sum w_i^2\hat{\varepsilon}_i^2\). Thus,

\begin{aligned} \widehat{\operatorname{Var}}_{\text{HC0}}(\hat{\beta}_0) &=\frac{1}{(1.5)^2} \left[ \frac{25}{36}+\frac{1}{16}\frac{49}{36} +\frac{1}{16}\frac{169}{36} \right]\\ &=\frac{103}{216}\approx 0.4769, \qquad SE_{\text{HC0}}\approx \sqrt{0.4769}=0.6905. \end{aligned}

The two standard errors differ because they use different assumptions about the variances. Neither number should be treated as definitive for only three observations: HC0 is not generally dependable in such a small sample, and the estimated weights add another source of uncertainty. The calculation illustrates how a robust covariance estimate is formed; it does not justify a strong inferential conclusion from these data.

A Short Justification and a Practical Workflow

For fixed weights, the weighted sum of squares is \(Q(\boldsymbol{\beta})=(\mathbf{Y}-X\boldsymbol{\beta})^\mathsf{T}W(\mathbf{Y}-X\boldsymbol{\beta})\). Differentiating with respect to \(\boldsymbol{\beta}\) and setting the derivative to zero gives

\frac{\partial Q}{\partial\boldsymbol{\beta}} =-2X^\mathsf{T}W(\mathbf{Y}-X\boldsymbol{\beta})=0 \quad\Longrightarrow\quad X^\mathsf{T}WX\hat{\boldsymbol{\beta}}=X^\mathsf{T}W\mathbf{Y}.

If \(X^\mathsf{T}WX\) is invertible, solving this equation yields the FWLS formula. The same algebra applies whether the weights are known or estimated. What changes is how much confidence we should place in the chosen weights and in the resulting uncertainty calculation.

A careful analysis separates the steps rather than treating the weights as unquestionable facts:

Fit the mean model and inspect residual spread.
Use residuals to look for a systematic variance pattern, while checking that apparent spread changes are not better explained by a misspecified mean.

Estimate a parsimonious variance pattern.
Choose groups or a variance function that the data can support. Avoid assigning a separate, unstable variance estimate to every observation.

Form inverse-variance weights and refit.
Use positive weights proportional to \(1/\hat{\sigma}_i^2\), then calculate the weighted regression coefficients.

Assess uncertainty and sensitivity.
Compare model-based and robust uncertainty where appropriate, and examine whether reasonable changes to the variance model materially alter the fit or conclusions.

Common Mistake: Treating Estimated Weights as Known

A common mistake is to estimate weights from residuals, fit FWLS, and then report conventional standard errors as if the weights had been supplied without error. The reasoning is that the formula is the same as in weighted least squares with known variances. The correction is that identical coefficient formulas do not imply identical uncertainty: estimated weights are themselves uncertain, and a misspecified variance pattern can make the conventional covariance formula misleading. Robust standard errors can reduce reliance on the exact variance pattern, but they do not account automatically for every consequence of estimating the weights.

Key takeaway: FWLS estimates relative variances and weights observations inversely to those estimates. It can improve precision when the mean model and variance pattern are adequate, but estimated weights are not known variances. Check the variance model, treat conventional standard errors as assumption-dependent, and use robust uncertainty calculations as a safeguard—not as a cure for all model problems.

Check Your Understanding

Try these questions before moving on.

If one observation is estimated to have four times the error variance of another, what is the ratio of their inverse-variance weights?
Why can residual squares provide information about a variance pattern, and why are they not direct measurements of the error variances?
Explain why multiplying every weight by the same positive constant changes neither the FWLS coefficient estimates nor the relative weighting.
What assumption about the variance pattern underlies the conventional weighted covariance formula?
What does the weighted HC0 sandwich calculation protect against, and name one important source of uncertainty it does not automatically include.