Common Errors in GLS Weighting and Interpretation

Why GLS Notation Must Match the Calculation

Generalized least squares (GLS) is sensitive to how variability is represented. A variance is a squared-scale quantity; a standard deviation is not. Likewise, a correlation matrix describes dependence but does not supply marginal variances. Confusing these objects can change the fitted coefficients, understate or overstate their uncertainty, or make a residual diagnostic answer a different question from the one intended.

The checks in this tutorial use the linear regression model $y=X\beta+u$, conditional on the design matrix $X$, with $E(u\mid X)=0$ and $\operatorname{Var}(u\mid X)=\Omega$, where $\Omega$ is positive definite. GLS uses the covariance structure in $\Omega$; it does not require normally distributed errors to define its coefficient estimator. Distributional assumptions may be needed for particular finite-sample inference, but they are not needed for the weighting identities below.

Definition: The GLS estimator is $$ \hat{\beta}_{GLS}=(X'\Omega^{-1}X)^{-1}X'\Omega^{-1}y, $$ when $X$ has full column rank. Equivalently, it minimizes $(y-X\beta)'\Omega^{-1}(y-X\beta)$. For independent errors with variances $\sigma_i^2$, $\Omega=\operatorname{diag}(\sigma_1^2,\ldots,\sigma_n^2)$; the corresponding diagonal weights in this quadratic objective are $1/\sigma_i^2$.

Variances, Standard Deviations, and Weights

For independent heteroskedastic errors, the GLS criterion can be written $$ \sum_{i=1}^n \frac{(y_i-x_i'\beta)^2}{\sigma_i^2}. $$ Thus the weight multiplying the squared residual is the inverse variance. If a regression is instead transformed by dividing each observation and its design row by the error standard deviation, the transformed residual is $(y_i-x_i'\beta)/\sigma_i$. Squaring it produces the inverse-variance weight. These are two equivalent descriptions of the same calculation: divide the data and design by standard deviations, or weight squared residuals by inverse variances.

A common source of confusion is software or written instructions that say “use weights.” The meaning of a weight can depend on the software convention. For GLS as defined above, a scalar weight $w_i$ in the objective means $w_i=1/\sigma_i^2$. The equivalent row multiplier for transforming the data is $\sqrt{w_i}=1/\sigma_i$, not $w_i$. Check whether a stated quantity multiplies the squared residual or multiplies the residual itself.

Worked Example: Inverse Variance or Inverse Standard Deviation?

Suppose two hypothetical observations follow an intercept-only model $y_i=\beta+u_i$, with independent errors, conditional mean zero, and known error standard deviations 1 and 2. The observed responses are 10 and 14. Find the GLS estimate and compare it with the result from incorrectly using inverse standard deviations as weights on squared residuals.

The variances are $1^2=1$ and $2^2=4$, so the correct objective is $$ Q(\beta)=(10-\beta)^2+\frac{(14-\beta)^2}{4}. $$

Minimizing this weighted sum of squares gives the weighted mean $$ \hat{\beta}_{GLS} =\frac{10(1)+14(1/4)}{1+1/4} =\frac{10+3.5}{1.25} =10.8. $$

If inverse standard deviations, 1 and $1/2$, are mistakenly used as the weights on squared residuals, the calculation becomes $$ \hat{\beta}_{wrong} =\frac{10(1)+14(1/2)}{1+1/2} =\frac{17}{1.5} =11.33. $$

The incorrect weighting gives the less precise second observation too much influence. The correct transformed regression divides the first response and design row by 1 and the second by 2; its squared residuals therefore receive weights 1 and $1/4$. The calculation assumes the stated variances are the appropriate conditional variances and the errors are uncorrelated, as specified.

Correlation Matrices Do Not Set Variance Scales

A correlation matrix records pairwise correlations but not marginal variances: its diagonal entries are all 1. In $\Omega=\sigma^2R$, the common variance scale is $\sigma^2$, while $R$ supplies the correlations. More generally, if $D$ is diagonal with standard deviations on its diagonal, $\Omega=DRD$ combines those marginal scales with the correlations. The diagonal entries of $\Omega$, not those of $R$ alone, are the marginal variances.

A related scaling fact is useful for checking calculations. If the entire covariance matrix is multiplied by a positive scalar $c$, the GLS coefficient estimate is unchanged: $$ (X'(c\Omega)^{-1}X)^{-1}X'(c\Omega)^{-1}y =(c^{-1}X'\Omega^{-1}X)^{-1}c^{-1}X'\Omega^{-1}y =\hat{\beta}_{GLS}. $$ But the model-based covariance matrix of that estimator is $$ \operatorname{Var}(\hat{\beta}_{GLS}\mid X)=(X'\Omega^{-1}X)^{-1}. $$ Replacing $\Omega$ by $c\Omega$ multiplies this covariance matrix by $c$. Therefore, a common covariance scale can cancel from the coefficient estimate while still being essential for standard errors and intervals.

Worked Example: A Correlation Matrix Is Not a Covariance Matrix

Consider a hypothetical pair of responses with covariance matrix $\Omega=4R$, where $$ R=\begin{pmatrix}1&0.5\\0.5&1\end{pmatrix}, \qquad \Omega=\begin{pmatrix}4&2\\2&4\end{pmatrix}. $$ The errors have variance 4 each and correlation 0.5. Suppose the model is an intercept-only regression, $y_i=\beta+u_i$, and the observed responses are $y=(2,6)'$. Assume the stated covariance is correct and the errors have conditional mean zero.

The inverse correlation matrix is $$ R^{-1}=\begin{pmatrix}4/3&-2/3\\-2/3&4/3\end{pmatrix}, \qquad \Omega^{-1}=\frac14R^{-1} =\begin{pmatrix}1/3&-1/6\\-1/6&1/3\end{pmatrix}. $$

The GLS intercept is $$ \hat{\beta}_{GLS} =\frac{\mathbf{1}'\Omega^{-1}y}{\mathbf{1}'\Omega^{-1}\mathbf{1}} =\frac{4/3}{1/3} =4, $$ where $\mathbf{1}=(1,1)'$. Its model-based variance is the reciprocal of the denominator: $$ \operatorname{Var}(\hat{\beta}_{GLS}\mid X) =(\mathbf{1}'\Omega^{-1}\mathbf{1})^{-1}=3. $$

If $R$ is incorrectly treated as the covariance matrix, the coefficient remains 4 because this substitutes a covariance matrix scaled by $1/4$. But the calculated variance becomes $(\mathbf{1}'R^{-1}\mathbf{1})^{-1}=0.75$, four times smaller than the correct variance. The correlation structure alone did not specify the units or marginal variance scale. Reporting a standard error from $R$ as though it were $\Omega$ would therefore understate uncertainty in this example.

Whitening and the Meaning of Transformed Residuals

A GLS transformation turns the covariance structure into identity covariance. If $\Omega=LL'$ is a Cholesky factorization, define $C=L^{-1}$, $y^*=Cy$, $X^*=CX$, and $u^*=Cu$. Then $$ \operatorname{Var}(u^*\mid X) =C\Omega C' =L^{-1}LL'(L^{-1})' =I. $$ The GLS estimate is the ordinary least-squares estimate from $y^*=X^*\beta+u^*$. For diagonal $\Omega$, this transformation divides each response and design row by its error standard deviation.

A transformed residual is a residual in whitened coordinates, not an original-unit residual. If $\hat{e}=y-X\hat{\beta}_{GLS}$, then the transformed residual is $e^*=C\hat{e}$. Its entries can be compared on the whitened error scale, subject to the usual fact that fitted residuals are constrained by estimation. The original residuals $\hat{e}$ remain in the response’s units. In correlated models, a row of $C$ may combine several original residuals, so a transformed residual need not correspond to a single observation’s residual divided by its standard deviation.

Worked Example: Original and Transformed Residuals

Use an intercept-only model with two hypothetical observations $y=(2,8)'$, independent errors, conditional mean zero, and known standard deviations 1 and 2. Thus $\Omega=\operatorname{diag}(1,4)$. Calculate the GLS estimate and both forms of residuals.

The inverse-variance weights are 1 and $1/4$, so $$ \hat{\beta}_{GLS} =\frac{2(1)+8(1/4)}{1+1/4} =\frac{4}{1.25} =3.2. $$ The original-unit residuals are $$ \hat{e}=y-\hat{\beta}_{GLS}\mathbf{1} =\begin{pmatrix}-1.2\\4.8\end{pmatrix}. $$

The standard-deviation transformation divides the first row by 1 and the second by 2. Hence $X^*=(1,\,0.5)'$, $y^*=(2,\,4)'$, and $$ e^*=y^*-X^*\hat{\beta}_{GLS} =\begin{pmatrix}-1.2\\2.4\end{pmatrix}. $$ The transformed residuals satisfy the least-squares normal equation $$ (X^*)'e^*=1(-1.2)+0.5(2.4)=0. $$ Their squared length is $1.44+5.76=7.20$, equal to the GLS criterion $\hat{e}'\Omega^{-1}\hat{e}=(-1.2)^2+(4.8)^2/4=7.20$. The original residuals instead have squared length $1.44+23.04=24.48$, which is not the GLS criterion. The two residual vectors answer different scale questions; they should not be interchanged.

A Short Check for Weighting and Scaling

Before interpreting a GLS output, write down the covariance matrix the calculation actually uses, then check its inverse and the transformation. The following sequence separates the most common errors.

Identify the covariance entries.
Diagonal entries of $\Omega$ are variances; their square roots are standard deviations. For correlated errors, off-diagonal entries are covariances.

Translate weights into the objective.
In GLS, $\Omega^{-1}$ weights the quadratic residual criterion. For independent errors, each squared residual is multiplied by inverse variance.

Check the transformation.
If $C\Omega C'=I$, transform both $y$ and every column of $X$ using the same $C$. Transforming only the response changes the regression model.

Keep coefficient and uncertainty scaling distinct.
A common positive multiplier of $\Omega$ leaves $\hat{\beta}_{GLS}$ unchanged but scales its model-based covariance matrix by that multiplier.

Common Mistakes and Why They Matter

Using inverse standard deviations as weights on squared residuals. This makes the GLS criterion wrong. Use inverse variances in that objective, or divide both the response and design by standard deviations in the transformed regression.
Calling a correlation matrix the covariance matrix. A correlation matrix has unit diagonal and contains no marginal variance scales. Use $\Omega=\sigma^2R$ only when the common variance scale is part of the specification, or use an appropriate scale matrix such as $DRD$.
Assuming a correct coefficient estimate guarantees correct uncertainty. A common scale error can leave the GLS coefficient unchanged while multiplying its covariance estimate by the wrong factor. Check the covariance scale before reporting standard errors.
Reading whitened residuals as if they were in the outcome’s original units. They are transformed residuals. State which residual scale is being examined, and do not use their numerical magnitudes as original-unit prediction errors.
Transforming only $y$. GLS whitening applies to the response and the design matrix. Leaving $X$ unchanged generally fits a different model and does not implement the stated GLS estimator.

Key takeaway: Keep three objects distinct: the covariance matrix $\Omega$, its inverse in the GLS quadratic criterion, and a whitening transformation such as $L^{-1}$. Variances determine inverse-variance weights; standard deviations determine row multipliers; correlation matrices alone do not determine marginal variances.

Check Your Understanding

For each question, state the relevant scale or calculation explicitly.

Independent errors have standard deviations 3 and 5. What are the corresponding inverse-variance weights in the GLS objective?
If a covariance matrix is multiplied by 9, what happens to the GLS coefficient estimate and its model-based covariance matrix?
In $\Omega=DRD$, what information is supplied by $R$, and what information is supplied by $D$?
Why must a GLS whitening transformation be applied to the design matrix as well as the response?
What is the difference between an original residual and a transformed residual in a GLS model?