Groupwise Heteroskedasticity and Weighted Regression

Why Groupwise Variances Change the Regression

A regression can have the same conditional mean relationship across observations but different error variances across groups. For example, measurements from one setting may be much more variable than measurements from another. If those variances are known, generalized least squares (GLS) gives each observation a contribution inversely proportional to its error variance. With groupwise heteroskedasticity, that principle becomes a simple group-specific rescaling.

The rescaling is not an arbitrary change to the data. It converts the model errors to a common variance while preserving the linear mean model in transformed variables. The resulting ordinary least-squares calculation is exactly GLS for the assumed covariance matrix.

Model and assumptions: For observations \(i=1,\ldots,n\), let \(y_i=x_i'\beta+u_i\), where \(x_i\) is a \(k\)-vector of regressors and \(\beta\) is the coefficient vector. Each observation belongs to a known group \(g(i)\in\{1,\ldots,G\}\). Assume \(E(u_i\mid X,G)=0\), and \(\operatorname{Var}(u_i\mid X,G)=\sigma_{g(i)}^2\), where the positive group variances \(\sigma_g^2\) are known. Also assume errors from distinct observations are uncorrelated, so the conditional error covariance matrix is diagonal. Normality is not required to calculate GLS; it is required for exact finite-sample normal-theory inference.

Here \(X\) is the \(n\times k\) design matrix, with row \(x_i'\), and \(G\) records the group membership. The model permits different variances between groups, but assumes a common variance within each group. It does not allow additional correlation between observations or an arbitrary variance pattern within a group.

From Group Variances to Rescaled Regression

Let \(\Omega\) be the conditional covariance matrix of the error vector \(u\). Under the assumptions above, its diagonal entry for observation \(i\) is \(\sigma_{g(i)}^2\). GLS minimizes the error sum of squares weighted by \(\Omega^{-1}\). For groupwise variances, this is the same as weighting each squared residual by the reciprocal of its group variance.

\widehat{\beta}_{GLS} =(X'\Omega^{-1}X)^{-1}X'\Omega^{-1}y, \qquad \sum_{i=1}^n\frac{(y_i-x_i'\beta)^2}{\sigma_{g(i)}^2}.

To see the transformed regression, divide observation \(i\)'s equation by its group standard deviation \(\sigma_{g(i)}\). The transformed error is \(u_i/\sigma_{g(i)}\), whose conditional variance is one. Thus the appropriate transformation divides both the outcome and every regressor, including the intercept column, by the group standard deviation.

y_i^*=\frac{y_i}{\sigma_{g(i)}}, \qquad x_i^*=\frac{x_i}{\sigma_{g(i)}}, \qquad u_i^*=\frac{u_i}{\sigma_{g(i)}}, \qquad y_i^*=(x_i^*)'\beta+u_i^*.

Key result: Under the stated covariance assumptions, OLS of \(y_i^*\) on \(x_i^*\) is GLS of the original outcomes on the original regressors. Equivalently, minimize \(\sum_i w_i(y_i-x_i'\beta)^2\) using \(w_i=1/\sigma_{g(i)}^2\). The regression transformation uses \(\sqrt{w_i}=1/\sigma_{g(i)}\), not \(w_i\), to multiply each row.

The distinction between the weight and the row multiplier is important. Squaring a transformed residual multiplies its original squared residual by \(1/\sigma_{g(i)}^2\). Hence the objective uses inverse variances, whereas the transformed variables themselves use inverse standard deviations.

Worked Example: Rescaling Two Variance Groups

Suppose a hypothetical regression includes an intercept and a predictor \(x\). Group A has known error variance \(1\), and Group B has known error variance \(9\). The hypothetical observations are:

Group	\(x_i\)	\(y_i\)	\(\sigma_g^2\)	Row multiplier
A	1	4	1	1
B	2	9	9	1/3

For Group A, \(\sigma_A=1\), so its transformed row is unchanged: \(y_A^*=4\), and its intercept and predictor entries are \(1\) and \(1\). For Group B, \(\sigma_B=3\), so divide the entire row by \(3\): \(y_B^*=9/3=3\), and its transformed regressors are \(1/3\) and \(2/3\).

The corresponding weighted least-squares objective gives Group A weight \(1/1=1\) and Group B weight \(1/9\). In particular, Group B's transformed residual is one-third of its original residual, and its squared transformed residual is one-ninth as large. This example illustrates the transformation; two observations alone do not provide enough information to estimate an intercept and slope.

Calculating the Weighted Coefficients

For a regression with an intercept and one predictor, the weighted coefficient formulas are convenient for hand calculations. Define \(W=\sum_iw_i\), \(\bar{x}_w=\sum_iw_ix_i/W\), and \(\bar{y}_w=\sum_iw_iy_i/W\). The weighted slope is the weighted centered cross-product divided by the weighted centered sum of squares; the intercept makes the fitted line pass through the weighted means.

\widehat{\beta}_1 = \frac{\sum_i w_i(x_i-\bar{x}_w)(y_i-\bar{y}_w)} {\sum_i w_i(x_i-\bar{x}_w)^2}, \qquad \widehat{\beta}_0=\bar{y}_w-\widehat{\beta}_1\bar{x}_w.

Worked Example: A Two-Group Weighted Regression

Consider these hypothetical observations from a regression of \(y\) on \(x\) with an intercept. Group A has known error variance \(1\), and Group B has known error variance \(4\). Assume zero conditional error means and uncorrelated errors, with a common variance within each group.

Observation	Group	\(x_i\)	\(y_i\)	\(w_i=1/\sigma_g^2\)
1	A	0	1	1
2	A	1	2	1
3	B	0	0	1/4
4	B	1	2	1/4

Group A's standard deviation is \(1\), and Group B's is \(2\); therefore the row multipliers are \(1\) and \(1/2\), respectively. The weights in the table are their squared multipliers. Accumulating the weighted sums gives \(W=1+1+1/4+1/4=2.5\), \(\sum_iw_ix_i=0+1+0+1/4=1.25\), \(\sum_iw_iy_i=1+2+0+1/2=3.5\), \(\sum_iw_ix_i^2=1+1/4=1.25\), and \(\sum_iw_ix_iy_i=2+1/2=2.5\).

Thus \(\bar{x}_w=1.25/2.5=0.5\), and \(\bar{y}_w=3.5/2.5=1.4\). The weighted centered denominator is \(1.25-(1.25)^2/2.5=0.625\). The weighted centered numerator is \(2.5-(1.25)(3.5)/2.5=0.75\). It follows that \(\widehat{\beta}_1=0.75/0.625=1.2\), and \(\widehat{\beta}_0=1.4-(1.2)(0.5)=0.8\). The fitted conditional mean is \(0.8+1.2x\). Its slope is estimated in units of \(y\) per one-unit increase in \(x\); the weights change the fitting criterion, not the units or interpretation of the original coefficients.

A Centered-Sum Calculation and Its Check

The weighted centered sums can be computed directly from the weighted cross-products. For example, the centered numerator equals \(\sum_iw_ix_iy_i-(\sum_iw_ix_i)(\sum_iw_iy_i)/W\). This identity is useful as an arithmetic check, but each weighted cross-product must include every observation's contribution.

Worked Example: Checking the Weighted Slope Arithmetic

For a separate hypothetical four-observation calculation, suppose the weighted sufficient sums are \(W=2.5\), \(\sum_iw_ix_i=2.25\), \(\sum_iw_iy_i=7\), \(\sum_iw_ix_i^2=4.25\), and \(\sum_iw_ix_iy_i=11\). These are the weighted totals for this illustration; the groups' known variances determine the corresponding \(w_i\).

The weighted means are \(\bar{x}_w=2.25/2.5=0.9\) and \(\bar{y}_w=7/2.5=2.8\). The denominator is \(4.25-(2.25)^2/2.5=4.25-2.025=2.225\). The numerator is \(11-(2.25)(7)/2.5=11-6.3=4.7\). Therefore, \(\widehat{\beta}_1=4.7/2.225\approx2.112\), and \(\widehat{\beta}_0=2.8-(2.11236)(0.9)\approx0.899\). Using the unrounded slope in the intercept calculation gives approximately \(0.899\). The fitted slope is about \(2.112\) outcome units per predictor unit.

A useful check is to add the individual weighted cross-products before centering. If the four contributions to \(\sum_iw_ix_iy_i\) are \(0,3,2,\) and \(6\), their sum is \(11\), not \(10\). Substituting \(10\) would incorrectly produce a centered numerator of \(3.7\), changing both the slope and intercept.

Why the Assumptions Matter

The groupwise formula is appropriate when the covariance structure is diagonal and the variance is constant within each specified group. A group label by itself does not justify assigning a common variance: the model must make that assumption, and the variances used for weighting must be known for this tutorial's GLS calculation. If errors are correlated, a diagonal set of group weights does not represent their full covariance matrix.

Under the stated assumptions, the GLS estimator is unbiased conditional on \(X\) when \(E(u\mid X,G)=0\), and its covariance matrix is \((X'\Omega^{-1}X)^{-1}\), provided \(X'\Omega^{-1}X\) is invertible. Invertibility requires enough independent weighted variation in the regressors to identify the coefficients. Weighting improves efficiency relative to unweighted least squares when the specified covariance is correct; it does not establish a causal interpretation for a regression coefficient.

The squared-error objective also explains the role of group precision. A residual of a given size contributes less in a high-variance group, because such deviations are less surprising under the model. This is not a claim that observations in that group are unimportant; rather, their residuals carry less precision about the common regression mean per observation.

Specify the variance groups.
Assign each observation to a group and state the assumed known variance \(\sigma_g^2\) for that group.

Check the covariance setup.
Confirm the model's zero conditional mean and that the assumed error covariance has no off-diagonal dependence.

Transform the entire row.
Divide \(y_i\) and every element of \(x_i\), including an intercept, by \(\sigma_{g(i)}\).

Fit and interpret the regression.
Run OLS on the transformed data, or use weights \(1/\sigma_{g(i)}^2\); interpret the resulting coefficients on the original outcome and regressor scales.

Common Mistakes

Using inverse standard deviations as the objective weights. Dividing rows by \(\sigma_g\) is correct, but the squared-residual weight is \(1/\sigma_g^2\). The row multiplier is the square root of the objective weight.
Scaling only the outcome. Dividing \(y_i\) while leaving the regressors unchanged does not implement GLS. Every regressor in that observation's row must receive the same multiplier.
Using variance instead of inverse variance. A group with larger error variance gets a smaller weight, not a larger one.
Assuming weights repair correlation. Group-specific diagonal weights address unequal variances only. Correlated errors require a covariance matrix that represents the dependence.
Dropping or misadding a weighted cross-product. Compute \(\sum_iw_ix_iy_i\) from all observations before centering, and verify its contributions as in the arithmetic check above.

Key takeaway: With known group variances and uncorrelated errors, groupwise GLS is OLS after dividing each observation's outcome and complete regressor row by its group standard deviation. The equivalent least-squares weight is the inverse group variance.

Check Your Understanding

Answer each question using the groupwise variance model and transformation developed here.

Group C has known error variance \(16\). What row multiplier and weighted least-squares objective weight does an observation in Group C receive?
Why must the intercept column be divided by the group standard deviation along with the other regressors?
If one group has four times the error variance of another, what is the ratio of their inverse-variance weights?
State two covariance assumptions needed for the diagonal groupwise GLS calculation.
In a weighted simple regression, explain why using \(\sum_iw_ix_iy_i=10\) instead of \(11\) changes the centered slope numerator by one unit.