Why Estimating the Covariance Need Not Change the First-Order Limit
Generalized least squares uses a specified error covariance matrix. In applications, that covariance structure is usually unknown and its parameters must be estimated, producing feasible GLS (FGLS). Estimation introduces another source of randomness, but it does not automatically alter the large-sample distribution of the regression coefficient estimator. The key question is whether the estimated covariance matrix becomes sufficiently close to the true one in the parts of the calculation that determine the coefficient’s limiting distribution.
This tutorial develops that comparison. The central result is that, under suitable design, central limit, and smoothness conditions, a consistent covariance-parameter estimator is enough for FGLS to have the same first-order limit as GLS that knows the true covariance. A root-\(n\) covariance-parameter estimator is common, but is not generally required for this first-order equivalence.
Model and Target Result
The corresponding oracle GLS estimator uses \(\Omega_n(\theta_0)\) in place of \(\Omega_n(\hat\theta_n)\). The target is to show that the difference between FGLS and oracle GLS is negligible after multiplication by \(\sqrt n\). If that difference is negligible and oracle GLS has a limiting normal distribution, FGLS has that same limiting distribution.
Here are sufficient regularity conditions. The number of regressors \(k\) is fixed, the design has full column rank for sufficiently large \(n\), and the normalized information matrix converges:
Assume also that the oracle GLS score satisfies a central limit condition:
The conditional covariance of the score on the left is \(Q_n\), since \(W_n=\Omega_n(\theta_0)^{-1}\) gives \(W_n\Omega_n(\theta_0)W_n=W_n\). The displayed central limit condition requires more than this covariance calculation; it also rules out error or design behavior that prevents a normal limit. It may be justified under appropriate independence or dependence conditions for the application.
Finally, the covariance estimator must be consistent and its effect on the normalized design and score must be stable. Write \(\hat W_n=\Omega_n(\hat\theta_n)^{-1}\) and \(W_n=\Omega_n(\theta_0)^{-1}\). A useful direct set of sufficient conditions is
The first condition says that estimated weights do not change the normalized information matrix by a nonvanishing amount. The second says they do not add a nonvanishing term to the normalized regression score. These conditions make explicit the stability needed; ordinary matrix closeness alone should not be used without checking how the design and errors enter the products.
Deriving the FGLS Limit
Because \(y_n=X_n\beta+u_n\), subtracting \(\beta\) from the FGLS formula yields an exact expression. Under the stated conditions, that expression can be normalized and compared with its oracle counterpart:
Since \(Q_n\) converges to the positive definite matrix \(Q\), its perturbed version also converges to \(Q\), and its inverse converges to \(Q^{-1}\). The score converges to \(N(0,Q)\) by the oracle central limit condition. Therefore the continuous mapping and Slutsky results give
This is also the oracle GLS limit. The covariance-parameter estimation has not contributed an additional first-order term under these conditions. This result concerns the coefficient estimator’s asymptotic distribution; it does not claim that the estimated covariance matrix is known, nor that every finite-sample procedure using it has the same behavior as a procedure using the true matrix.
What Rate of Covariance Estimation Is Needed?
Suppose \(\Omega_n(\theta)\) is a smooth, correctly specified covariance family and \(\hat\theta_n\) converges in probability to \(\theta_0\). Consistency by itself is not a complete proof: one also needs the normalized design and score stability conditions above. A common sufficient argument is that the derivatives of the normalized information and score with respect to \(\theta\) remain well behaved near \(\theta_0\). Then a Taylor expansion makes their perturbations proportional, to first order, to \(\|\hat\theta_n-\theta_0\|\).
In particular, if the normalized derivative of the score is \(O_p(1)\), then
Thus any rate \(\|\hat\theta_n-\theta_0\|=o_p(1)\) makes this perturbation vanish. A rate such as \(O_p(n^{-1/2})\) is faster than needed for this first-order conclusion; a slower rate can suffice if it still converges to zero and the regularity conditions hold. If the covariance estimator is inconsistent, or if the normalized derivative or stochastic stability conditions fail, the argument does not establish oracle equivalence.
Worked Example: A Slower-Than-Root-\(n\) Covariance Rate
Suppose a scalar covariance parameter has true value \(\theta_0=2\), and assume a smooth covariance model satisfies the score-stability condition just described. Imagine the estimator has rate \(\hat\theta_n-\theta_0=O_p(n^{-1/3})\). Does this rate suffice for the first-order FGLS limit?
The score perturbation is \(O_p(\|\hat\theta_n-\theta_0\|)=O_p(n^{-1/3})\), which converges in probability to zero. The normalized information perturbation also vanishes if its corresponding smoothness and stability condition holds. Consequently the covariance estimation adds no nonvanishing term to the root-\(n\) coefficient expansion, and FGLS has the oracle limit under the other assumptions.
For scale, at \(n=1{,}000\), the rate factor is \(n^{-1/3}=0.100\). This is a rate illustration, not a claim that the parameter error equals \(0.100\) in every sample. The asymptotic requirement is that the random error tends to zero in probability; the rate indicates how quickly the associated first-order perturbation can shrink.
Worked Asymptotic Variance: Two Variance Groups
Consider a hypothetical regression through the origin, \(y_i=\beta x_i+u_i\), with two equally sized groups. In group 1, \(x_i=1\) and the error variance is \(1\); in group 2, \(x_i=2\) and the error variance is \(4\). Errors have conditional mean zero, and assume a central limit condition for the weighted score. The covariance model correctly represents these two group variances.
Worked Example: Oracle and Feasible GLS Information
Calculate the limiting information per observation \(Q\) and the limiting variance of \(\sqrt n(\hat\beta_{\mathrm{FGLS}}-\beta)\). Then examine the effect of using illustrative estimated variances \(1.21\) and \(3.61\).
Oracle information. GLS weights are the inverse variances. Each group accounts for one half of the observations, so
The limiting variance is \(Q^{-1}=1.000\), so the asymptotic distribution is \(N(0,1)\). This variance is for the root-\(n\) scaled coefficient estimator; the approximate variance of the unscaled estimator is \(1/n\).
Illustrative estimated weights. Using the hypothetical variance estimates, the normalized estimated information is
Its inverse is \(1/0.967240=1.0339\). This is an illustrative plug-in calculation at those particular variance estimates, not the limiting variance. If the estimates converge to \(1\) and \(4\), respectively, the normalized information converges to \(1\), and the limiting variance returns to \(1\), provided the score and other regularity conditions also hold.
Assumption check. The example specifies positive variances, a full-rank scalar design with nonzero \(x_i\), and a correctly specified two-group covariance structure. The required central limit condition is stipulated for the weighted score; it would need justification from the actual sampling or dependence structure in an application.
A Rate Comparison and Its Meaning
Worked Example: Comparing Root-\(n\) and Slower Rates
Suppose the regularity conditions imply that the root-\(n\) difference between FGLS and oracle GLS is \(O_p(\|\hat\theta_n-\theta_0\|)\). Compare a root-\(n\) covariance-parameter rate with a slower \(n^{-1/4}\) rate.
If \(\|\hat\theta_n-\theta_0\|=O_p(n^{-1/2})\), the perturbation in the root-\(n\) coefficient statistic is \(O_p(n^{-1/2})\), which vanishes. If instead the rate is \(O_p(n^{-1/4})\), the perturbation is \(O_p(n^{-1/4})\), which also vanishes. At \(n=256\), these rate factors are \(256^{-1/2}=0.0625\) and \(256^{-1/4}=0.250\), respectively.
The slower rate can therefore be sufficient for first-order equivalence, although its finite-sample perturbation may be larger. This comparison is a rate statement, not a guarantee about the realized error in any particular sample. It also does not establish a second-order approximation; that requires a more detailed expansion.
Common Pitfall and Practical Significance
A common mistake is to say, “The covariance parameters must be estimated at rate \(n^{-1/2}\), or FGLS is not asymptotically normal.” That conclusion is too strong. Root-\(n\) estimation is a familiar rate, but the first-order argument needs the covariance-induced changes in the normalized information and score to vanish. Under smoothness and stability, consistency can be enough.
The correction is not to assume that any consistent covariance estimate automatically works. Verify the model’s covariance family, the behavior of its inverse near the true parameter, the normalized design, and the weighted-score stability. The rate alone cannot rescue a covariance estimator whose limit is wrong or whose estimated weights become unstable. In applied work, this distinction explains why estimating a well-behaved covariance structure need not change the leading large-sample coefficient distribution, while still leaving finite-sample and covariance-estimation uncertainty to be considered separately.
Check Your Understanding
Use the model and conditions in this tutorial to answer the following questions.
- What two normalized quantities must be stable when replacing the true inverse covariance matrix by its estimate?
- Why does a positive definite limit \(Q\) matter when deriving the FGLS limit?
- Under the stated smoothness conditions, why can a consistent covariance-parameter estimator with rate \(n^{-1/3}\) suffice for first-order equivalence?
- In the two-group example, show how the group design values and error variances produce \(Q=1\).
- Does a covariance estimator’s convergence rate alone establish oracle equivalence? What additional conditions are needed?