Joint Limits for Coefficients and Residual Variance

Why a Joint Limit Is Needed

OLS coefficients and the residual-based estimate of the error variance are computed from the same sample. Their separate limiting distributions do not, by themselves, describe how the two estimates fluctuate together. That relationship matters when a later calculation uses both quantities, and it determines whether their Gaussian limits can be treated as independent.

The key point is that the coefficient estimate is driven, to first order, by the regression scores \(X_i\varepsilon_i\), while the residual variance estimate is driven by \(\varepsilon_i^2-\sigma^2\). Conditional homoskedasticity determines the coefficient covariance, but it does not generally make these two kinds of fluctuations uncorrelated. Their cross-covariance depends on a third moment.

Setup and Assumptions

For each observation, let \(X_i\) be a \(p\)-dimensional random regressor vector and let \[ Y_i=X_i'\beta_0+\varepsilon_i, \] where \(\beta_0\) is fixed. Assume the pairs \((X_i,\varepsilon_i)\) are independent and identically distributed. The dimension \(p\) is fixed as \(n\) increases.

Write \(Q_n=n^{-1}\sum_{i=1}^nX_iX_i'\). On the event that the sample design has full column rank, the OLS estimator is \(\hat\beta=(\sum_iX_iX_i')^{-1}\sum_iX_iY_i\). The assumptions imply \(Q_n\xrightarrow{p}Q\), so the sample design is nonsingular with probability tending to one. Define the residual-based variance estimator with degrees-of-freedom correction by \[ \hat\sigma^2=\frac{1}{n-p}\sum_{i=1}^n(Y_i-X_i'\hat\beta)^2. \] The same limit below holds if the denominator is \(n\).

The Joint Limit

The upper-left block is the usual homoskedastic OLS covariance. The lower-right entry describes the first-order variation in squared errors. The off-diagonal block is the feature specific to the joint limit: it records whether large or small coefficient-score fluctuations tend to occur alongside fluctuations in squared errors.

Derivation of the Joint Limit

First, the regression equation gives the exact representation \[ \sqrt n(\hat\beta-\beta_0) =Q_n^{-1}\frac{1}{\sqrt n}\sum_{i=1}^nX_i\varepsilon_i. \] The random vector \((X_i\varepsilon_i,\varepsilon_i^2-\sigma^2)\) has mean zero and finite covariance. Indeed, conditional homoskedasticity gives \(E[\|X_i\varepsilon_i\|^2]=\sigma^2E[\|X_i\|^2]<\infty\), and the fourth-moment assumption controls the second moment of \(\varepsilon_i^2\). The multivariate central limit theorem therefore applies to its sample sum. The law of large numbers gives \(Q_n^{-1}\xrightarrow{p}Q^{-1}\).

To verify the expansion, put \(\delta_n=\hat\beta-\beta_0\). The residual sum of squares equals \[ \sum_{i=1}^n(Y_i-X_i'\hat\beta)^2 =\sum_{i=1}^n\varepsilon_i^2-n\delta_n'Q_n\delta_n. \] The coefficient representation and the central limit theorem imply \(\delta_n=O_p(n^{-1/2})\), while \(Q_n=O_p(1)\). Consequently \(n\delta_n'Q_n\delta_n=O_p(1)\). Dividing by \(n\) and multiplying by \(\sqrt n\) makes this fitting correction \(O_p(n^{-1/2})=o_p(1)\). Also, replacing the denominator \(n\) by \(n-p\) changes the estimator by \(O_p(n^{-1})\), which is negligible after multiplication by \(\sqrt n\). This proves the expansion.

It remains to calculate the covariance of one summand in the joint central limit theorem. Conditional mean zero gives \(E[X_i\varepsilon_i]=0\). The coefficient-score covariance is \[ E[X_iX_i'\varepsilon_i^2] =E\!\left[X_iX_i'E[\varepsilon_i^2\mid X_i]\right] =\sigma^2Q. \] The variance of the squared-error component is \(\tau^2\). Finally, \[ \operatorname{Cov}(X_i\varepsilon_i,\varepsilon_i^2-\sigma^2) =E[X_i\varepsilon_i^3]-\sigma^2E[X_i\varepsilon_i] =m_3. \] Multiplication of the coefficient score by \(Q^{-1}\), followed by Slutsky’s theorem and the variance expansion, gives the stated joint limit. \(\square\)

When Are the Limits Independent?

The limiting vector is jointly Gaussian, possibly with a singular covariance matrix. For jointly Gaussian subvectors, zero cross-covariance is equivalent to independence. Since \(Q\) is invertible, the coefficient limit and the variance-estimator limit are therefore independent exactly when \[ m_3=E[X_i\varepsilon_i^3]=0. \] A useful sufficient condition is \(E[\varepsilon_i^3\mid X_i]=0\) almost surely. Conditional symmetry of the errors about zero is one way to ensure this condition. Conditional homoskedasticity alone says nothing about the conditional third moment, so it does not guarantee independence.

When \(E[\varepsilon_i^3\mid X_i]\) is nonzero, the cross-covariance is \(Q^{-1}E[X_iE[\varepsilon_i^3\mid X_i]]\). It can still equal zero through cancellation across regressor values, but that must be checked; it does not follow merely from constant conditional variance. These statements concern the limiting distributions. They do not assert exact finite-sample independence of \(\hat\beta\) and \(\hat\sigma^2\).

Common Mistake and Practical Interpretation

• Wrong reasoning: “The errors are conditionally homoskedastic, so coefficient estimation and residual-variance estimation are asymptotically independent.” Constant conditional variance establishes the coefficient covariance block, but it does not imply \(E[X_i\varepsilon_i^3]=0\).
• Correction: Calculate or justify the third-moment vector \(m_3\). Conditional symmetry is sufficient for it to vanish; conditional homoskedasticity by itself is not.
• Another pitfall: Treating the fitted residual squares as if their first-order fluctuation were exactly the raw error squares ignores the fitted-coefficient correction. The expansion above shows why that correction is negligible at the \(\sqrt n\) scale under the stated fixed-dimensional assumptions.

The joint limit is useful whenever an argument combines coefficient uncertainty with an estimated error variance. It identifies which covariance terms are present before any such combination is made. In particular, independence should be concluded from the joint Gaussian limit and its cross-covariance, not assumed from the separate marginal limits.