Random-Intercept Covariance and Its GLS Form

Why a Shared Intercept Changes the Covariance

When several observations come from the same unit, they may share an unobserved unit-level influence. A school, for example, may have a persistent feature affecting every student observed there; a firm may have a persistent feature affecting each of its recorded years. If that shared influence is represented by a random intercept, observations from one unit are correlated even when their observation-specific errors are not. The resulting covariance is structured, not diagonal, and GLS uses that structure in its weighting.

The key is to distinguish the shared component from the observation-specific component, and to state their assumptions conditional on the regressors. Unconditional independence alone does not in general justify conditional covariance calculations.

Model and Conditional Assumptions

Index units by \(i=1,\ldots,G\), and observations within unit \(i\) by \(j=1,\ldots,T_i\). Let \(x_{ij}\) be a \(k\)-vector of regressors, and write the model as

Here \(\beta\) is the common coefficient vector, \(u_i\) is the random intercept shared by all observations in unit \(i\), and \(\varepsilon_{ij}\) is the observation-specific error. Let \(X\) denote the full design matrix. For the covariance formulas below, assume conditional on \(X\) that

• \(E(u_i\mid X)=0\), \(E(\varepsilon_{ij}\mid X)=0\), \(\operatorname{Var}(u_i\mid X)=\sigma_u^2\), and \(\operatorname{Var}(\varepsilon_{ij}\mid X)=\sigma_\varepsilon^2\), where the variances do not depend on \(i\) or \(j\);
• the \(u_i\)'s are conditionally independent across units, and the \(\varepsilon_{ij}\)'s are conditionally independent across observations and units;
• the two error components are conditionally independent of one another.

These assumptions imply the conditional cross-covariances used below are zero. They also give \(E(y_{ij}\mid X)=x_{ij}'\beta\). The covariance structure does not require normally distributed errors. For GLS, suppose the variance components are known, \(\sigma_\varepsilon^2>0\), and \(\sigma_u^2\geq 0\). The positive observation-specific variance ensures each covariance block is positive definite.

Deriving the Within-Unit and Between-Unit Covariances

For an observation, the two error components are conditionally independent and have conditional mean zero. Thus its conditional variance is

For two different observations \(j\ne \ell\) from the same unit, the shared \(u_i\) appears in both errors. The observation-specific errors have zero conditional covariance, and the cross-covariances between \(u_i\) and either observation-specific error are zero by conditional independence. Therefore,

For observations from distinct units \(i\ne r\), the shared components are different unit effects, and all relevant cross-unit and cross-component conditional covariances are zero. Hence

The correlation between two distinct observations in the same unit is consequently

This is the intraclass correlation under this model. It is the fraction of the marginal error variance attributable to the shared unit component; it is not the correlation between the regressors and the outcome.

The Covariance Matrix for One Unit

Stack the \(T_i\) outcomes in unit \(i\) into \(y_i\), and define \(\mathbf{1}_{T_i}\) as a \(T_i\)-vector of ones. The error vector is \(u_i\mathbf{1}_{T_i}+\varepsilon_i\). Its conditional covariance block is

The identity-matrix term contributes \(\sigma_\varepsilon^2\) to each diagonal entry. The matrix of ones contributes \(\sigma_u^2\) to every entry, including the diagonal. Thus the diagonal entries are \(\sigma_\varepsilon^2+\sigma_u^2\), and the off-diagonal entries are \(\sigma_u^2\). Across all units, conditional independence gives the block-diagonal covariance matrix \(V=\operatorname{diag}(V_1,\ldots,V_G)\).

For example, if \(T_i=3\), \(\sigma_\varepsilon^2=4\), and \(\sigma_u^2=2\), then

The diagonal value \(6\) is the variance of one observation; each off-diagonal value \(2\) is the covariance of two distinct observations in that unit.

Inverting the Covariance Block

The inverse is needed to construct GLS. The rank-one form of \(V_i\) gives a direct expression:

To verify it, set \(a=\sigma_\varepsilon^2\), \(b=\sigma_u^2\), and \(J=\mathbf{1}_{T_i}\mathbf{1}_{T_i}'\). Since \(J^2=T_iJ\), multiply \(aI+bJ\) by \(a^{-1}(I-cJ)\), where \(c=b/(a+T_i b)\). The product is

The coefficient of \(J\) vanishes because \(c(1+T_i b/a)=b/a\). This verifies the inverse. In the numerical block above, the formula gives \(V_i^{-1}=0.25I_3-0.05J\), with diagonal entries \(0.20\) and off-diagonal entries \(-0.05\). Multiplication by \(V_i\) gives the identity matrix.

The Corresponding GLS Estimator

Stack the observations and regressors across units as \(y\) and \(X\), so that \(y=X\beta+v\), with \(\operatorname{Var}(v\mid X)=V\). If \(V\) is known and \(X\) has full column rank, the GLS estimator is

Its conditional expectation is \(\beta\), and its conditional covariance matrix is

Indeed, substituting \(y=X\beta+v\) into the estimator gives \(\hat{\beta}_{GLS}=\beta+(X'V^{-1}X)^{-1}X'V^{-1}v\). Taking the conditional expectation uses \(E(v\mid X)=0\), yielding \(\beta\). For the covariance, substitute \(\operatorname{Var}(v\mid X)=V\); the middle matrix simplifies because \(V^{-1}VV^{-1}=V^{-1}\). The resulting covariance is \((X'V^{-1}X)^{-1}\).

The inverse covariance downweights directions in the data with larger error variance. Because observations in a unit share a component, they do not contribute as much independent information about \(\beta\) as the same number of uncorrelated observations would. The exact effect on coefficients depends on the regressors and on the unit sizes.

Interpretation and a Common Pitfall

The random-intercept structure is useful when theory or design suggests a common unobserved unit influence and when the conditional assumptions are credible. Its covariance matrix makes that dependence explicit, allowing GLS to use it when estimating regression coefficients. The structure alone does not establish that a random-intercept model is appropriate, nor does it make regressors exogenous: the conditional mean assumption \(E(v\mid X)=0\) remains essential for the stated GLS expectation.

• Pitfall: using unconditional independence to claim conditional zero covariance. It is not generally valid to say that errors are independent without specifying the conditioning set, then infer the required conditional covariances. A regressor can be related to error components in a way that changes their conditional dependence.
• Correction: State conditional independence given \(X\), as assumed here, or directly impose the zero conditional cross-covariances used in deriving \(V\). Then compute every covariance conditional on \(X\).
• Pitfall: placing \(\sigma_u^2\) only on the diagonal. The unit effect appears in every observation from that unit, so it contributes to both variances and within-unit covariances.
• Correction: Add \(\sigma_u^2\) to every entry of the unit block and add \(\sigma_\varepsilon^2\) only to its diagonal.

In particular, treating all observations as independent would discard the off-diagonal covariance and generally produce a different estimator and an incorrect model-based covariance calculation. GLS uses the full block structure; it does not simply multiply each observation by one independent-observation weight.