Estimating Random-Effects Variance Components

Why Estimate Variance Components?

In a random-effects panel model, the composite error contains both a persistent unit component and a time-varying component. Their relative sizes determine how strongly feasible generalized least squares (GLS) quasi-demeans the data. Estimating the regression coefficients therefore requires estimates of both variance components, not just a regression residual variance.

The within and between parts of a panel provide different information. Demeaning removes the unit effect, leaving information about the idiosyncratic-error variance. Unit averages retain the unit effect, but also contain an averaged idiosyncratic error; this contribution must be subtracted when estimating the unit-effect variance.

Model and Assumptions

Consider a balanced panel with $N$ units observed in each of $T$ periods. The model is

y_{it}=x_{it}'\beta+a_i+u_{it},\qquad i=1,\ldots,N,\quad t=1,\ldots,T.

Here $y_{it}$ is the outcome, $x_{it}$ is a vector of regressors, $a_i$ is the unit effect, and $u_{it}$ is the idiosyncratic error. Write the two variance components as $\sigma_a^2=\operatorname{Var}(a_i)$ and $\sigma_u^2=\operatorname{Var}(u_{it})$.

Assumptions for the variance-component calculations: The regressors are strictly exogenous with respect to the errors, so $E[a_i\mid X_i]=0$ and $E[u_{it}\mid X_i,a_i]=0$, where $X_i$ denotes all regressors for unit $i$. Assume also that $\operatorname{Var}(a_i\mid X_i)=\sigma_a^2$ and $\operatorname{Var}(u_i\mid X_i,a_i)=\sigma_u^2I_T$. The $a_i$ are independent across units with common variance $\sigma_a^2$. The $u_{it}$ are independent across units and periods, have common variance $\sigma_u^2$, and are independent of the unit effects. The regressors used in each auxiliary regression have the required rank. For the conditional residual-sum-of-squares arguments below, also require, conditional on the full panel design $X$, that $E[\mathbf a\mid X]=0$, $E[\mathbf u\mid X]=0$, $\operatorname{Var}(\mathbf a\mid X)=\sigma_a^2I_N$, $\operatorname{Var}(\mathbf u\mid X)=\sigma_u^2I_{NT}$, and $\operatorname{Cov}(\mathbf a,\mathbf u\mid X)=0$, where $\mathbf a=(a_1,\ldots,a_N)'$ and $\mathbf u=(u_{11},\ldots,u_{NT})'$ is stacked by unit. These full-design conditions ensure the within and between residual-sum-of-squares arguments below apply; if the error structure differs, these formulas need not estimate the intended components.

Define unit averages by $\bar y_i=T^{-1}\sum_t y_{it}$ and $\bar x_i=T^{-1}\sum_t x_{it}$. The within equation subtracts unit averages from each observation, eliminating $a_i$. The between equation regresses $\bar y_i$ on $\bar x_i$, with an intercept where appropriate. Its error is $a_i+\bar u_i$, where $\bar u_i=T^{-1}\sum_t u_{it}$.

Residual-Based Component Estimates

Fit the within regression and let $S_W$ be its residual sum of squares. Let $r_W$ be the rank of the within-regressor matrix, excluding any intercept eliminated by demeaning. There are $N(T-1)$ independent within-space dimensions before fitting the regressors. The residual degrees of freedom are therefore $d_W=N(T-1)-r_W$, and the method-of-moments estimate is

\widehat{\sigma}_u^2=\frac{S_W}{d_W}.

Next, fit the between regression of unit averages on the included unit-average regressors, and let $S_B$ be its residual sum of squares. If the between design matrix, including its intercept, has rank $r_B$, its residual degrees of freedom are $d_B=N-r_B$. The between residual variance estimates $\sigma_a^2+\sigma_u^2/T$, not $\sigma_a^2$ alone. Thus the unconstrained estimate of the unit-effect variance is the between residual variance minus the estimated contribution from averaged idiosyncratic errors.

Method-of-moments estimates: With $d_W=N(T-1)-r_W$ and $d_B=N-r_B$, $$ \widehat{\sigma}_u^2=\frac{S_W}{d_W},\qquad \widehat{\sigma}_a^2=\max\left\{0,\ \frac{S_B}{d_B}-\frac{\widehat{\sigma}_u^2}{T}\right\}. $$ The maximum with zero enforces the nonnegativity required of a variance component.

Why These Residual Moments Work

The within residuals are not independent observations with variance $\sigma_u^2$. For one unit, define $\bar u_i=T^{-1}\sum_t u_{it}$. Under the stated assumptions, $\operatorname{Var}(\bar u_i)=\sigma_u^2/T$, and $\operatorname{Cov}(u_{it},\bar u_i)=\sigma_u^2/T$. Therefore

\operatorname{Var}(u_{it}-\bar u_i)=\sigma_u^2\left(1-\frac{1}{T}\right),\qquad \operatorname{Cov}(u_{it}-\bar u_i,u_{is}-\bar u_i)=-\frac{\sigma_u^2}{T}\quad(t\ne s).

Nevertheless, the covariance operator on the within space is $\sigma_u^2$ times the identity. To see why, let $M=I_T-\mathbf{1}\mathbf{1}'/T$ be the demeaning matrix. It is symmetric and idempotent, has rank $T-1$, and satisfies $M\mathbf{1}=0$. Since $\operatorname{Var}(u_i\mid X_i)=\sigma_u^2I_T$, the demeaned error has covariance $\sigma_u^2M$. Fitting the within regressors removes $r_W$ further dimensions, leaving a residual projection matrix of rank $d_W$. The expected residual sum of squares is its variance scale times its rank:

E[S_W\mid X]=\sigma_u^2\,\operatorname{tr}(P_W) =\sigma_u^2\{N(T-1)-r_W\}.

This justifies dividing $S_W$ by the within residual degrees of freedom, while also distinguishing the variance of an individual demeaned error from the variance scale on the within space.

For the between equation, the error in each unit average is $a_i+\bar u_i$. Independence gives

\operatorname{Var}(a_i+\bar u_i\mid X_i)=\sigma_a^2+\frac{\sigma_u^2}{T}.

Across units these errors are independent. The between residual projection has rank $d_B$, so $E[S_B\mid X]=d_B(\sigma_a^2+\sigma_u^2/T)$. Dividing by $d_B$ estimates the combined variance; subtracting $\widehat{\sigma}_u^2/T$ isolates the unit-effect component. These expectation arguments rely on the model assumptions and correct auxiliary-regression specification.

Worked Example: Estimating the Idiosyncratic Variance

Suppose a hypothetical balanced panel has $N=6$ units and $T=3$ periods. A within regression with one linearly independent regressor has residual sum of squares $S_W=48$. Assume the within design has rank one and the model assumptions above hold.

There are $N(T-1)=6(2)=12$ within-space dimensions. After estimating the one within-regression coefficient, the residual degrees of freedom are $d_W=12-1=11$. Thus

\widehat{\sigma}_u^2=\frac{48}{11}=4.3636.

The estimate is $4.364$ outcome-units squared. It estimates the variance scale for the within space; it is not the variance of an individual demeaned residual, which under this model is $(1-1/3)\sigma_u^2$.

Worked Example: Estimating Both Components

For a separate hypothetical panel, take $N=10$, $T=3$, and one time-varying regressor. Suppose the within regression has rank $r_W=1$ and residual sum of squares $S_W=190$. Suppose the between regression includes an intercept and the unit-average regressor, has rank $r_B=2$, and has residual sum of squares $S_B=160$. Assume both designs have the stated ranks and the required exogeneity and variance assumptions hold.

The within residual degrees of freedom are $10(3-1)-1=19$, giving

\widehat{\sigma}_u^2=\frac{190}{19}=10.

The between residual degrees of freedom are $10-2=8$, so the between residual variance estimate is $160/8=20$. Each unit average contains idiosyncratic-error variance $\widehat{\sigma}_u^2/T=10/3$. Subtracting this contribution gives

\widehat{\sigma}_a^2 =\max\left\{0,\ 20-\frac{10}{3}\right\} =\frac{50}{3}\approx16.67.

The estimated unit-effect variance is $16.67$ outcome-units squared. It is not the between residual variance of $20$, because that residual variance also includes the averaged idiosyncratic error.

From Estimated Components to Feasible GLS

For the composite error $a_i+u_{it}$, the variance within a period is $\sigma_a^2+\sigma_u^2$, and the covariance between distinct periods for the same unit is $\sigma_a^2$. In a balanced panel, the random-effects quasi-demeaning parameter is

\theta=1-\sqrt{\frac{\sigma_u^2}{\sigma_u^2+T\sigma_a^2}}.

Feasible GLS replaces the unknown variances with their estimates:

\widehat{\theta} =1-\sqrt{\frac{\widehat{\sigma}_u^2} {\widehat{\sigma}_u^2+T\widehat{\sigma}_a^2}}.

The transformed outcome and regressors are $y_{it}^*=y_{it}-\widehat{\theta}\bar y_i$ and $x_{it}^*=x_{it}-\widehat{\theta}\bar x_i$. A constant, if included, is transformed to $1-\widehat{\theta}$, rather than removed. The transformation is called feasible because it uses estimated, rather than known, variance components.

Worked Example: Calculating the Feasible Transformation

Use the component estimates from the preceding example: $\widehat{\sigma}_u^2=10$, $\widehat{\sigma}_a^2=50/3$, and $T=3$. Then

\widehat{\theta} =1-\sqrt{\frac{10}{10+3(50/3)}} =1-\sqrt{\frac{1}{6}} \approx0.5918.

For illustration, suppose one hypothetical unit has outcomes $12,15,18$, with mean $\bar y_i=15$, and regressor values $1,2,4$, with mean $\bar x_i=7/3$. Its first transformed outcome and regressor are

y_{i1}^*=12-(0.5918)(15)\approx3.123,\qquad x_{i1}^*=1-(0.5918)(7/3)\approx-0.3809.

The same calculation is applied to every observation, using that unit's averages. GLS is then performed on the transformed panel. The quasi-demeaning parameter is estimated from the variance components; it is not chosen by inspecting which transformation produces a preferred coefficient.

Common Mistakes and Practical Checks

Using the wrong within variance. An individual demeaned error has variance $\sigma_u^2(1-1/T)$, and demeaned errors from distinct periods in the same unit have covariance $-\sigma_u^2/T$. The within residual sum-of-squares formula is justified by the covariance operator on the within space, not by treating each demeaned error as having variance $\sigma_u^2$.
Forgetting the average-error contribution. The between residual variance estimates $\sigma_a^2+\sigma_u^2/T$. Subtract $\widehat{\sigma}_u^2/T$, not $\widehat{\sigma}_u^2$, to estimate the unit-effect variance.
Using observation counts instead of residual degrees of freedom. Subtract the rank of the fitted design from the relevant dimension. In particular, the between regression's intercept counts toward $r_B$.
Rounding too early. Keep full precision in intermediate calculations and round only the reported result. For example, $80/6-(96/14)/3=11.047619\ldots$, which rounds to $11.048$, not $11.047$.

Sampling variation can make the unconstrained estimate $S_B/d_B-\widehat{\sigma}_u^2/T$ negative even though a variance cannot be negative. The stated method replaces such a value with zero; a zero estimate is a feature of this constrained calculation, not proof that the population unit-effect variance is exactly zero. Also, these classical residual formulas do not automatically accommodate serial correlation, unequal variances, or other departures from the assumed error structure.

Key takeaway: Estimate $\sigma_u^2$ from within residual variation, estimate $\sigma_a^2+\sigma_u^2/T$ from between residual variation, and subtract the averaged idiosyncratic contribution. Use the resulting nonnegative component estimates in the quasi-demeaning parameter for feasible GLS.

Check Your Understanding

Answer each question using the balanced-panel assumptions stated above.

For $N=12$, $T=4$, and within-regressor rank $r_W=2$, what are the within residual degrees of freedom?
Why does the between residual variance estimate $\sigma_a^2+\sigma_u^2/T$ rather than $\sigma_a^2$ alone?
If $S_B/d_B=9$, $\widehat{\sigma}_u^2=6$, and $T=3$, calculate the unconstrained estimate of $\sigma_a^2$.
What is the variance of an individual demeaned idiosyncratic error, and how does it differ from the variance scale on the within space?
Which variance estimates are substituted into the formula for $\widehat{\theta}$ in feasible GLS?