Deriving Nickell Bias In The Within Estimator

Why Within Estimation Can Be Biased in a Dynamic Panel

In a static panel model, subtracting each unit’s time average removes its fixed effect. In a dynamic panel, the same within transformation also subtracts averages of the lagged outcome and the error. Those averages link the transformed regressor to the transformed error. As a result, adding more units does not generally make the fixed-effects slope converge to the true lag coefficient when the number of time periods is held fixed.

This tutorial derives that result for a first-order dynamic panel with one lagged dependent variable. The central calculation is a covariance: the lagged outcome at a given date is uncorrelated with that date’s innovation, but the within-transformed lagged outcome is correlated with the within-transformed innovation. Under stationary conditions, this creates a bias of order \(1/T\).

Model and Assumptions

For units \(i=1,\ldots,N\), observe \(y_{it}\) for \(t=1,\ldots,T\), along with an initial value \(y_{i0}\). Consider

y_{it}=\alpha_i+\rho y_{i,t-1}+u_{it},\qquad t=1,\ldots,T.

Here \(\alpha_i\) is a unit-specific effect, \(\rho\) is the common coefficient of interest, and \(u_{it}\) is an innovation. The within estimator subtracts each unit’s time mean from both \(y_{it}\) and \(y_{i,t-1}\), then regresses the demeaned outcome on the demeaned lag.

Assumptions for the derivation: The units are independent and identically distributed across \(i\), with finite second moments. The innovations have mean zero, are serially uncorrelated with variance \(\sigma^2\), and are independent of the initial deviation from the unit’s stationary mean. Assume \(|\rho|<1\), and that the initial value is drawn from the stationary distribution conditional on \(\alpha_i\). The fixed effect may be correlated with the initial outcome. The panel has \(T\) usable outcome observations, and the probability limit below takes \(N\) to infinity while \(T\) is fixed.

Stationarity here applies to deviations from the unit-specific mean \(\alpha_i/(1-\rho)\). It gives the lagged outcome a constant variance and a geometric autocovariance. The within transformation removes \(\alpha_i\), so the derivation focuses on the dependence between the lagged deviations and the innovations.

The Within Slope and Its Bias

Define \(x_{it}=y_{i,t-1}\), and let \(\bar y_i=T^{-1}\sum_{t=1}^{T}y_{it}\), \(\bar x_i=T^{-1}\sum_{t=1}^{T}x_{it}\), and \(\bar u_i=T^{-1}\sum_{t=1}^{T}u_{it}\). The within estimator is

\hat\rho_{\mathrm{within}} = \frac{\sum_{i=1}^{N}\sum_{t=1}^{T}(x_{it}-\bar x_i)(y_{it}-\bar y_i)} {\sum_{i=1}^{N}\sum_{t=1}^{T}(x_{it}-\bar x_i)^2}.

Demeaning the model gives \(y_{it}-\bar y_i=\rho(x_{it}-\bar x_i)+(u_{it}-\bar u_i)\). Therefore, when the cross-sectional averages of the within moments converge as \(N\) grows, the probability limit of the slope is

\operatorname{plim}_{N\to\infty}\hat\rho_{\mathrm{within}} = \rho+ \frac{\sum_{t=1}^{T}E[(x_{it}-\bar x_i)(u_{it}-\bar u_i)]} {\sum_{t=1}^{T}E[(x_{it}-\bar x_i)^2]}.

The ratio is the asymptotic bias at fixed \(T\). It is a ratio of population within moments; it is not, in general, a formula for the expectation of a finite-\(N\) ratio.

Worked Example: The Covariance Appears After Demeaning

Consider just the numerator of the bias ratio. Since \(\sum_{t=1}^{T}(x_{it}-\bar x_i)=0\), its expectation for one unit can be written as

\sum_{t=1}^{T}E[(x_{it}-\bar x_i)(u_{it}-\bar u_i)] = \sum_{t=1}^{T}E[(x_{it}-\bar x_i)u_{it}].

For each \(t\), \(x_{it}=y_{i,t-1}\) depends on innovations dated before \(t\), so \(E[x_{it}u_{it}]=0\) under the assumptions. Thus the remaining contribution is \(-T E[\bar x_i\bar u_i]\). Take \(T=3\), \(\rho=0.5\), and \(\sigma^2=1\). The only nonzero covariances between a lagged outcome \(y_{s-1}\) and an innovation \(u_t\) occur when \(t<s\):

E\!\left[\left(\sum_{s=1}^{3}y_{i,s-1}\right) \left(\sum_{t=1}^{3}u_{it}\right)\right] = \sigma^2(2+\rho) = 2.5.

The factor \(1/T^2\) in \(E[\bar x_i\bar u_i]\), followed by multiplication by \(-T\), gives the within covariance sum

-T E[\bar x_i\bar u_i] = -\frac{2.5}{3} = -0.8333.

The original, undemeaned regressor has no contemporaneous covariance with the innovation. The negative covariance comes from the averages: the lagged-outcome average contains past innovations that also enter the innovation average.

Deriving the Covariance and Variance Moments

For the stationary process, the covariance between a lagged outcome and an innovation is

E[y_{i,s-1}u_{it}] = \begin{cases} \sigma^2\rho^{s-1-t}, & t\leq s-1,\\ 0, & t\geq s. \end{cases}

The first line follows by iterating the dynamic equation: the effect of \(u_{it}\) on \(y_{i,s-1}\) is multiplied by \(\rho\) once for each intervening period. Since the regressor’s contemporaneous covariance with \(u_{it}\) is zero, the expected within covariance sum is

\begin{aligned} \sum_{t=1}^{T}E[(x_{it}-\bar x_i)(u_{it}-\bar u_i)] &=-T E[\bar x_i\bar u_i]\\ &=-\frac{1}{T}\sum_{t=1}^{T-1}\sum_{s=t+1}^{T} \sigma^2\rho^{s-1-t}\\ &=-\frac{\sigma^2}{T}\sum_{k=0}^{T-2}(T-1-k)\rho^k. \end{aligned}

This expression is negative for \(0\leq\rho<1\). The same demeaning mechanism produces a negative covariance even when \(\rho=0\): in that case, the lagged outcome is an earlier innovation, which can enter the unit’s mean error.

To calculate the denominator, the stationary variance of \(x_{it}\) is \(\gamma_0=\sigma^2/(1-\rho^2)\), and its lag-\(h\) autocovariance is \(\gamma_h=\gamma_0\rho^h\). The identity \(\sum_t(x_{it}-\bar x_i)^2=\sum_t x_{it}^2-T\bar x_i^2\) then gives

\begin{aligned} \sum_{t=1}^{T}E[(x_{it}-\bar x_i)^2] &=T\gamma_0-\frac{1}{T}\left(T\gamma_0+ 2\sum_{h=1}^{T-1}(T-h)\gamma_h\right)\\ &=\gamma_0\left[ T-\frac{1}{T}\left\{T+2\sum_{h=1}^{T-1}(T-h)\rho^h\right\} \right]. \end{aligned}

Combining these moments yields the stationary fixed-\(T\) probability limit:

\operatorname{plim}_{N\to\infty}\hat\rho_{\mathrm{within}} = \rho- \frac{\dfrac{\sigma^2}{T}\sum_{k=0}^{T-2}(T-1-k)\rho^k} {\dfrac{\sigma^2}{1-\rho^2} \left[ T-\dfrac{1}{T}\left\{T+2\sum_{h=1}^{T-1}(T-h)\rho^h\right\} \right]}.

The bias is the negative fraction in this expression. For a particular short panel it can be substantial; it is not generally equal to a simple \(1/T\) approximation.

Worked Example: Exact Moment Bias for Three Periods

Continue with \(T=3\), \(\rho=0.5\), and \(\sigma^2=1\). The numerator covariance sum is \(-0.8333\), as calculated above. The stationary lagged-outcome variance is \(\gamma_0=1/(1-0.5^2)=1.3333\). The within-regressor variance sum is

\begin{aligned} \sum_{t=1}^{3}E[(x_{it}-\bar x_i)^2] &=1.3333\left[3-\frac{1}{3}\{3+2(2(0.5)+1(0.25))\}\right]\\ &=1.3333(3-5.5/3)\\ &=1.5556. \end{aligned}

The probability-limit bias is therefore \(-0.8333/1.5556=-0.5357\), and the within slope converges, as \(N\) grows with \(T=3\), to \(0.5-0.5357=-0.0357\). This is a population-moment limit under the stated assumptions, not a claim that every small sample will have that slope.

Why the Bias Is of Order \(1/T\)

The covariance numerator remains of constant order as \(T\) grows, while the within-regressor variance sum grows proportionally to \(T\). To see the numerator’s scale, divide its weighted geometric sum by \(T\):

\frac{1}{T}\sum_{k=0}^{T-2}(T-1-k)\rho^k \longrightarrow \sum_{k=0}^{\infty}\rho^k = \frac{1}{1-\rho}, \qquad |\rho|<1.

The denominator divided by \(T\) converges to \(\gamma_0=\sigma^2/(1-\rho^2)\). The bias ratio consequently has leading term

\operatorname{plim}_{N\to\infty}\hat\rho_{\mathrm{within}}-\rho = -\frac{1+\rho}{T}+O(T^{-2}), \qquad T\to\infty,\quad |\rho|<1.

Thus the bias vanishes as the time dimension grows, but only at rate \(1/T\) under these stationary assumptions. The often-used expression \(-(1+\rho)/(T-1)\) has the same leading large-\(T\) order; it is an approximation, not the exact finite-\(T\) moment ratio derived above.

Worked Example: Zero Persistence Still Gives Within Bias

Set \(\rho=0\) and take \(T=5\). The covariance formula reduces to

-\frac{\sigma^2}{5}\sum_{k=0}^{3}(4-k)0^k = -\frac{4\sigma^2}{5}.

Here the \(k=0\) term is \(4\); all positive powers of \(\rho\) vanish. With \(\rho=0\), the lagged outcomes are uncorrelated over time, so the expected within-regressor variance sum is \((T-1)\sigma^2=4\sigma^2\). The bias ratio is \((-4\sigma^2/5)/(4\sigma^2)=-0.2000\). The within slope therefore converges to \(-0.2000\), although the true lag coefficient is zero. This confirms that the finite-\(T\) bias is created by within demeaning, not only by persistence in the outcome.

What the Result Does—and Does Not—Say

The key asymptotic distinction is between the number of units and the number of periods. Increasing \(N\) at fixed \(T\) makes the estimator converge to its fixed-\(T\) probability limit, which differs from \(\rho\). It does not eliminate the within covariance in the numerator. Increasing \(T\), under the stationary assumptions used here, reduces the bias at rate \(1/T\).

The exact finite-\(T\) expression depends on the assumptions that determine the covariances, including the initial-condition distribution and innovation process. The leading-order result should not be treated as a universal exact correction. In particular, changing the initialization or allowing serial correlation in the innovations changes the covariance calculations.

Common Mistakes

Claiming that the lagged outcome is correlated with its own innovation. Under the model assumptions, \(y_{i,t-1}\) is uncorrelated with \(u_{it}\). The problem is the correlation between the demeaned regressor and the demeaned error.
Assuming a large \(N\) removes this bias. At fixed \(T\), the within moments converge to their population values, including the nonzero covariance. More units reduce sampling variability around the biased probability limit.
Using the leading approximation as an exact short-panel result. The term \(-(1+\rho)/T\) describes the leading order as \(T\) grows. For short panels, calculate the finite-\(T\) moment ratio if its value is needed.
Describing the result as evidence of a causal effect. The derivation concerns the probability limit of a slope under a specified dynamic model and assumptions; it does not by itself establish a causal interpretation.

Key takeaway: The within transformation removes the unit effect but makes the lagged dependent variable correlated with the transformed innovation. In a stationary AR(1) panel, this produces a negative fixed-\(T\) probability-limit bias whose leading magnitude is \((1+\rho)/T\).

Check Your Understanding

Use the assumptions and moment expressions above to answer the following.

Why is \(E[y_{i,t-1}u_{it}]=0\) compatible with a nonzero within covariance between the lagged outcome and the innovation?
For \(T=4\) and \(\rho=0\), calculate the within covariance sum and the within-regressor variance sum in terms of \(\sigma^2\). What is their ratio?
In the stationary derivation, which covariance assumption allows the lagged-outcome variance to be written as \(\sigma^2/(1-\rho^2)\)?
What happens to the within estimator’s probability limit when \(N\) grows but \(T\) stays fixed?
Why is \(-(1+\rho)/T\) not generally the exact bias for a short panel?