Inference With Serially Correlated Outcomes

Why Serial Dependence Matters for Difference-in-Differences

Difference-in-differences compares changes over time across treatment and comparison groups. Its identifying assumptions concern the causal comparison; its standard errors concern how variable that comparison would be across repeated samples. These are separate questions. Even when the parallel-trends assumption is credible, conventional standard errors can be wrong if outcomes for the same group are correlated over time.

For example, a district may have persistently high or low outcomes because of stable local conditions. Its pre-policy and post-policy outcomes will then tend to move together. Treating each district-year observation as independent discards that dependence. When treatment is assigned to districts, the independent information for inference is primarily the number of districts assigned—not the number of people or district-year records observed.

Definition: A cluster is a group of observations whose errors may be dependent within the group but are assumed independent of errors in other groups. In a difference-in-differences design, cluster at the level at which treatment is assigned when outcomes may be correlated within that level. Serial correlation is dependence across time within a unit or cluster.

The two-period difference-in-differences estimand and its regression representation are useful starting points. For clarity, consider a panel with groups \(g=1,\ldots,G\), observed at times \(t\). Let \(D_g=1\) for a group assigned to treatment and \(D_g=0\) otherwise, and let \(Post_t=1\) in the post-treatment period. A standard regression is

Y_{gt}=\alpha_g+\lambda_t+\tau(D_g\times Post_t)+\varepsilon_{gt}.

Here \(\alpha_g\) are group fixed effects, \(\lambda_t\) are time effects, and \(\tau\) is the difference-in-differences coefficient. The regression error \(\varepsilon_{gt}\) may be correlated over time within group \(g\). Clustering by group permits this dependence without requiring a specific model for its form. It still requires independence across clusters for the usual large-cluster justification.

How Serial Correlation Changes the Variance

The algebra of a two-period change shows why independence over time matters. Write the error change for a group as \(\Delta\varepsilon_g=\varepsilon_{g,post}-\varepsilon_{g,pre}\). Its variance is

\operatorname{Var}(\Delta\varepsilon_g) =\operatorname{Var}(\varepsilon_{g,post}) +\operatorname{Var}(\varepsilon_{g,pre}) -2\operatorname{Cov}(\varepsilon_{g,post},\varepsilon_{g,pre}).

An independence calculation sets the covariance term to zero. If the covariance is positive, that calculation overstates the variance of a single change. But in a difference-in-differences regression the error covariance also affects the uncertainty of the treatment-control contrast and the regression residuals. The direction of the standard-error error is not universal. In common designs with persistent group-level shocks and treatment assigned at group level, ignoring within-group dependence often understates uncertainty and can substantially inflate rejection rates.

Worked Example: Variance of a Two-Period Change

Suppose, hypothetically, that a group's outcome error has variance \(4\) in each period and correlation \(0.75\) between pre and post. What is the variance of its error change, and what would an independence calculation give?

The covariance is \(0.75\sqrt{4\cdot4}=3\). Substituting into the change-variance formula gives

\operatorname{Var}(\Delta\varepsilon_g)=4+4-2(3)=2.

If the two errors were incorrectly treated as independent, the calculation would be \(4+4=8\), four times as large for this single change. This example does not imply that conventional difference-in-differences standard errors are generally too large: their error depends on the complete covariance structure and the design. It demonstrates why the covariance cannot simply be assumed away.

Deriving the Cluster-Robust Sandwich

Consider the linear regression \(Y=X\beta+\varepsilon\), with observations partitioned into \(G\) clusters. Let \(X_g\) and \(\varepsilon_g\) collect the regressors and errors in cluster \(g\). Assume the regression is correctly specified for the coefficient being estimated, clusters are independent of one another, and within-cluster errors may have arbitrary covariance. The OLS estimator satisfies

\hat{\beta}-\beta =(X'X)^{-1}\sum_{g=1}^{G}X_g'\varepsilon_g.

Because the clusters are independent, the variance of the sum is the sum of the cluster variances. Thus, conditional on the regressors,

\operatorname{Var}(\hat{\beta}\mid X) =(X'X)^{-1} \left(\sum_{g=1}^{G}X_g'\Omega_gX_g\right) (X'X)^{-1},

where \(\Omega_g=\operatorname{Var}(\varepsilon_g\mid X)\) may have nonzero off-diagonal entries. Replacing the unknown errors with OLS residuals \(\hat{\varepsilon}_g\) gives the cluster-robust variance estimator. Its middle component, called the meat of the sandwich, is the sum of outer products of cluster score contributions:

\widehat{\operatorname{Var}}_{\mathrm{CR}}(\hat{\beta}) =(X'X)^{-1} \left[ \sum_{g=1}^{G} (X_g'\hat{\varepsilon}_g)(X_g'\hat{\varepsilon}_g)' \right] (X'X)^{-1}.

This estimator allows arbitrary heteroskedasticity and serial correlation within a cluster. It does not allow arbitrary correlation across clusters, and it does not repair bias in the estimated causal effect from a failure of parallel trends, anticipation, or other identification problems.

For a single treatment coefficient, the same idea can be expressed after partialling out the other regressors. Let \(\tilde D_{ig}\) be the residual from projecting the treatment regressor on the other regressors, including fixed effects. The cluster-robust variance for its coefficient is

\widehat{\operatorname{Var}}_{\mathrm{CR}}(\hat{\tau}) = \frac{\displaystyle\sum_{g=1}^{G} \left(\sum_i\tilde D_{ig}\hat{\varepsilon}_{ig}\right)^2} {\displaystyle\left(\sum_{g=1}^{G}\sum_i\tilde D_{ig}^{\,2}\right)^2}.

The inner sum is the cluster’s score for the coefficient. Squaring the sum preserves the contribution of within-cluster covariation; summing separate squared observation-level contributions would discard the cross-products that serial dependence creates.

Worked Example: Cluster Scores in a Two-by-Two Contrast

Suppose four independent groups have hypothetical outcome changes of \(5,7,1,\) and \(3\) units. The first two are controls and the last two are treated, so \(D_g=(0,0,1,1)\). Regress the change on an intercept and \(D_g\). This is the group-level form of the two-by-two difference-in-differences contrast.

The control mean is \((5+7)/2=6\), and the treated mean is \((1+3)/2=2\). Therefore the intercept is \(6\), the treatment coefficient is \(\hat{\tau}=2-6=-4\) units, and the fitted residuals are \((-1,1,-1,1)\) units.

The mean treatment indicator is \(\bar D=0.5\). In this two-regressor model, the treatment regressor after partialling out the intercept is \(D_g-\bar D\), and \(\sum_g(D_g-\bar D)^2=1\). The four cluster scores are

(0-0.5)(-1)=0.5,\quad (0-0.5)(1)=-0.5,\quad (1-0.5)(-1)=-0.5,\quad (1-0.5)(1)=0.5.

Each score is in outcome units. The uncorrected sandwich variance and standard error are therefore

\widehat{\operatorname{Var}}_{\mathrm{CR}}(\hat{\tau}) =\frac{0.5^2+(-0.5)^2+(-0.5)^2+0.5^2}{1^2} =1, \qquad SE_{\mathrm{CR}}(\hat{\tau})=\sqrt{1}=1.

The point estimate is \(-4\) units, and its uncorrected cluster-robust standard error is \(1\) unit. This calculation illustrates the sandwich, not trustworthy significance testing: there are only four independent clusters. A common finite-sample correction would multiply the variance by \(G/(G-1)\times(N-1)/(N-K)=\frac{4}{3}\times\frac{3}{2}=2\), where \(N=4\) and \(K=2\), giving a corrected standard error of \(\sqrt{2}\approx1.414\). Software corrections and degrees-of-freedom conventions can differ; none makes four clusters equivalent to a large number of clusters.

Choosing the Cluster Level and Counting Clusters

The cluster level should follow the assignment mechanism and the plausible dependence in outcomes. If a policy is assigned to districts, observations on people within a district do not create independent treatment assignments. With repeated observations, the district’s outcomes can also share shocks across years. Clustering only by person would not generally capture dependence between people in the same treated district.

Worked Example: Many Records, Few Assignments

Imagine a hypothetical evaluation with 8 districts, 4 assigned to a policy and 4 to comparison, and 100 people observed in each district. There are \(8\times100=800\) person-level records per period, but the policy assignment varies across only 8 independent districts. The relevant cluster count is therefore 8, not 800.

For scale, under an illustrative equal-size, exchangeable-correlation model with \(m=100\) observations per district and within-district correlation \(\rho=0.20\), the variance inflation factor for an average is \(1+(m-1)\rho=1+99(0.20)=20.8\). The corresponding standard-error factor is \(\sqrt{20.8}\approx4.561\) relative to an independent-observation calculation for that average. This design-effect calculation is only an illustration, not an exact difference-in-differences standard error. Its lesson is that a large record count need not provide a large amount of independent information.

Cluster-robust inference is justified by having many independent clusters, not merely many observations within each cluster. With few clusters, the estimated cluster-score distribution is based on few terms, so the usual large-sample normal or \(t\) approximations may be poor. A small-sample multiplier can adjust scale but cannot manufacture independent information. Report the number of clusters, identify the assignment level, and interpret conventional p-values and confidence intervals cautiously when that number is limited. There is no universal cluster-count threshold that guarantees reliable inference.

Common Pitfalls

Clustering at the observation level. Treating every person-year as independent ignores shared group shocks. Cluster at the treatment-assignment level when that is where treatment varies and dependence is plausible.
Assuming clustering always increases the standard error. The covariance structure determines the change; cluster-robust estimates can be larger or smaller than conventional estimates.
Confusing robust inference with causal identification. Cluster-robust standard errors address uncertainty under within-cluster dependence. They do not establish parallel trends or remove confounding.
Counting observations instead of independent clusters. Hundreds of people within a handful of assigned districts do not amount to hundreds of independent assignments.

Key takeaway: In difference-in-differences, serial correlation can invalidate standard errors that treat observations as independent. Cluster-robust inference aggregates regression scores within independent treatment-assignment clusters, and its large-sample reliability depends on having enough such clusters.

Check Your Understanding

Use these questions to check both the variance logic and the design implications.

If pre- and post-period errors have positive covariance, which term in the variance of an outcome change accounts for it?
A program is assigned to schools, with many pupils measured in each school. What is the natural clustering level, and why is the pupil count not the cluster count?
In the cluster sandwich, why are observation-level score contributions summed within cluster before taking the outer product?
Does a cluster-robust standard error protect against a violation of parallel trends? Explain.
What limitation remains when a study has many observations but only a few independent assignment clusters?