Common Mistakes in Clustered and HAC Inference

Why Robust Inference Can Still Be Wrong

A standard error can be calculated correctly for the wrong dependence structure. Clustering at an inappropriate level, treating a large observation count as a large number of independent units, or truncating serial covariances too aggressively can all produce misleading uncertainty. Even after estimating a defensible covariance matrix, a test can still be miscalibrated if its reference distribution does not suit the design and sample size.

The goal here is diagnostic: ask what units can reasonably be treated as independent, how much independent information the data contain, which dependence the covariance estimator includes, and what distribution the reported test actually uses. Robust covariance estimation addresses specified forms of dependence; it does not make every modeling or inferential choice harmless.

Set Up the Dependence Problem

Consider the linear regression model \(y=X\beta+u\), where \(y\) is an \(n\)-vector of outcomes, \(X\) is an \(n\times k\) regressor matrix, \(\beta\) is the coefficient vector, and \(u\) is the error vector. Suppose the coefficient estimator is ordinary least squares, with \(\hat\beta=(X'X)^{-1}X'y\). Inference requires a covariance estimator for \(\hat\beta\), not just an estimate of \(\beta\).

Cluster-robust inference allows errors to be dependent within specified clusters while relying on independence across clusters, alongside suitable regularity and exogeneity conditions. HAC (heteroskedasticity-and-autocorrelation-consistent) inference instead allows serial dependence across a time-ordered sequence, under conditions such as weak dependence, suitable moments, and an appropriate bandwidth sequence. Neither approach repairs endogeneity, an incorrect regression specification, or a dependence structure the estimator was not designed to capture.

For clusters \(g=1,\ldots,G\), a basic cluster-robust covariance estimator has the sandwich form

where \(X_g\) and \(\hat u_g\) contain the regressors and residuals in cluster \(g\). Finite-sample scaling factors or leverage adjustments may modify this expression. Its central feature is that it aggregates each cluster's score contribution before forming the outer product, allowing arbitrary covariance among observations in the same cluster.

Mistake 1: Clustering at the Wrong Level

The clustering level should reflect how treatment or exposure is assigned and where shocks can be shared. Clustering at a level below the assignment unit can count observations as independent even when they share a common treatment decision or shock. Conversely, clustering too broadly can discard useful independent variation and yield imprecise estimates. The relevant structure may also involve nested or crossed sources of dependence; a single cluster label is not automatically sufficient for every design.

Mistake 2: Treating Observation Count as Independent Information

A large \(n\) can coexist with a small \(G\). The usual cluster asymptotics rely on the number of independent clusters growing, not merely on adding observations within a fixed set of clusters. When \(G\) is small, a cluster sandwich estimate can be unstable, and normal-reference tests may reject too often. Finite-sample corrections, leverage adjustments, and degrees-of-freedom methods can help, but none turns a handful of clusters into abundant independent information. The number and balance of clusters, and whether a few clusters dominate, matter.

Mistake 3: Choosing a HAC Bandwidth That Is Too Short

A HAC estimator combines estimated autocovariances of score contributions, weighting them by a kernel and truncating or tapering their influence according to a bandwidth. If the bandwidth excludes important positive serial covariance, the long-run variance can be underestimated and the standard error too small. If included covariances are negative, or if noisy distant-lag estimates are given too much weight, a larger bandwidth can instead lower the estimate or increase its variability. Thus, “larger bandwidth means larger standard errors” is not a general rule.

For a Bartlett estimator with bandwidth \(L\), the weight at lag \(j\) is \(w_j=1-j/(L+1)\) for \(j=1,\ldots,L\). Under standard weak-dependence and regularity conditions, HAC consistency typically requires the bandwidth to grow with the sample size while growing more slowly than it. In a finite sample, a bandwidth is an approximation choice: it should be assessed in relation to plausible serial dependence and sensitivity to reasonable alternatives.

Mistake 4: Using an Unjustified Reference Distribution

A covariance estimate and a reference distribution are separate parts of inference. A cluster-robust standard error does not itself specify whether a test should use a normal distribution, a \(t\) distribution with a particular \(df\), or a more design-specific procedure. The choice depends on the sampling and assignment structure, the number and influence of clusters, and the inference method being used. In particular, \(G-1\) is a common convention in some settings, not a theorem that applies to every clustered coefficient or contrast. Satterthwaite degrees of freedom, for example, can differ across contrasts.

For HAC inference, a normal reference is common in large samples under the relevant asymptotic conditions. Applying a small-cluster \(t\) correction to a time-series HAC result merely because the sample has a certain number of observations is not automatically justified. Likewise, a large-sample normal test should not be assumed reliable just because the total \(n\) is large when there are few independent clusters. Report the covariance estimator and the reference distribution or degrees of freedom, not only the standard error.

A Short Justification: Why Cluster Counts Matter

Let \(S_g\) be the score contribution for cluster \(g\), and suppose distinct clusters are independent. The variance of the total score is then

because independence makes every cross-cluster covariance zero. But within a cluster, the score is a sum of observation-level contributions, and its variance contains their cross-covariances. For two contributions \(s_{gi}\) and \(s_{gj}\), that variance includes \(2\mathrm{Cov}(s_{gi},s_{gj})\). A calculation that treats the observations as independent drops these terms. Clustering retains them by forming the cluster score first and then taking its outer product. This explains both why the dependence unit matters and why the number of independent clusters—not simply the number of rows—is central to the approximation.

Practical Diagnostics and Common Pitfalls

Before reporting clustered or HAC inference, make the assumptions visible and check how sensitive the conclusion is to defensible choices. The following questions help distinguish a computationally successful analysis from a credible inferential one.

• What generates dependence? Identify the assignment unit, shared shocks, repeated measurements, or time ordering that motivates the covariance estimator.
• How many independent units are there? Report \(G\) for cluster inference, along with cluster sizes and any concentration of leverage or influence.
• Does the bandwidth reflect plausible dependence? Check whether the main conclusion changes across reasonable bandwidths, and explain the basis for the selected value.
• What reference distribution was used? State whether inference is normal or \(t\)-based and give the degrees of freedom when relevant.
• What happens under alternatives? Sensitivity analysis can reveal fragile uncertainty estimates, but it cannot establish that one arbitrary choice is correct.

A common mistake is to argue, “There are thousands of observations, so the robust standard error and normal p-value must be reliable.” The correction is to identify the independent units supporting the relevant asymptotic approximation. In a clustered design, that may be a small \(G\); in a time series, dependence may extend across many lags. Large \(n\) does not answer either question by itself.