Why Heavy Tails Change OLS Asymptotics
The familiar random-design OLS argument has several distinct steps: the empirical Gram matrix must stabilize, the empirical regression score must obey a central limit theorem, and the covariance estimator must converge to a finite target. These steps rely on different moment conditions. A heavy-tailed regressor can disrupt the first step, while a heavy-tailed error can disrupt the score central limit theorem or make the usual covariance target infinite.
An infinite moment does not, by itself, prove that OLS is inconsistent. It does mean that a standard proof requiring that moment cannot simply be applied. The effect depends on which quantity is heavy-tailed, whether the errors are centered conditionally on the regressors, and how the OLS numerator compares with its denominator.
For example, if the observations are iid, \(E[Z_i u_i]=0\), \(Q\) is finite and positive definite, and \(E[\|Z_i u_i\|^2]<\infty\), then the usual law of large numbers and central limit theorem can support the standard asymptotic linear representation and normal limit. Heavy tails matter when they violate a condition used in one of these steps. The examples below separate failure of a familiar argument from failure of the estimator itself.
A Heavy-Tailed Regressor Can Still Give Consistency
Consider regression through the origin, \(Y_i=\beta X_i+u_i\), with iid observations. The OLS slope is \(\hat\beta=\sum_i X_iY_i/\sum_i X_i^2\), provided the denominator is nonzero. In ordinary finite-moment arguments, the denominator divided by \(n\) converges to \(E[X_i^2]\). If that expectation is infinite, there is no finite population Gram entry to use in that way. Nevertheless, the slope can still be consistent.
Worked Example: Pareto Regressors and Bounded Errors
Let \(X_i\) have a Pareto distribution with \(P(X_i>t)=t^{-3/2}\) for \(t\geq 1\). Its density is \((3/2)t^{-5/2}\), so \(E[X_i]=3\), while \[ E[X_i^2]=\int_1^\infty t^2(3/2)t^{-5/2}\,dt =(3/2)\int_1^\infty t^{-1/2}\,dt=\infty. \] Let \(u_i\) be independent of \(X_i\), with \(P(u_i=1)=P(u_i=-1)=1/2\), and set \(Y_i=\beta X_i+u_i\). Then \(E[u_i\mid X_i]=0\).
For \(S_{xx}=\sum_{i=1}^nX_i^2\), the slope error is \[ \hat\beta-\beta=\frac{\sum_{i=1}^n X_i u_i}{S_{xx}}. \] Since \(|u_i|=1\), the Cauchy–Schwarz inequality gives \[ |\hat\beta-\beta| \leq \frac{\sqrt{n\sum_{i=1}^n u_i^2}}{S_{xx}} =\sqrt{\frac{n}{S_{xx}}} =\frac{1}{\sqrt{S_{xx}/n}}. \] Although \(E[X_i^2]=\infty\), \(S_{xx}/n\to\infty\) almost surely. Therefore this upper bound tends to zero and \(\hat\beta\xrightarrow{a.s.}\beta\). This example shows that an infinite regressor second moment invalidates the usual finite-Gram-matrix argument, but does not automatically invalidate consistency.
To verify the result about the denominator without assuming a finite expectation, fix any \(K>0\) and define \(V_i=\min(X_i^2,K)\). The \(V_i\) are iid and bounded, so the strong law gives \(n^{-1}\sum_iV_i\to E[V_i]\) almost surely. Also \(X_i^2\geq V_i\), hence \[ \liminf_{n\to\infty}\frac1n\sum_{i=1}^nX_i^2\geq E[\min(X_i^2,K)] \quad\text{almost surely}. \] Take a countable sequence of \(K\)'s tending to infinity. By monotone convergence, the right side tends to \(E[X_i^2]=\infty\). Thus the empirical average of \(X_i^2\) diverges almost surely. For the stated slope bound, \(\sum_i u_i^2\leq nC^2\), and substitution yields the displayed inequality. This proves the result. \(\square\)
Consistency here does not establish the usual root-\(n\) normal limit. In particular, the finite population Gram matrix \(Q=E[X_i^2]\) does not exist. The rate and distribution must be analyzed under conditions appropriate to this design; they cannot be read off from the ordinary finite-\(Q\) formula.
Infinite Error Variance: Consistency Without Root-n Normality
Now consider a regression containing only an intercept, \(Y_i=\beta+u_i\). Its OLS estimator is \(\hat\beta=\bar Y=\beta+\bar u\). When the errors have a finite first moment and mean zero, the strong law can still give consistency even if their variance is infinite. But the usual central limit theorem and finite-variance covariance formula need more than a first moment.
Worked Example: Symmetric Stable Errors with Index Three Halves
Let the iid errors have the symmetric stable distribution specified by the characteristic function \(E[e^{it u_i}]=\exp(-|t|^{3/2})\). This distribution has mean zero and a finite first absolute moment, but \(E[u_i^2]=\infty\). Sums of independent variables with this characteristic function satisfy \[ E\left[e^{it\sum_{i=1}^n u_i}\right] =\prod_{i=1}^n\exp(-|t|^{3/2}) =\exp(-n|t|^{3/2}). \] Consequently, \(n^{-2/3}\sum_{i=1}^n u_i\) has the same distribution as \(u_1\), because its characteristic function is \(\exp(-|t|^{3/2})\). Equivalently, \(n^{1/3}\bar u\) has that same fixed, nondegenerate distribution.
The strong law applies because \(E|u_i|<\infty\), so \(\bar u\to0\) almost surely and the intercept estimator is consistent. But the usual root-\(n\) sequence is \[ \sqrt n\,\bar u=n^{1/6}W_n,\qquad W_n=n^{1/3}\bar u, \] where each \(W_n\) has the same continuous, nondegenerate stable distribution. For any \(\epsilon>0\), \[ P(|\sqrt n\,\bar u|\leq\epsilon) =P(|W_n|\leq\epsilon n^{-1/6})\longrightarrow P(W_1=0)=0. \] Thus \(|\sqrt n\,\bar u|\to\infty\) in probability; it does not converge in probability to zero or to a finite normal limit. The stable rate is \(n^{1/3}\), not \(\sqrt n\).
The covariance issue is separate from consistency. In this example \(E[u_i^2]\) is infinite, so there is no finite error variance to estimate by the usual residual variance. In fact, the sample residual variance diverges. The strong law for truncated nonnegative variables, as in the previous argument, implies \(n^{-1}\sum_i u_i^2\to\infty\) almost surely. Meanwhile \(\bar u\to0\) almost surely, and \[ \frac1n\sum_{i=1}^n(u_i-\bar u)^2 =\frac1n\sum_{i=1}^n u_i^2-\bar u^2 \longrightarrow\infty \quad\text{almost surely}. \] It therefore cannot consistently estimate a finite error variance or supply the usual finite asymptotic covariance for the intercept.
When the OLS Estimator Itself Is Not Consistent
A finite first moment is important too. If errors have no mean, the familiar interpretation of the regression coefficient as a conditional mean may fail, and even a location parameter need not be consistently estimated by OLS. The Cauchy example makes this failure exact rather than relying on an approximate rate argument.
Worked Example: A Cauchy Intercept
Let \(Y_i=\beta+u_i\), where \(u_i\) are iid standard Cauchy variables with characteristic function \(E[e^{it u_i}]=e^{-|t|}\). The OLS intercept is \(\hat\beta=\beta+n^{-1}\sum_i u_i\). For any \(t\), \[ E[e^{it(\hat\beta-\beta)}] =\prod_{i=1}^n E[e^{i(t/n)u_i}] =\left(e^{-|t|/n}\right)^n =e^{-|t|}. \] Thus \(\hat\beta-\beta\) has the same standard Cauchy distribution for every sample size \(n\). For example, \[ P(|\hat\beta-\beta|>1)=P(|u_1|>1)=\frac12 \] for every \(n\). This probability does not tend to zero, so \(\hat\beta\) is not consistent for the location parameter \(\beta\). The Cauchy error has no finite mean, so this example is not a conditional-mean regression model; it demonstrates how badly OLS can behave when even the first-moment conditions behind that interpretation fail.
What Can Be Trusted, and What Cannot
Heavy-tailed regression analysis requires checking conclusions one at a time. Consistency may survive despite an infinite design second moment, as the Pareto example showed. A central limit theorem may fail even when the estimator is consistent, as in the stable-error intercept example. If the score has no finite second moment, the usual sandwich covariance target may not be finite. If the errors have no first moment, consistency itself can fail.
This does not mean that every covariance estimate is automatically wrong whenever an error distribution is heavy-tailed. It means that a finite covariance target and the limit theorem used to justify its estimate must be established under the actual distribution. Residual-based variance formulas cannot manufacture a finite population variance that does not exist. Likewise, a large empirical Gram matrix is not a substitute for a finite population Gram matrix in a standard root-\(n\) proof.
A useful diagnostic is to name the exact quantity whose moment is at issue: \(Z_iZ_i^\top\) for the empirical Gram matrix, \(Z_i u_i\) for the score central limit theorem, or \(u_i^2\) and \(Z_iZ_i^\top u_i^2\) for common covariance formulas. Then check whether the estimator's numerator and denominator might still yield consistency by another argument. This separates a failed standard proof from a failed statistical conclusion.
Check Your Understanding
For each question, identify which moment condition is relevant before deciding what follows.
- In the Pareto-regressor example, which population matrix entry is infinite, and why can the slope nevertheless be consistent?
- For symmetric stable errors of index \(3/2\), what is the exact scaling that leaves the sample mean nondegenerate?
- Why does \(n^{1/3}\bar u\) having a fixed nondegenerate distribution imply that \(|\sqrt n\,\bar u|\) diverges in probability?
- Which moment needed by the usual residual variance estimator is infinite in the stable-error intercept example?
- What feature of the Cauchy example prevents the OLS intercept from converging to its location parameter?