Heavy Tails and the Limits of Normal Approximation

Why Tail Behavior Matters

The Central Limit Theorem (CLT) says that, under its conditions, the standardized sample mean becomes approximately normal as the sample size grows. But a statement about what happens as \(n\) grows does not promise that a normal approximation will be accurate for every particular sample size. One reason convergence can be slow is that a population has a heavy tail: values far from the mean are uncommon, but not as uncommon as they would be in a normal population.

An extreme observation can have a noticeable effect on a sample mean, especially when the sample is not large. The key distinction is whether the population has finite variance. If it does, the usual CLT applies to independent, identically distributed observations, though the approximation may take a long time to become useful. If the variance is infinite, the usual finite-variance CLT does not apply: the standard deviation and the familiar standard error \(\sigma/\sqrt{n}\) are not available.

Definition: A distribution is called heavy-tailed when its probabilities of very large values decline relatively slowly. This term describes tail behavior, not by itself whether the mean or variance exists. For the CLT for sample means, the central distinction is whether the population variance \(\sigma^2\) is finite.

The Finite-Variance CLT and Its Limit

For independent and identically distributed observations \(X_1,\ldots,X_n\), each with population mean \(\mu\) and finite, positive population variance \(\sigma^2\) (\(0<\sigma^2<\infty\)), the CLT states that

\frac{\bar{x}-\mu}{\sigma/\sqrt{n}} \ \xrightarrow{\ d\ }\ N(0,1)

Here convergence in distribution means that the standardized sample mean's distribution approaches the standard normal distribution as \(n\) increases. The theorem is asymptotic: it does not specify a universal sample size at which the approximation is already good. Heavy tails can make large observations more influential and slow the approach to the normal shape.

A useful family for examining this distinction is the Pareto distribution. For \(x\geq 1\), let its density be \(f(x)=\alpha x^{-(\alpha+1)}\), where \(\alpha>0\). Its tail probability is \(P(X>x)=x^{-\alpha}\) for \(x\geq1\). The smaller \(\alpha\) is, the more slowly this tail declines.

The moments show exactly when this model has finite mean and variance. For a positive power \(r\),

E[X^r]=\int_1^\infty x^r\alpha x^{-(\alpha+1)}\,dx =\alpha\int_1^\infty x^{r-\alpha-1}\,dx =\frac{\alpha}{\alpha-r}, \qquad r<\alpha.

If \(r\geq\alpha\), the integral diverges. Thus the mean is finite when \(\alpha>1\), and the second moment—and hence the variance—is finite when \(\alpha>2\). A population can therefore have a finite mean but infinite variance.

Worked Example: A Pareto Population With Finite Variance

Suppose observations are iid from a Pareto distribution with \(\alpha=3\). Find the mean and variance, then calculate the CLT standard error for a sample of \(n=100\).

Since \(3>2\), both the mean and variance exist. Using the moment formula,

\mu=E[X]=\frac{3}{3-1}=1.5, \qquad E[X^2]=\frac{3}{3-2}=3.

The variance is the second moment minus the squared mean:

\sigma^2=E[X^2]-\mu^2=3-(1.5)^2=0.75, \qquad \sigma=\sqrt{0.75}\approx0.8660.

For \(n=100\), the standard error of the sample mean is

SE(\bar{x})=\frac{\sigma}{\sqrt{n}} =\frac{0.8660}{\sqrt{100}} =0.08660.

The finite-variance CLT gives a normal approximation to the distribution of \(\bar{x}\) when the sample is sufficiently large. For example, the corresponding approximate standardized value for \(\bar{x}=1.7\) is \((1.7-1.5)/0.08660\approx2.309\). This calculation supplies a normal-model approximation, not a guarantee that its accuracy is high at \(n=100\); tail behavior matters when judging that limitation.

Extreme Observations in an iid Sample

A direct way to describe the chance of seeing at least one extreme value is to consider the sample maximum. Suppose \(X_1,\ldots,X_n\) are independent and identically distributed, and let \(q=P(X>c)\) be the probability that one observation exceeds a threshold \(c\). Each observation has probability \(1-q\) of being at most \(c\). Independence gives the probability that all \(n\) observations are at most \(c\) as \((1-q)^n\). Taking the complement,

P\!\left(\max(X_1,\ldots,X_n)>c\right) =1-\left(1-P(X>c)\right)^n.

Identical distributions are needed for the single exceedance probability to be raised to the \(n\)th power. If observations are independent but not identically distributed, the corresponding expression is \(1-\prod_{i=1}^n(1-P(X_i>c))\).

Worked Example: A Rare Extreme Value in a Finite-Variance Population

Let \(X_1,\ldots,X_{200}\) be iid observations from a Pareto distribution with \(\alpha=2.5\). Find the probability that at least one observation exceeds 20, and calculate the standard error of the sample mean.

The mean and variance are finite because \(\alpha>2\). The moment formula gives

\mu=\frac{2.5}{2.5-1}=\frac{5}{3}\approx1.667, \qquad E[X^2]=\frac{2.5}{2.5-2}=5.

Therefore,

\sigma^2=5-\left(\frac{5}{3}\right)^2 =\frac{20}{9}\approx2.222, \qquad \sigma=\frac{\sqrt{20}}{3}\approx1.491.

For one observation, the probability of exceeding 20 is

P(X>20)=20^{-2.5}\approx0.0005590.

Using independence and the identical Pareto distribution,

P\!\left(\max(X_1,\ldots,X_{200})>20\right) =1-(1-0.0005590)^{200} \approx0.1058.

Thus, there is about a 10.6% chance that the sample contains at least one observation above 20. The sample-mean standard error is

SE(\bar{x})=\frac{1.491}{\sqrt{200}}\approx0.1054.

The maximum calculation concerns an individual observation, not the probability that the sample mean is extreme. It illustrates that even a population with finite variance can produce unusually large observations in a sample. Such observations can slow normal approximation, but this calculation alone does not quantify the approximation error for \(\bar{x}\).

When the Variance Is Infinite

For iid observations with finite variance, the CLT uses \(\sigma\) to standardize \(\bar{x}\). If the variance is infinite, there is no finite \(\sigma\), and the usual standard error \(\sigma/\sqrt{n}\) cannot be calculated. Consequently, one cannot invoke the finite-variance CLT or interpret a calculation using that standard error as its normal approximation.

This does not necessarily mean that the mean is infinite. For a Pareto distribution with \(1<\alpha\leq2\), the mean is finite but the variance is infinite. When the mean is finite, the sample mean still converges in probability to \(\mu\) under the usual iid assumptions. That long-run consistency does not supply the finite-variance CLT's normal shape or its \(\sigma/\sqrt{n}\) scale.

Worked Example: A Finite Mean but Infinite Variance

Suppose iid observations follow a Pareto distribution with \(\alpha=1.5\). Determine whether the mean and variance exist, and find the probability that at least one of 100 observations exceeds 100.

Because \(1.5>1\), the mean is finite:

\mu=E[X]=\frac{1.5}{1.5-1}=3.

But the second moment requires \(2<1.5\), which is false. More directly, its defining integral is

E[X^2]=1.5\int_1^\infty x^{2-1.5-1}\,dx =1.5\int_1^\infty x^{-0.5}\,dx =\infty.

Therefore \(\sigma^2=\infty\), and the usual standard error \(\sigma/\sqrt{100}\) is not finite. For one observation,

P(X>100)=100^{-1.5}=0.001.

The observations are iid, so the probability of at least one exceedance is

P\!\left(\max(X_1,\ldots,X_{100})>100\right) =1-(1-0.001)^{100} =1-0.999^{100} \approx0.09521.

There is about a 9.52% chance that at least one observation exceeds 100. One such observation contributes more than \(100/100=1\) to the sample mean, though the other observations also contribute and determine its final value. The calculation illustrates the potential influence of extremes; it does not imply that every sample mean is large.

What the Conditions Do—and Do Not—Tell Us

The finite-variance condition separates two different conclusions. With iid observations and finite variance, the usual CLT applies as \(n\) grows, but convergence may be slow. With infinite variance, that version of the CLT does not apply, even if the mean exists. Neither conclusion is a complete description of how well a normal curve approximates a particular sample mean at a particular \(n\).

One useful diagnostic result makes the role of tail behavior explicit. A Berry–Esseen bound states that if the observations are iid, \(0<\sigma^2<\infty\), and \(E[|X-\mu|^3]<\infty\), then the maximum difference between the standardized sample-mean distribution function and the standard normal distribution function is bounded by a constant times

\frac{E[|X-\mu|^3]}{\sigma^3\sqrt{n}}.

The ratio \(E[|X-\mu|^3]/\sigma^3\) measures standardized third absolute moment. A large value makes this bound less informative at a given \(n\). If the third absolute moment is infinite, this particular bound cannot be used; that does not prove that a normal approximation is necessarily poor. For example, the Pareto distribution with \(\alpha=3\) has finite variance but infinite third moment. The finite-variance CLT still applies, while the Berry–Esseen bound just stated does not.

For a Pareto distribution, the tail exponent makes the contrast concrete: \(\alpha>2\) gives finite variance, whereas \(1<\alpha\leq2\) gives finite mean and infinite variance. These are statements about moment existence, not a rule that assigns an exact accuracy to every sample size. The data-generating process and the question being approximated still matter.

Common Pitfalls

Assuming finite variance guarantees a good approximation for a modest \(n\). The CLT gives a limiting result, not a universal accuracy threshold. Heavy tails can leave extreme observations influential in samples that are not large enough.
Using \(\sigma/\sqrt{n}\) when \(\sigma^2=\infty\). An infinite variance does not produce a finite standard error. The usual finite-variance standardization is unavailable.
Concluding that an infinite-variance population has no finite mean. In the Pareto family, \(1<\alpha\leq2\) gives a finite mean but infinite variance.
Treating the probability of an extreme observation as the probability of an extreme sample mean. A maximum calculation concerns individual observations. Other observations also contribute to the mean.
Using the iid maximum formula for independent observations with different distributions. For non-identical independent observations, use the product of their separate probabilities of not exceeding the threshold.

Key takeaway: A finite variance allows the usual CLT for iid sample means, but heavy tails can make its normal approximation slow to become useful. Infinite variance removes the usual CLT standardization, even when the population mean remains finite.

Check Your Understanding

Try these questions using the conditions and distinctions developed above.

For a Pareto distribution with \(\alpha=2.2\), is the mean finite? Is the variance finite? Explain using the moment conditions.
Why does a 10% chance of at least one very large observation not mean there is a 10% chance that the sample mean is unusually large?
For iid observations with \(P(X>c)=0.02\), write an expression for the probability that the sample maximum exceeds \(c\) when \(n=50\).
What part of the usual sample-mean CLT calculation fails if the population variance is infinite?
Does the finite-variance CLT guarantee an accurate normal approximation at every fixed sample size? Explain.