The IID Central Limit Theorem And Its Derivation

Why a Normal Limit Appears

A sample average can settle near its population mean as the sample grows, but that alone does not describe the average’s fluctuations at a useful scale. The iid central limit theorem identifies that scale and gives the limiting distribution of the fluctuations. Its central conclusion is that many independent, identically distributed observations with finite, nonzero variance produce a normal limit after centering and scaling.

The theorem is an asymptotic statement: it describes a sequence of distributions as the sample size tends to infinity. It does not say that every finite-sample average is exactly normal, nor does it say that the original observations must themselves be normally distributed.

Statement of the IID Central Limit Theorem

Theorem: Let \(X_1,X_2,\ldots\) be independent and identically distributed random variables with \(E[X_i]=\mu\) and \(Var(X_i)=\sigma^2\), where \(0<\sigma^2<\infty\). Define \(\bar{X}_n=n^{-1}\sum_{i=1}^n X_i\). Then

\frac{\sqrt{n}(\bar{X}_n-\mu)}{\sigma} = \frac{\sum_{i=1}^n(X_i-\mu)}{\sigma\sqrt{n}} \xrightarrow{d} N(0,1).

Equivalently, the centered sum divided by \(\sqrt{n}\) converges in distribution to a normal random variable with mean zero and variance \(\sigma^2\).

The assumptions are specific. The observations must be independent and identically distributed, and their common distribution must have a finite, strictly positive variance. The theorem centers each observation by its common mean and divides by its common standard deviation. The factor \(\sqrt{n}\) is the scale at which the aggregate fluctuation has a nondegenerate limit.

The notation \(\xrightarrow{d}\) means convergence in distribution: the cumulative distribution functions converge at every continuity point of the limiting cumulative distribution function. This is not a claim that the scaled random variables become equal to a normal random variable, or converge to one in probability. The result identifies their limiting distribution.

Derivation Using Characteristic Functions

A characteristic function of a random variable \(Y\) is \(\phi_Y(t)=E[e^{itY}]\), for real \(t\). It exists for every real \(t\), since \(|e^{itY}|=1\). A standard characteristic-function continuity theorem states that if a sequence of characteristic functions converges pointwise to a function that is a characteristic function continuous at zero, then the corresponding random variables converge in distribution to the variable with that limiting characteristic function. The standard normal characteristic function is \(e^{-t^2/2}\), which is continuous at zero.

To derive the theorem, standardize one observation by setting \(Y_i=(X_i-\mu)/\sigma\). The \(Y_i\) are iid, have mean zero and variance one, and the normalized sum in the theorem is \(n^{-1/2}\sum_{i=1}^nY_i\). The key step is to show that the characteristic function of one standardized observation has a quadratic expansion near zero.

Key expansion: If \(E[Y]=0\) and \(E[Y^2]=1\), then, as \(u\) tends to zero,

\phi_Y(u)=E[e^{iuY}]=1-\frac{u^2}{2}+o(u^2).

Here \(o(u^2)\) denotes a remainder whose ratio to \(u^2\) tends to zero as \(u\) tends to zero.

This expansion follows under the finite second-moment assumption; no finite third moment is needed. For each fixed real \(y\), the second-order difference quotient satisfies

\lim_{u\to 0}\frac{e^{iuy}-1-iuy}{u^2}=-\frac{y^2}{2}.

Also, for real \(v\), \(|e^{iv}-1-iv|\leq v^2/2\). Therefore the absolute value of the quotient with \(y=Y\) is bounded by \(Y^2/2\), which has finite expectation. The dominated convergence theorem allows the limit to pass through the expectation. Since \(E[Y]=0\), this gives

\begin{aligned} \phi_Y(u) &=1+E[e^{iuY}-1-iuY],\\ \lim_{u\to 0}\frac{\phi_Y(u)-1}{u^2} &=E\left[-\frac{Y^2}{2}\right]=-\frac12. \end{aligned}

That limit is precisely the asserted expansion. To obtain the characteristic function of the normalized sum, use independence: the characteristic function of a sum of independent random variables is the product of their characteristic functions. For fixed \(t\), the normalized sum has characteristic function

\begin{aligned} \phi_{\frac{1}{\sqrt{n}}\sum_{i=1}^n Y_i}(t) &=\left[\phi_Y\left(\frac{t}{\sqrt{n}}\right)\right]^n\\ &=\left[1-\frac{t^2}{2n}+o\left(\frac{1}{n}\right)\right]^n\\ &\longrightarrow e^{-t^2/2}. \end{aligned}

For the second line, \(t\) is fixed and the expansion is applied at \(u=t/\sqrt{n}\). The final limit uses the standard exponential limit: if \(a_n\) tends to \(a\), then \((1+a_n/n)^n\) tends to \(e^a\). Here the term inside the brackets has that form up to a remainder that is \(o(1/n)\). The limit \(e^{-t^2/2}\) is the characteristic function of \(N(0,1)\) and is continuous at zero. The characteristic-function continuity theorem therefore gives the stated convergence in distribution.

Worked Examples: The Normal Limit in Three Distributions

Worked Example: Normally Distributed Observations

Suppose a hypothetical sequence of iid measurements has \(X_i\sim N(12,9)\), with measurement units \(U\). Thus \(\mu=12\,U\) and \(\sigma=3\,U\). The standardized observation \(Y_i=(X_i-12)/3\) has distribution \(N(0,1)\), whose characteristic function is \(\phi_Y(u)=e^{-u^2/2}\).

For a sample of size \(n\), independence gives the characteristic function of the normalized sum as

\left[\phi_Y\left(\frac{t}{\sqrt{n}}\right)\right]^n = \left[e^{-t^2/(2n)}\right]^n = e^{-t^2/2}.

It is already the standard normal characteristic function for every \(n\). In this example, the normalized sum is exactly standard normal at every sample size, not merely asymptotically. The calculation illustrates the theorem’s normalization: \(\sqrt{n}(\bar{X}_n-12\,U)/(3\,U)\) has a standard normal distribution.

Worked Example: A Bounded, Asymmetric Distribution

Consider hypothetical iid observations \(Y_i\) taking values \(-1\), \(0\), and \(2\), with probabilities \(1/3\), \(1/2\), and \(1/6\), respectively. Their mean and variance are

E[Y_i]=-\frac13+2\left(\frac16\right)=0,\qquad E[Y_i^2]=\frac13+4\left(\frac16\right)=1.

So the observations have mean zero and variance one, as required for the standardized form of the theorem. Their characteristic function is

\phi_Y(u)=\frac13e^{-iu}+\frac12+\frac16e^{2iu}.

Expanding each exponential about zero gives a constant term of \(1\), a linear term of \(iuE[Y_i]=0\), and a quadratic term of \(-u^2E[Y_i^2]/2=-u^2/2\). Because this distribution has finite support, its next terms give a remainder of order \(u^3\). In fact, \(E[Y_i^3]=1\), so

\phi_Y(u)=1-\frac{u^2}{2}-\frac{iu^3}{6}+O(u^4).

At \(u=t/\sqrt{n}\), the quadratic term is \(-t^2/(2n)\), while the cubic term is of order \(n^{-3/2}\). Thus

\left[\phi_Y\left(\frac{t}{\sqrt{n}}\right)\right]^n = \left[1-\frac{t^2}{2n}+O(n^{-3/2})\right]^n \longrightarrow e^{-t^2/2}.

The observations are not normally distributed, but they are iid and have finite, nonzero variance. Their normalized sum therefore has the standard normal limit.

Worked Example: A Uniform Distribution and a Finite-Sample Characteristic Function

Suppose hypothetical iid observations \(X_i\) are uniformly distributed on \([-\sqrt{3},\sqrt{3}]\), measured in units \(U\). Symmetry gives \(E[X_i]=0\,U\), and the variance of a uniform distribution on \([-a,a]\) is \(a^2/3\). With \(a=\sqrt{3}\,U\), this gives \(Var(X_i)=1\,U^2\). The assumptions of the theorem hold.

The characteristic function is obtained by integrating the uniform density:

\phi_X(u)=\frac{1}{2\sqrt{3}}\int_{-\sqrt{3}}^{\sqrt{3}}e^{iux}\,dx =\frac{\sin(\sqrt{3}u)}{\sqrt{3}u},

with value \(1\) at \(u=0\) by continuity. Using \(\sin(z)/z=1-z^2/6+O(z^4)\), we obtain

\phi_X(u)=1-\frac{u^2}{2}+O(u^4).

For instance, at \(t=1\) and \(n=100\), the characteristic function of the normalized sum is

\left[\phi_X\left(\frac{1}{10}\right)\right]^{100} = \left[\frac{\sin(\sqrt{3}/10)}{\sqrt{3}/10}\right]^{100} \approx 0.6062.

The limiting characteristic function at \(t=1\) is \(e^{-1/2}\approx 0.6065\). These close values are consistent with convergence toward the normal characteristic function; they do not establish that the finite-sample distribution is exactly normal.

What the Assumptions Do—and Do Not—Say

The finite-variance condition makes the quadratic term in the characteristic-function expansion well defined and finite. The iid condition lets each observation have the same expansion and lets independence turn the sum’s characteristic function into a power. If observations are dependent, differently distributed, or have infinite variance, this derivation does not establish the theorem’s conclusion; other central limit theorems may apply under different conditions.

A common mistake is to reason that a sufficiently large sample makes the original observations normal. The theorem says something different: the distribution of the centered and scaled sum approaches a normal distribution. Individual observations retain their original distribution. Another mistake is to omit the scaling: \(\bar{X}_n-\mu\) itself typically shrinks toward zero, whereas \(\sqrt{n}(\bar{X}_n-\mu)/\sigma\) has a nondegenerate limiting distribution.

Key takeaway: Under iid sampling with finite, positive variance, the standardized sum has a standard normal limit. The characteristic-function proof rests on the second-order expansion \(1-u^2/2+o(u^2)\), independence, and the continuity theorem for characteristic functions.

Check Your Understanding

Use the assumptions and characteristic-function argument above to answer these questions.

State the iid central limit theorem, including the conditions on the common mean and variance.
Why does the characteristic-function expansion require only a finite second moment, rather than a finite third moment?
For iid \(Y_i\) with mean zero and variance one, write the characteristic function of \(n^{-1/2}\sum_{i=1}^nY_i\) in terms of \(\phi_Y\).
Why does the normal limit not imply that the individual observations are normally distributed?
What fails in the stated derivation if the observations are dependent?