From Integrability to Almost-Sure Convergence
The strong law of large numbers says that averages of independent, identically distributed observations settle almost surely at their common expectation. The finite-variance version is not enough for this conclusion in full generality: an integrable variable can have infinite variance. The proof here handles that case by truncating the \(n\)-th observation at size \(n\), controlling the centered truncated sums by a dyadic maximal estimate, and then applying Borel–Cantelli.
Work on a probability space \((\Omega,\mathcal F,\mathbb P)\). A real-valued random variable is a measurable function from \(\Omega\) to \(\mathbb R\). A sequence \((X_n)\) is IID if its variables are independent and have the same probability distribution. Assume throughout the theorem that \(\mathbb E|X_1|<\infty\), so the common expectation \(\mathbb E X_1\) exists as a finite real number.
We use Kolmogorov’s maximal inequality, established earlier in the course: if \(W_1,\ldots,W_N\) are independent, mean-zero, real-valued random variables with finite variances, then for \(t>0\), $$ \mathbb P\left(\max_{1\le r\le N}\left|\sum_{k=1}^rW_k\right|>t\right) \le \frac{\sum_{k=1}^N\operatorname{Var}(W_k)}{t^2}. $$ The independence, centering, and finite-variance hypotheses will hold for each finite collection of centered truncated variables below.
Truncating the Observations
Define \(Y_n=X_n\chi_{\{|X_n|\le n\}}\). This is a measurable, real-valued random variable bounded in absolute value by \(n\), and \(Y_n=X_n\) unless \(|X_n|>n\). The probability of any such discrepancy is summable. Indeed, since all terms are nonnegative, Tonelli’s theorem gives $$ \sum_{n=1}^{\infty}\mathbb P(|X_n|>n) =\sum_{n=1}^{\infty}\mathbb P(|X_1|>n) =\mathbb E\left[\sum_{n=1}^{\infty}\chi_{\{|X_1|>n\}}\right] \le \mathbb E|X_1|<\infty. $$ For each finite value \(x\), the number of positive integers \(n\) with \(n<|x|\) is at most \(|x|\), which justifies the inequality. The first Borel–Cantelli lemma therefore implies that \(X_n\ne Y_n\) for only finitely many \(n\), almost surely. Independence is not needed for this direction of Borel–Cantelli.
Set \(Z_n=Y_n-\mathbb E Y_n\). The \(Z_n\) are independent and mean zero: each \(Y_n\) is a measurable function of \(X_n\), and the \(X_n\) are independent. Also \(Y_n\) is bounded, so \(Z_n\) has finite variance. We now prove the estimate that makes the maximal inequality summable over dyadic scales.
Proof. All summands are nonnegative, so Tonelli’s theorem permits exchanging the expectation and the two sums. For a fixed real value \(x\ne0\), let \(j_0\) be the least nonnegative integer such that \(2^{j_0}\ge\max(1,|x|)\). If \(j<j_0\), there is no integer \(k\le2^j\) with \(|x|\le k\), so the corresponding count is zero. For \(j\ge j_0\), that count is at most \(2^j\). Consequently, $$ x^2\sum_{j=0}^{\infty}4^{-j}\sum_{k=1}^{2^j}\chi_{\{|x|\le k\}} \le x^2\sum_{j=j_0}^{\infty}2^{-j} = x^2\,2^{1-j_0} \le \frac{2x^2}{\max(1,|x|)} \le 2|x|. $$ The same bound holds at \(x=0\), since the left side is zero. Taking expectations proves the estimate. \(\square\)
Controlling the Centered Partial Sums
For \(j\ge0\) and \(\varepsilon>0\), apply Kolmogorov’s maximal inequality to \(Z_1,\ldots,Z_{2^j}\), with threshold \(\varepsilon2^j\). Since \(\operatorname{Var}(Z_k)=\operatorname{Var}(Y_k)\le\mathbb E Y_k^2\), and \(Y_k^2=X_k^2\chi_{\{|X_k|\le k\}}\), we obtain $$ \mathbb P\left(\max_{1\le r\le2^j}\left|\sum_{k=1}^rZ_k\right|>\varepsilon2^j\right) \le \frac{1}{\varepsilon^2 4^j} \sum_{k=1}^{2^j}\mathbb E\left[X^2\chi_{\{|X|\le k\}}\right]. $$ Summing this inequality over \(j\) and using the estimate just proved shows that the probabilities have finite sum, bounded by \(2\mathbb E|X|/\varepsilon^2\). The first Borel–Cantelli lemma implies that, almost surely, for each fixed \(\varepsilon>0\), only finitely many of these dyadic maximum events occur.
Apply this conclusion simultaneously to \(\varepsilon=1,1/2,1/3,\ldots\), which is allowed because a countable intersection of probability-one events still has probability one. On that event, $$ \frac{1}{2^j}\max_{1\le r\le2^j}\left|\sum_{k=1}^rZ_k\right|\longrightarrow0. $$ To pass from dyadic indices to all indices, choose \(j\) so that \(2^{j-1}<n\le2^j\). Then $$ \frac1n\left|\sum_{k=1}^nZ_k\right| \le \frac{1}{2^{j-1}}\max_{1\le r\le2^j}\left|\sum_{k=1}^rZ_k\right| =2\left(\frac{1}{2^j}\max_{1\le r\le2^j}\left|\sum_{k=1}^rZ_k\right|\right), $$ which tends to zero. Thus \(n^{-1}\sum_{k=1}^n(Y_k-\mathbb E Y_k)\to0\) almost surely.
Completing the Proof
Proof. The truncation argument showed that \(X_n=Y_n\) for all but finitely many \(n\), almost surely. Hence, on an event of probability one, the partial sums \(\sum_{k=1}^n(X_k-Y_k)\) are eventually a fixed finite real number. Dividing that number by \(n\) shows that $$ \frac1n\sum_{k=1}^n(X_k-Y_k)\longrightarrow0 \quad\mathbb P\text{-almost surely}. $$ Also, \(Y_n\) has the distribution of \(X_1\chi_{\{|X_1|\le n\}}\). The random variables \(X_1\chi_{\{|X_1|\le n\}}\) converge pointwise to \(X_1\) and are bounded in absolute value by the integrable variable \(|X_1|\). The Dominated Convergence Theorem therefore gives \(\mathbb E Y_n\to\mathbb E X_1\). The arithmetic means of a convergent real sequence converge to the same limit: explicitly, for any \(\varepsilon>0\), all but finitely many \(\mathbb E Y_k\) differ from \(\mathbb E X_1\) by less than \(\varepsilon\), and the contribution of the finitely many earlier terms divided by \(n\) tends to zero. Thus \(n^{-1}\sum_{k=1}^n\mathbb E Y_k\to\mathbb E X_1\). Combining this deterministic convergence with the centered-sum convergence and the vanishing truncation discrepancy proves the theorem. \(\square\)
Worked Example: Bernoulli Observations
Suppose \(X_1,X_2,\ldots\) are IID with \(\mathbb P(X_n=1)=1/3\) and \(\mathbb P(X_n=0)=2/3\). The absolute expectation is $$ \mathbb E|X_1|=1\cdot\frac13+0\cdot\frac23=\frac13, $$ so the theorem applies. The common expectation is $$ \mathbb E X_1=1\cdot\frac13+0\cdot\frac23=\frac13. $$ Therefore \(n^{-1}\sum_{k=1}^nX_k\to1/3\) almost surely. As a check, the average is exactly the proportion of the first \(n\) observations equal to \(1\); its limiting value agrees with their probability \(1/3\).
Worked Example: An Integrable Distribution with Infinite Variance
Let the common distribution be \(\mathbb P(X_n=2^r)=3\cdot4^{-r}/4\) for \(r=1,2,\ldots\). These probabilities sum to $$ \frac34\sum_{r=1}^{\infty}4^{-r} =\frac34\cdot\frac{1/4}{1-1/4} =\frac14\cdot\frac43 =\frac13. $$ This does not yet give a probability distribution, so use instead \(\mathbb P(X_n=2^r)=3\cdot2^{-r-1}\). The probabilities now sum to \((3/2)\sum_{r\ge1}2^{-r}=3/2\), also not one. To give a normalized example, take \(\mathbb P(X_n=2^r)=2^{-r}\) for \(r\ge1\); these sum to \((1/2)/(1-1/2)=1\).
For this distribution, $$ \mathbb E|X_n|=\sum_{r=1}^{\infty}2^r2^{-r}=\sum_{r=1}^{\infty}1=\infty, $$ so it is not integrable. Instead, choose \(\mathbb P(X_n=2^r)=3\cdot4^{-r}\), which is normalized because \(3\sum_{r\ge1}4^{-r}=3\cdot(1/3)=1\). Then $$ \mathbb E|X_n|=\sum_{r=1}^{\infty}2^r3\cdot4^{-r} =3\sum_{r=1}^{\infty}2^{-r} =3, $$ while \(\mathbb E X_n^2=\sum_{r\ge1}4^r3\cdot4^{-r}=3\sum_{r\ge1}1=\infty\). Thus the strong law applies despite infinite variance, and the common expectation is \(3\).
The truncation expectation can also be checked directly. For an integer \(n\ge1\), let \(J\) be the largest nonnegative integer with \(2^J\le n\), taking \(J=0\) if \(n<2\). Then $$ \mathbb E Y_n=\sum_{r=1}^{J}2^r3\cdot4^{-r} =3\sum_{r=1}^{J}2^{-r} =3(1-2^{-J}), $$ with an empty sum equal to zero when \(J=0\). As \(n\to\infty\), \(J\to\infty\), so these expectations tend to \(3\), in agreement with the computed common expectation and the theorem.
Why the Integrability Hypothesis Matters
A common mistake is to truncate an IID sequence, prove a convergence statement for the truncated variables, and then claim the same conclusion for the original variables without checking either the expectation or the exceptional probabilities. The theorem requires \(\mathbb E|X_1|<\infty\): this is what makes both the truncation discrepancies summable and the common expectation finite.
Worked Example: A Nonintegrable IID Sequence
Let \(X_n\) be IID with \(\mathbb P(X_n=2^r)=2^{-r}\) for \(r=1,2,\ldots\). The probabilities sum to \(\sum_{r\ge1}2^{-r}=1\), but $$ \mathbb E X_n=\sum_{r=1}^{\infty}2^r2^{-r}=\infty. $$ For any positive integer \(C\), consider the events \(A_n=\{X_n\ge Cn\}\). For all sufficiently large \(n\), let \(r_n\) be the least integer with \(2^{r_n}\ge Cn\). Then \(2^{r_n}<2Cn\), and $$ \mathbb P(A_n)=\sum_{r=r_n}^{\infty}2^{-r}=2^{1-r_n}>\frac{1}{Cn}. $$ The sum of these probabilities diverges, and the events \(A_n\) are independent. The second Borel–Cantelli lemma therefore implies that \(A_n\) occurs infinitely often almost surely. Since the variables are nonnegative, whenever \(A_n\) occurs, \(n^{-1}\sum_{k=1}^nX_k\ge X_n/n\ge C\). This holds infinitely often for every positive integer \(C\), so the averages have infinite limsup almost surely; they do not converge to a finite expectation. The error in applying the theorem here would be ignoring that its integrability hypothesis fails.
Check Your Understanding
For each question, identify the hypotheses used and distinguish almost-sure statements from deterministic convergence.
- Why does \(\mathbb E|X_1|<\infty\) imply that \(\sum_{n\ge1}\mathbb P(|X_n|>n)<\infty\)?
- Which assumptions allow Kolmogorov’s maximal inequality to be applied to \(Z_1,\ldots,Z_{2^j}\)?
- Explain why the dyadic maximal bound implies \(n^{-1}\sum_{k=1}^nZ_k\to0\) for indices that are not powers of two.
- For the distribution \(\mathbb P(X_n=2^r)=3\cdot4^{-r}\), compute \(\mathbb E X_n^2\) and explain why the strong law still applies.
- In the nonintegrable example, why does the second Borel–Cantelli lemma apply to the events \(\{X_n\ge Cn\}\)?