The Finite-Variance Strong Law
A weak law controls the probability of a large error at one specified sample size. The strong law makes a pathwise assertion: with probability one, the averages along a single infinite sequence settle down to the population mean. For independent observations with finite variance, the key extra tool is a maximal inequality. It controls the chance that any partial sum in a range becomes large, not just the sum at its endpoint.
Work on a probability space \((\Omega,\mathcal{F},\mathbb{P})\), and let \(X_1,X_2,\ldots\) be independent and identically distributed real-valued random variables. Write \(\mu=\mathbb{E}[X_1]\), assume \(\operatorname{Var}(X_1)=\sigma^2<\infty\), and define the sample average \(\overline X_n=n^{-1}\sum_{i=1}^n X_i\). The natural filtration is \(\mathcal{F}_n=\sigma(X_1,\ldots,X_n)\). Almost-sure convergence of \(\overline X_n\) to \(\mu\) means that \(\mathbb{P}(\{\omega:\overline X_n(\omega)\to\mu\})=1\).
Finite second moment implies that \(\mu\) is finite and \(\mathbb{E}|Z_1|<\infty\). We prove the theorem using a truncation that depends on the index \(i\). The truncation handles large individual values; the maximal inequality controls the accumulated centered terms.
Kolmogorov’s Maximal Inequality
First we prove the inequality used in the strong-law argument. For independent, mean-zero random variables \(U_1,\ldots,U_n\) with finite variances, put \(R_k=\sum_{i=1}^k U_i\). For every \(\lambda>0\), the inequality bounds the probability that at least one of these partial sums reaches magnitude \(\lambda\).
Proof. Let \(A_k\) be the event that \(|R_k|\ge\lambda\) and \(|R_j|<\lambda\) for every \(j<k\). These events are disjoint, and their union is \(\{\max_{1\le j\le n}|R_j|\ge\lambda\}\). The event \(A_k\) is determined by \(U_1,\ldots,U_k\). Write \(R_n=R_k+(R_n-R_k)\). Independence and the zero mean of \(R_n-R_k\) give $$ \mathbb{E}[R_n^2\mathbf{1}_{A_k}] =\mathbb{E}[R_k^2\mathbf{1}_{A_k}] +\mathbb{E}[(R_n-R_k)^2]\mathbb{P}(A_k) \ge \lambda^2\mathbb{P}(A_k). $$ The cross term has expectation zero because \(R_n-R_k\) is independent of \(A_k\) and \(R_k\), and has mean zero. Summing over the disjoint events and using nonnegativity, $$ \lambda^2\sum_{k=1}^n\mathbb{P}(A_k) \le \mathbb{E}[R_n^2\mathbf{1}_{\cup_k A_k}] \le \mathbb{E}[R_n^2] =\sum_{i=1}^n\operatorname{Var}(U_i). $$ The last equality follows by expanding the square: independence and zero means make every cross term vanish. Dividing by \(\lambda^2\) proves the result. \(\square\)
The inequality also applies to a finite stretch \(U_m,\ldots,U_n\): apply it to the partial sums starting at \(m\). This tail version will show that a series of independent centered terms converges almost surely when the sum of their variances is finite.
Proof of the Strong Law
Set \(Z_i=X_i-\mu\), so the \(Z_i\) are iid, have mean zero, and have variance \(\sigma^2\). Truncate each \(Z_i\) at its index by defining \(Y_i=Z_i\mathbf{1}_{\{|Z_i|\le i\}}\), and center the truncated variable by setting \(W_i=Y_i-\mathbb{E}[Y_i]\). The \(Y_i\), and hence the \(W_i\), are independent. We first verify the two summability facts needed in the argument.
Since \(\mathbb{E}|Z_1|<\infty\), the tail probabilities satisfy $$ \sum_{i=1}^{\infty}\mathbb{P}(|Z_i|>i) =\sum_{i=1}^{\infty}\mathbb{P}(|Z_1|>i) \le \mathbb{E}|Z_1|<\infty. $$ Indeed, for each fixed value \(z\), the number of positive integers \(i\) with \(|z|>i\) is at most \(|z|\); taking expectations gives the bound. The first Borel–Cantelli lemma says that if \(\sum_i\mathbb{P}(A_i)<\infty\), then \(\mathbb{P}(A_i\text{ infinitely often})=0\). It follows that \(Y_i=Z_i\) for all but finitely many \(i\), almost surely.
Next, we show \(\sum_i\operatorname{Var}(W_i)/i^2<\infty\). Since variance is at most the second moment, $$ \sum_{i=1}^{\infty}\frac{\operatorname{Var}(W_i)}{i^2} \le \mathbb{E}\!\left[Z_1^2\sum_{\substack{i\ge1\\i\ge |Z_1|}}\frac{1}{i^2}\right]. $$ For \(0\le |Z_1|\le1\), the inner sum is at most \(\sum_{i\ge1}i^{-2}\le2\), using \(1+\int_1^\infty x^{-2}\,dx=2\). For \(|Z_1|>1\), let \(m=\lceil |Z_1|\rceil\). Then $$ \sum_{i=m}^{\infty}\frac{1}{i^2} \le \frac{1}{m^2}+\int_m^\infty \frac{dx}{x^2} =\frac{1}{m^2}+\frac{1}{m} \le\frac{2}{|Z_1|}. $$ Thus the expectation is bounded by \(2\mathbb{E}[Z_1^2]+2\mathbb{E}|Z_1|\), which is finite.
We now show that \(\sum_{i=1}^{\infty}W_i/i\) converges almost surely. The terms \(U_i=W_i/i\) are independent, mean zero, and have summable variances. Choose integers \(m_\ell\) increasing to infinity so that \(\sum_{i=m_\ell}^{\infty}\operatorname{Var}(U_i)\le2^{-4\ell}\). The tail form of Kolmogorov’s inequality, followed by the limit over finite upper endpoints, gives $$ \mathbb{P}\!\left(\sup_{k\ge m_\ell}\left|\sum_{i=m_\ell}^{k}U_i\right|\ge2^{-\ell}\right) \le 2^{-2\ell}. $$ The right-hand sides sum to a finite number. By the first Borel–Cantelli lemma, almost surely, for all sufficiently large \(\ell\), every partial sum starting at \(m_\ell\) has magnitude less than \(2^{-\ell}\). For \(q\ge p\ge m_\ell\), the sum from \(p\) to \(q\) is the difference of two such partial sums, so its magnitude is at most \(2\cdot2^{-\ell}\). These bounds tend to zero as \(\ell\to\infty\); therefore the partial sums of \(\sum_i U_i\) are Cauchy almost surely and the series converges.
We use the following summation fact, with its short proof. If \(\sum_i a_i/i\) converges and \(b_n=\sum_{i=1}^n a_i/i\), then \(a_i=i(b_i-b_{i-1})\), where \(b_0=0\). Summing and rearranging gives $$ \frac1n\sum_{i=1}^n a_i=b_n-\frac1n\sum_{i=1}^{n-1}b_i\longrightarrow0. $$ Both terms on the right tend to the same limit: the second does so because the averages of a convergent sequence converge to its limit. Applying this fact on the probability-one event where \(\sum_i W_i/i\) converges yields \(n^{-1}\sum_{i=1}^n W_i\to0\) almost surely.
Finally, \(|\mathbb{E}Y_i|=|\mathbb{E}[Z_i\mathbf{1}_{\{|Z_i|\le i\}}]|\le\mathbb{E}[|Z_1|\mathbf{1}_{\{|Z_1|>i\}}]\to0\), because \(Z_1\) is integrable. Hence \(n^{-1}\sum_{i=1}^n\mathbb{E}Y_i\to0\), by averaging a sequence tending to zero. Also, \(Y_i=Z_i\) eventually almost surely, so \(n^{-1}\sum_{i=1}^n(Z_i-Y_i)\to0\) almost surely: the numerator is eventually a fixed finite sum. Combining these three limits in $$ \frac1n\sum_{i=1}^n Z_i =\frac1n\sum_{i=1}^n W_i +\frac1n\sum_{i=1}^n\mathbb{E}Y_i +\frac1n\sum_{i=1}^n(Z_i-Y_i) $$ proves the strong law. If \(\sigma^2=0\), then \(\mathbb{E}[Z_1^2]=0\) implies \(Z_1=0\) almost surely, and the conclusion holds directly as well. \(\square\)
Worked Applications
Worked Example: A Bernoulli Sample Average
Let \(X_i\) be iid Bernoulli random variables with \(\mathbb{P}(X_i=1)=\tfrac13\) and \(\mathbb{P}(X_i=0)=\tfrac23\). Their mean is $$ \mu=1\cdot\tfrac13+0\cdot\tfrac23=\tfrac13. $$ Their second moment is \(\mathbb{E}[X_i^2]=1^2\cdot\tfrac13+0^2\cdot\tfrac23=\tfrac13\), so $$ \operatorname{Var}(X_i)=\tfrac13-\left(\tfrac13\right)^2=\tfrac29. $$ The variables are iid and have finite variance, so the theorem gives \(\overline X_n\to\tfrac13\) almost surely. As a numerical check on the scale of the fluctuations, independence gives \(\operatorname{Var}(\overline X_n)=\tfrac{1}{n^2}\cdot n\cdot\tfrac29=\tfrac{2}{9n}\). This variance tends to zero, consistent with convergence, while the theorem proves the stronger almost-sure statement.
Worked Example: An Unbounded Distribution with Finite Variance
Let \(X_i\) be iid with \(\mathbb{P}(X_i=k)=2^{-k}\) for integers \(k\ge1\). This is a probability distribution because \(\sum_{k=1}^{\infty}2^{-k}=1\). Its probability generating function is \(G(s)=\sum_{k\ge1}2^{-k}s^k=s/(2-s)\) for \(|s|<2\). Differentiating gives \(G'(s)=2/(2-s)^2\) and \(G''(s)=4/(2-s)^3\); hence \(\mathbb{E}X_i=G'(1)=2\) and \(\mathbb{E}[X_i(X_i-1)]=G''(1)=4\). Therefore \(\mathbb{E}[X_i^2]=4+2=6\) and \(\operatorname{Var}(X_i)=6-2^2=2\).
The distribution is unbounded, but its variance is finite. The theorem therefore applies and yields \(\overline X_n\to2\) almost surely. Independently checking the variance of the average gives \(\operatorname{Var}(\overline X_n)=n^{-2}\cdot n\cdot2=2/n\). This example shows why bounded observations are not required.
Worked Example: Why Independence Matters
Let \(Y\) take values \(-1\) and \(1\) with probability \(\tfrac12\) each, and define \(X_i=Y\) for every \(i\). Each \(X_i\) has mean \(\tfrac12(-1)+\tfrac12(1)=0\) and variance \(\mathbb{E}[X_i^2]=1\). But the variables are not independent: they are all the same random variable. For every \(n\), $$ \overline X_n=\frac1n\sum_{i=1}^n Y=Y. $$ Thus \(\overline X_n\) does not converge to \(0\) almost surely; in fact, \(\mathbb{P}(Y=0)=0\). The finite-variance condition on each variable alone is not enough. Independence is essential to the maximal-inequality argument and to this theorem.
A Common Misstep and What the Proof Adds
A tempting argument is to apply Chebyshev’s inequality at every \(n\). Since \(\operatorname{Var}(\overline X_n)=\sigma^2/n\), it gives $$ \mathbb{P}(|\overline X_n-\mu|\ge\varepsilon) \le\frac{\sigma^2}{n\varepsilon^2}. $$ The concrete problem is that summing these bounds over \(n\) gives \(\frac{\sigma^2}{\varepsilon^2}\sum_{n\ge1}1/n=\infty\). The first Borel–Cantelli lemma cannot be applied to this bound. The incorrect calculation is not a counterexample to the theorem; it is a failure of that proof strategy. The correction is to truncate the variables and control whole stretches of partial sums with Kolmogorov’s inequality, producing a summable series of probabilities.
The roles of the assumptions are distinct. Finite variance supplies the summability estimate for the centered truncated terms. Truncation makes the probability of changing infinitely many observations zero. Independence makes the maximal inequality available. Together these yield almost-sure convergence, rather than only convergence in probability.
Check Your Understanding
Use the definitions and inequalities in the proof to answer these questions.
- For independent, mean-zero \(U_1,\ldots,U_n\), write Kolmogorov’s bound for \(\mathbb{P}(\max_{1\le k\le n}|R_k|\ge\lambda)\).
- Why does \(\sum_i\mathbb{P}(|Z_i|>i)<\infty\) imply that \(Y_i=Z_i\) for all but finitely many indices almost surely?
- In the proof, what property of the sequence \(W_i/i\) allows the maximal inequality to control the tails of its partial sums?
- For iid variables with mean \(5\) and variance \(3\), what does the finite-variance strong law say about their sample averages?
- Why does the Chebyshev bound proportional to \(1/n\) fail to prove almost-sure convergence by a direct application of the first Borel–Cantelli lemma?