From Variance Bounds to Almost-Sure Convergence
A variance estimate often gives useful control of deviations at a fixed sample size. To obtain convergence almost surely for all sample sizes, one needs more: a way to turn those estimates into an event that occurs only finitely often, and a way to fill the gaps between the selected sample sizes. For bounded independent, identically distributed variables, square indices provide a direct route. The variance bounds are summable along the squares, and bounded increments control the intervals between them.
Work on a probability space \((\Omega,\mathcal{F},\mathbb{P})\), which is a finite measure space with total measure \(1\). Let \(X_1,X_2,\ldots\) be measurable, real-valued random variables that are independent and identically distributed. Suppose there is a finite constant \(K\ge 0\) such that \(\mathbb{P}(|X_j|\le K)=1\) for every \(j\). Write \(m=\mathbb{E}[X_1]\), which is finite because \(X_1\) is bounded almost surely. The sample average is \(n^{-1}\sum_{j=1}^n X_j\). The conclusion below is convergence to \(m\), \(\mathbb{P}\)-almost surely, not merely convergence in probability.
The proof uses only a variance estimate, the first Borel–Cantelli lemma, and boundedness to interpolate. We give the needed probability estimate explicitly. If \(Z\) is a real-valued random variable with finite second moment and \(\mathbb{E}[Z]=0\), then for every \(t>0\), $$ \mathbb{P}(|Z|\ge t)\le \frac{\mathbb{E}[Z^2]}{t^2}. $$ Indeed, \(Z^2\ge t^2\) on \(\{|Z|\ge t\}\), so \(\mathbb{E}[Z^2]\ge t^2\mathbb{P}(|Z|\ge t)\). This is Chebyshev’s inequality in the centered case.
Proof. For each \(N\), the event that infinitely many \(A_k\) occur is contained in \(\bigcup_{k\ge N}A_k\). Countable subadditivity gives $$ \mathbb{P}(A_k\text{ for infinitely many }k) \le \mathbb{P}\left(\bigcup_{k\ge N}A_k\right) \le \sum_{k\ge N}\mathbb{P}(A_k). $$ The last quantity tends to \(0\) as \(N\to\infty\), since the series converges. The probability on the left is therefore zero. \(\square\)
Proof Along the Squares, Then Between Them
Set \(Y_j=X_j-m\) and \(S_n=\sum_{j=1}^nY_j\). Then \(\mathbb{E}[Y_j]=0\), and the \(Y_j\) remain independent and identically distributed. Let \(\sigma^2=\mathbb{E}[Y_1^2]\). This is finite by boundedness. Also, \(|m|\le K\), so \(|Y_j|\le 2K\) almost surely for each \(j\). As there are only countably many indices, all these bounds hold simultaneously outside one null set.
Independence and zero means give \(\mathbb{E}[Y_iY_j]=\mathbb{E}[Y_i]\mathbb{E}[Y_j]=0\) when \(i\ne j\). Expanding the square, and using identical distribution for the diagonal terms, yields \(\mathbb{E}[S_n^2]=n\sigma^2\). Thus, for every \(\varepsilon>0\) and integer \(k\ge1\), Chebyshev’s inequality gives $$ \mathbb{P}\left(\frac{|S_{k^2}|}{k^2}\ge\varepsilon\right) \le \frac{\mathbb{E}[S_{k^2}^2]}{\varepsilon^2 k^4} =\frac{\sigma^2}{\varepsilon^2 k^2}. $$ The series of these upper bounds converges. By the first Borel–Cantelli lemma, for each fixed \(\varepsilon>0\), the inequality \(|S_{k^2}|/k^2\ge\varepsilon\) occurs only finitely often almost surely. Apply this to \(\varepsilon=1/r\) for every positive integer \(r\). The intersection of these countably many probability-one events still has probability one. On that intersection, \(S_{k^2}/k^2\to0\).
It remains to control sample sizes between squares. On the probability-one event where \(|Y_j|\le 2K\) for all \(j\), take \(k^2\le n<(k+1)^2\). There are at most \(2k\) terms between \(S_{k^2}\) and \(S_n\), so \(|S_n-S_{k^2}|\le 2K(2k+1)\); the slightly larger bound \(2K(2k+1)\) also covers the endpoint gap. More directly, since \(n-k^2\le 2k\), we have $$ \frac{|S_n|}{n} \le \frac{|S_{k^2}|}{n}+\frac{2K(n-k^2)}{n} \le \frac{|S_{k^2}|}{k^2}+\frac{2K(2k+1)}{k^2}. $$ As \(n\to\infty\), its associated \(k\) tends to infinity. The first term tends to zero by square-subsequence convergence; the second tends to zero because \((2k+1)/k^2\to0\). Hence \(S_n/n\to0\) almost surely. Since \(S_n/n=n^{-1}\sum_{j=1}^nX_j-m\), the theorem follows. If \(K=0\), every \(X_j=0\) almost surely and the same conclusion holds directly. \(\square\)
Worked Applications
Worked Example: Bernoulli Variables with Success Probability One Third
Let \(X_j\) be IID Bernoulli variables with \(\mathbb{P}(X_j=1)=1/3\) and \(\mathbb{P}(X_j=0)=2/3\). Each variable is real-valued and bounded by \(1\). Direct calculation gives $$ \mathbb{E}[X_j]=1\cdot\frac13+0\cdot\frac23=\frac13, \qquad \mathbb{E}[X_j^2]=1^2\cdot\frac13+0^2\cdot\frac23=\frac13, $$ so \(\operatorname{Var}(X_j)=\mathbb{E}[X_j^2]-(\mathbb{E}[X_j])^2=\frac13-\frac19=\frac29\). For centered variables \(Y_j=X_j-\frac13\), the possible values are \(-\frac13\) and \(\frac23\), and hence \(|Y_j|\le\frac23\). Along square indices, the estimate in the proof becomes $$ \mathbb{P}\left(\left|\frac{1}{k^2}\sum_{j=1}^{k^2}X_j-\frac13\right|\ge\varepsilon\right) \le \frac{2}{9\varepsilon^2 k^2}. $$ The right side sums to a finite value over \(k\). The theorem therefore gives \(n^{-1}\sum_{j=1}^nX_j\to1/3\) almost surely. Check: the target is the directly computed mean \(1/3\), and the centered variance used in the estimate is \(2/9\).
Worked Example: A Symmetric Two-Point Distribution
Suppose the IID variables take the values \(-1\) and \(1\), each with probability \(1/2\). Their expectation is \((-1)(1/2)+(1)(1/2)=0\), and their second moment is \((-1)^2(1/2)+(1)^2(1/2)=1\), so their variance is \(1\). They satisfy \(|X_j|\le1\). For every \(\varepsilon>0\), the square-index estimate is $$ \mathbb{P}\left(\left|\frac{S_{k^2}}{k^2}\right|\ge\varepsilon\right) \le \frac{1}{\varepsilon^2 k^2}. $$ It is summable in \(k\), so the square averages tend to \(0\) almost surely. Between \(k^2\) and \((k+1)^2\), the centered increments have absolute value at most \(1\), and the interpolation term is bounded by \((2k+1)/k^2\), which tends to \(0\). Thus the full sample averages tend to \(0\) almost surely. Check: the limiting value agrees with the direct mean \((-1+1)/2=0\), and the variance is \(1\).
Why Independence Matters
Boundedness controls increments, but it does not make sums concentrate. Independence was used to make the cross terms \(\mathbb{E}[Y_iY_j]\) vanish. The next example shows that identical distributions and boundedness alone do not imply the conclusion.
Worked Example: Bounded Identical Variables That Are Not Independent
Take \(\Omega=\{-1,1\}\) with probability \(1/2\) at each point, and define \(X_j(\omega)=\omega\) for every \(j\). Each \(X_j\) is bounded by \(1\), and each has expectation \((-1)(1/2)+(1)(1/2)=0\). But the variables are not independent: for example, \(\mathbb{P}(X_1=1,X_2=1)=1/2\), whereas independence would require \(\mathbb{P}(X_1=1)\mathbb{P}(X_2=1)=1/4\). For every \(n\), $$ \frac1n\sum_{j=1}^nX_j(\omega)=\frac1n(n\omega)=\omega. $$ This average is always either \(-1\) or \(1\), so it does not converge to the expectation \(0\) at either point of \(\Omega\). In particular, the event of convergence to \(0\) has probability \(0\). This verifies concretely that independence cannot simply be omitted from the theorem.
A frequent mistake is to apply Chebyshev’s inequality at every integer \(n\), then expect the resulting probabilities to be summable. Here the direct bound is \(\mathbb{P}(|S_n|/n\ge\varepsilon)\le\sigma^2/(\varepsilon^2 n)\), whose upper bounds have a divergent harmonic sum. That calculation does not justify Borel–Cantelli at all indices. The square subsequence changes the bound to a constant times \(1/k^2\), which is summable; the bounded-increment estimate is then essential to recover convergence at every intervening index.
The argument illustrates a useful proof structure: summable deviation probabilities establish almost-sure control on a subsequence, while a deterministic estimate transfers that control to the full sequence. Here the common distribution supplies a finite common mean and variance, independence gives the variance of each partial sum, and boundedness supplies the interpolation estimate. The conclusion is specifically almost-sure convergence; it says that the set of sample paths on which the averages fail to converge has probability zero.
Check Your Understanding
For each question, identify where boundedness, identical distribution, or independence enters the reasoning.
- For IID Bernoulli variables with success probability \(p\), calculate the mean and variance, then write the Chebyshev bound at sample size \(k^2\).
- Why does summability of the square-index deviation probabilities imply convergence along squares? State the role of the countable thresholds \(1/r\).
- If \(|X_j|\le K\) almost surely, what bound can be used for \(|X_j-\mathbb{E}[X_j]|\), and why does it hold simultaneously for all indices outside one null set?
- Explain why a bound proportional to \(1/n\) at all integer indices does not suffice for the first Borel–Cantelli lemma.
- For the common-variable construction \(X_j(\omega)=\omega\) on \(\{-1,1\}\), calculate the sample average and compare it with the expectation.