The Strong Law for IID Variables with Finite Variance

From Finite Variance to Almost-Sure Convergence

For independent observations with a common finite variance, the sample average is close to its mean with high probability. The strong law makes a stronger assertion: with probability one, the averages converge to that mean along the entire sequence of sample sizes. A direct Chebyshev estimate at each sample size gives probabilities of order \(1/n\), which are not summable. The useful strategy is to control the averages at dyadic sample sizes and then control all the observations between consecutive dyadic sizes with a maximal inequality.

Throughout, \((\Omega,\mathcal{F},\mathbb{P})\) is a probability space, so \(\mathbb{P}(\Omega)=1\). Let \(X_1,X_2,\ldots\) be IID real-valued random variables: they are independent and have the same distribution. Assume \(\mathbb{E}[X_1^2]<\infty\). In particular, the mean \(\mu=\mathbb{E}[X_1]\) is finite, and the variance \(\sigma^2=\mathbb{E}[(X_1-\mu)^2]\) is finite. Define the centered variables \(Y_j=X_j-\mu\) and partial sums \(S_n=\sum_{j=1}^nY_j\). The sample average of the \(X_j\) is \(\mu+S_n/n\).

The Two Probability Estimates

We use Kolmogorov’s maximal inequality in its standard form, established in the earlier material on Kolmogorov’s maximal inequality. If \(Z_1,\ldots,Z_N\) are independent, real-valued, mean-zero random variables with finite variances, then for every \(t>0\),

The strict inequality in the event is the one used here; the same bound also holds with \(\ge t\). For the IID centered variables \(Y_j\), each has mean zero and variance \(\sigma^2\). Thus a block of \(N\) consecutive \(Y_j\)'s has total variance \(N\sigma^2\), and the maximal inequality applies to the partial sums within that block.

We also use the first Borel–Cantelli lemma: if events \(A_k\) satisfy \(\sum_{k=1}^{\infty}\mathbb{P}(A_k)<\infty\), then with probability one only finitely many \(A_k\) occur. Indeed, for every \(K\), the probability that some \(A_k\) with \(k\ge K\) occurs is at most \(\sum_{k\ge K}\mathbb{P}(A_k)\), by countable subadditivity. This tail sum tends to zero as \(K\to\infty\). The event that infinitely many \(A_k\) occur is contained in each of these tail unions, so its probability is zero.

Proof on Dyadic Blocks

Proof. Fix \(\varepsilon>0\). For each integer \(k\ge0\), first consider the sum at the left endpoint of the dyadic block. Since the \(Y_j\) are independent, centered, and have variance \(\sigma^2\), \(\mathbb{E}[S_{2^k}^2]=2^k\sigma^2\). Chebyshev’s inequality therefore gives

To control the rest of the block, apply Kolmogorov’s maximal inequality to \(Y_{2^k+1},\ldots,Y_{2^{k+1}}\), a block of \(2^k\) independent centered variables. Write \(T_{k,j}=\sum_{i=1}^{j}Y_{2^k+i}\) for \(1\le j\le2^k\). Then

Both bounds are summable over \(k\), because \(\sum_{k=0}^{\infty}2^{-k}=2\). In particular, the sum of the endpoint-event probabilities is at most \(2\sigma^2/\varepsilon^2\), and the sum of the block-event probabilities is also at most \(2\sigma^2/\varepsilon^2\). Applying the first Borel–Cantelli lemma to each sequence of events shows that, with probability one, for all sufficiently large \(k\), both

For \(2^k\le n\le2^{k+1}\), write \(n=2^k+j\) with \(0\le j\le2^k\). If \(j=0\), then \(S_n=S_{2^k}\); if \(j\ge1\), then \(S_n=S_{2^k}+T_{k,j}\). Consequently, whenever the two displayed bounds hold, \(|S_n|\le2\varepsilon 2^k\) throughout the block, and hence \(|S_n|/n\le2\varepsilon\), since \(n\ge2^k\).

To conclude convergence to zero rather than just a bound for one fixed \(\varepsilon\), apply this argument for \(\varepsilon=1/r\), for each positive integer \(r\). The intersection of these countably many probability-one events still has probability one. On that intersection, for every \(r\), all sufficiently large \(n\) satisfy \(|S_n|/n\le2/r\). Given any \(\delta>0\), choose an integer \(r>2/\delta\); then \(2/r<\delta\), so \(|S_n|/n<\delta\) eventually. Thus \(S_n/n\to0\) almost surely, and adding \(\mu\) proves the asserted convergence of sample averages. \(\square\)

If \(\sigma^2=0\), the same estimates remain valid and have right side zero. Equivalently, \(\mathbb{E}[(X_1-\mu)^2]=0\) implies \(X_1=\mu\) almost surely. Since there are countably many \(j\), with probability one every \(X_j=\mu\), and the averages are then exactly \(\mu\). Thus the zero-variance case is included, not excluded by the proof.

Worked Applications

Why the Dyadic Step Matters

A tempting but incomplete argument is to apply Chebyshev’s inequality separately for every \(n\) and then invoke Borel–Cantelli. For fixed \(\varepsilon>0\), the actual estimate is

The resulting upper bounds have a harmonic series, which diverges when \(\sigma^2>0\). This does not show that the strong law is false; it shows that this direct estimate does not verify the summability hypothesis of Borel–Cantelli. The dyadic proof changes the task: it uses a summable bound at the endpoints \(2^k\), and the maximal inequality controls every partial sum across the entire intervening block using a single event. The block length and the squared threshold together produce the summable factor \(2^{-k}\).

The finite-variance hypothesis enters quantitatively through \(\mathbb{E}[Y_j^2]=\sigma^2<\infty\). Independence is needed both for the variance of a sum to be the sum of the variances and for Kolmogorov’s maximal inequality. Identical distribution ensures that the common mean and variance used throughout are those of every \(X_j\). The next strengthening of the strong law drops the finite-variance requirement, so this proof should not be read as covering all integrable IID variables.