Almost-Sure Convergence and the Strong Law

Why the Strong Law Matters

A sample average can vary from one sample size to another. The strong law describes what happens when we follow the averages from one growing sample to the next: under specified conditions, they settle toward the population mean with probability one. This is a long-run guarantee about the sequence of averages, not a claim that any particular finite sample average must be close to the mean.

The distinction between “with probability one” and “with probability approaching one” is important. The first describes almost-sure convergence; the second describes convergence in probability, the mode of convergence used in the weak law. The strong law gives the stronger conclusion, but its assumptions must be checked before applying it.

Definition: Let $X_1,X_2,\ldots$ be independent and identically distributed (iid) random variables, and define their sample average by $\bar X_n=(X_1+\cdots+X_n)/n$. If $E[|X_1|]<\infty$, let $\mu=E[X_1]$. The strong law of large numbers states that $$ P\left(\lim_{n\to\infty}\bar X_n=\mu\right)=1. $$

The condition $E[|X_1|]<\infty$ ensures that the mean is finite and well-defined. The observations must have the same distribution and be independent. A finite variance is not required for this version of the strong law. The theorem is about an infinite sequence of observations and the limit of its averages; it does not specify how large $n$ must be for a chosen level of accuracy.

Almost-Sure Convergence and Convergence in Probability

Almost-sure convergence means there is a set of outcomes with probability one on which the sequence of sample averages converges to $\mu$. There may be exceptional outcomes where convergence fails, but their total probability is zero. This is not the same as saying that every possible sequence of observations converges.

Convergence in probability instead describes the probability of being a fixed distance from the limit at a given sample size.

Definition: A sequence $Y_n$ converges in probability to a constant $\mu$ if, for every $\varepsilon>0$, $$ \lim_{n\to\infty}P(|Y_n-\mu|>\varepsilon)=0. $$ The weak law of large numbers gives this type of convergence for sample averages under its stated conditions. The strong law gives almost-sure convergence.

For the iid finite-mean setting in the definition, the strong law implies the weak law. To see why, fix any $\varepsilon>0$. If $\bar X_n$ converges to $\mu$ almost surely, then the indicator of the event $|\bar X_n-\mu|>\varepsilon$ converges to zero almost surely. Each indicator is bounded between zero and one, so the bounded convergence theorem gives

P(|\bar X_n-\mu|>\varepsilon) =E\!\left[\mathbf{1}_{\{|\bar X_n-\mu|>\varepsilon\}}\right] \longrightarrow 0.

Thus the strong law entails convergence in probability. The reverse implication is not true for general sequences of random variables. An example below shows how probabilities of deviations can tend to zero even though deviations occur infinitely often with probability one. That example is not an iid sample average, so it does not contradict the strong law.

A Short Justification in the Bounded Case

A simple proof illustrates why averages can converge along an entire sequence, rather than only at selected sample sizes. Suppose the $X_i$ are iid and bounded in the sense that $|X_i-\mu|\le C$ for a fixed constant $C$. Write $\sigma^2=\operatorname{Var}(X_1)$, which is finite because the observations are bounded. Independence gives $\operatorname{Var}(\bar X_n)=\sigma^2/n$.

Control averages at square sample sizes.
For any $\varepsilon>0$, Chebyshev’s inequality gives $P(|\bar X_{k^2}-\mu|>\varepsilon)\le \sigma^2/(\varepsilon^2 k^2)$. The sum of these bounds over $k\ge1$ is finite.

Conclude convergence along the squares.
By the first Borel–Cantelli lemma, events whose probabilities have a finite sum occur only finitely often with probability one. Applying this for $\varepsilon=1,1/2,1/3,\ldots$ shows that $\bar X_{k^2}\to\mu$ almost surely.

Fill the gaps between squares.
For $k^2\le n\le(k+1)^2$, write $\bar X_n-\mu=\frac{k^2}{n}(\bar X_{k^2}-\mu)+\frac{1}{n}\sum_{i=k^2+1}^n(X_i-\mu)$. The absolute value of the second term is at most $C(2k+1)/k^2$, which tends to zero. The first term also tends to zero almost surely because $k^2/n\le1$ and the square-indexed averages converge.

This proves the strong law in the bounded case. The finite-mean iid theorem does not require bounded observations or finite variance; its general proof uses additional control of large observations. The argument above is a justification of the long-run mechanism in a useful special case, not a proof of the full finite-mean theorem.

Worked Examples

Worked Example: A Discrete Annual Claim Amount

In a hypothetical portfolio, let $X_i$ be the claim amount, in dollars, for policy $i$ during one year. Assume the policy claim amounts are iid, with $P(X_i=0)=0.70$, $P(X_i=200)=0.20$, and $P(X_i=800)=0.10$. Find the mean claim amount and state the strong-law conclusion for the average across $n$ policies.

Method. Calculate the population mean, verify the finite-mean condition, and then apply the strong law. Independence and identical distribution are given in the model.

Step 1: Calculate the mean. The probability-weighted claim amount is

\mu=0(0.70)+200(0.20)+800(0.10)=0+40+80=120.

Step 2: Check the moment condition. The claim amount is nonnegative and bounded above by $800$, so $E[|X_i|]$ is finite. In particular, the strong law’s finite-mean condition holds.

Step 3: State the conclusion. For $\bar X_n=(X_1+\cdots+X_n)/n$,

P\left(\lim_{n\to\infty}\bar X_n=120\text{ dollars}\right)=1.

The average claim amount therefore converges almost surely to $120$ dollars as the number of policies grows. This does not state that the average for a particular finite portfolio equals $120$ dollars. The common mistake is to read the limit as a guarantee of exact equality at every $n$; the theorem guarantees convergence in the limit, not a finite-sample value.

Worked Example: A Heavy-Tailed Claim With Infinite Variance

Suppose hypothetical claim severities $X_i$, measured in thousands of dollars, are iid Pareto with density $f_X(x)=\frac32x^{-5/2}$ for $x\ge1$, and density zero otherwise. Determine whether the strong law applies, and find the limiting average severity.

Method. Check the first absolute moment, rather than requiring a finite variance. Since $X_i\ge1$, its absolute first moment is its mean.

Step 1: Verify the density’s mean.

E[X_i]=\int_1^\infty x\frac32x^{-5/2}\,dx =\frac32\int_1^\infty x^{-3/2}\,dx =\frac32\left[-2x^{-1/2}\right]_1^\infty =3.

Thus $E[|X_i|]=3<\infty$, in thousands of dollars.

Step 2: Check whether the variance is finite. The second moment is

E[X_i^2]=\frac32\int_1^\infty x^{-1/2}\,dx=\infty.

The variance is infinite, so a variance-based argument requiring finite variance cannot be used here. But independence, identical distribution, and a finite mean are all satisfied.

Step 3: Apply the strong law.

P\left(\lim_{n\to\infty}\bar X_n=3\text{ thousand dollars}\right)=1.

The average claim severity converges almost surely to $3{,}000$ dollars. The trap is to reject the strong law because the variance is infinite. For the finite-mean iid strong law, finite variance is not a required assumption.

Worked Example: Convergence in Probability Without Almost-Sure Convergence

For each integer $n\ge1$, let $Y_n$ be a Bernoulli random variable with $P(Y_n=1)=1/n$ and $P(Y_n=0)=1-1/n$. Assume the $Y_n$ are independent. Determine whether $Y_n$ converges in probability and almost surely to zero.

Method. Check the deviation probability for convergence in probability, then calculate the probability of having no more successes after a fixed index. This sequence is independent but not identically distributed, and it is not a sequence of sample averages.

Step 1: Check convergence in probability. For any $0<\varepsilon<1$, the event $|Y_n|>\varepsilon$ is exactly the event $Y_n=1$. Therefore

P(|Y_n|>\varepsilon)=\frac1n\longrightarrow0.

For $\varepsilon\ge1$, the deviation event is empty. Hence $Y_n$ converges in probability to zero.

Step 2: Check almost-sure convergence. Fix $m\ge2$. Independence gives

P(Y_n=0\text{ for every }n=m,\ldots,N) =\prod_{n=m}^N\left(1-\frac1n\right) =\prod_{n=m}^N\frac{n-1}{n} =\frac{m-1}{N}.

As $N\to\infty$, this probability tends to zero. Thus the probability of no successes at any index $n\ge m$ is zero. Taking the union over all $m\ge2$, the event of only finitely many successes also has probability zero. Consequently $Y_n=1$ infinitely often almost surely, so $Y_n$ does not converge almost surely to zero.

The correction to the mistaken claim “convergence in probability always means almost-sure convergence” is that these are distinct modes of convergence. There is no conflict with the iid strong law: the $Y_n$ have different distributions, and are not sample averages of iid observations.

Common Exam Traps

Requiring finite variance. The iid strong law stated here requires $E[|X_1|]<\infty$, not a finite variance. A distribution can have a finite mean and infinite variance, as in the Pareto example.
Confusing “probability one” with “for every outcome.” Almost-sure convergence allows an exceptional set of probability zero. It does not assert that every conceivable infinite sequence converges.
Claiming a finite-sample guarantee. The statement $\bar X_n\to\mu$ almost surely does not give a numerical bound on $P(|\bar X_n-\mu|>\varepsilon)$ for a particular $n$. Such a bound needs additional information and an appropriate result.
Ignoring the model assumptions. Do not apply the iid theorem to dependent observations or to observations that do not share a distribution without a theorem that covers that different setting.

Key takeaway: For iid observations with finite absolute mean, the sample average converges almost surely to the population mean. Almost-sure convergence implies convergence in probability, but the two notions are not interchangeable for arbitrary sequences.

Check Your Understanding

For each problem, identify the convergence conclusion justified by the stated model.

Hypothetical iid repair costs $X_i$, in dollars, equal $0$, $50$, or $150$ with probabilities $0.5$, $0.3$, and $0.2$, respectively. Find the mean and the almost-sure limit of $\bar X_n$. (Answer: $\mu=0(0.5)+50(0.3)+150(0.2)=45$ dollars; $\bar X_n\to45$ almost surely.)
Let iid observations have density $f_X(x)=2x^{-3}$ for $x\ge1$, zero otherwise. Find the mean and state whether the finite-mean strong law applies. (Answer: $E[X]=2\int_1^\infty x^{-2}\,dx=2$; the strong law applies, so $\bar X_n\to2$ almost surely.)
In the preceding problem, calculate $E[X^2]$. Does its value change the strong-law conclusion? (Answer: $E[X^2]=2\int_1^\infty x^{-1}\,dx=\infty$; it does not change the conclusion because the mean is finite.)
A sequence $Z_n$ has $P(Z_n=1)=1/n^2$ and $P(Z_n=0)=1-1/n^2$. Does $Z_n\to0$ in probability? (Answer: Yes; for $0<\varepsilon<1$, $P(|Z_n|>\varepsilon)=1/n^2\to0$.)
An analyst observes iid losses with $E[|X_1|]<\infty$ but does not know whether the variance is finite. What can be concluded about the sample average, and what cannot be concluded from the strong law alone? (Answer: $\bar X_n\to E[X_1]$ almost surely; the strong law alone gives no finite-$n$ error probability or sample-size guarantee.)

What You'll Learn