The Weak Law of Large Numbers by Chebyshev's Inequality

From Variance to a Weak Law

Suppose repeated measurements share a common distribution and are independent. Their average may fluctuate from sample to sample, but as the sample grows, those fluctuations become less likely to exceed any fixed positive tolerance. The weak law of large numbers makes this statement precise: the sample average converges to the common mean in probability. With a finite variance, Chebyshev’s inequality gives a direct proof and an explicit upper bound on the probability of a deviation.

Work on a probability space $(\Omega,\mathcal{F},\mathbb{P})$. Let $(X_k)_{k\ge1}$ be independent and identically distributed random variables with $\mathbb{E}[X_k]=\mu$ and $\operatorname{Var}(X_k)=\sigma^2<\infty$. For each positive integer $n$, define the sample average $\overline{X}_n=\frac{1}{n}\sum_{k=1}^n X_k$. Convergence in probability of $\overline{X}_n$ to $\mu$ means that for every $\varepsilon>0$, $\mathbb{P}(|\overline{X}_n-\mu|>\varepsilon)$ tends to zero as $n$ tends to infinity.

Definition: The weak law of large numbers for this sequence is the assertion $\overline{X}_n\xrightarrow{\mathbb{P}}\mu$: for every fixed $\varepsilon>0$, the probability that the sample average differs from $\mu$ by more than $\varepsilon$ tends to zero.

The proof has two ingredients. First, independence makes the variance of the average equal to $\sigma^2/n$. Second, Chebyshev’s inequality bounds the probability of a deviation using that variance. Because $\varepsilon$ is fixed and positive, the resulting bound decreases to zero.

Chebyshev’s Inequality and the Variance Calculation

Theorem (Chebyshev’s inequality): If $Y$ has finite mean $a=\mathbb{E}[Y]$ and finite variance $v=\operatorname{Var}(Y)$, then for every $\varepsilon>0$, $$ \mathbb{P}(|Y-a|\ge\varepsilon)\le \frac{v}{\varepsilon^2}. $$

Proof. On the event $\{|Y-a|\ge\varepsilon\}$, the nonnegative quantity $(Y-a)^2$ is at least $\varepsilon^2$. Therefore, pointwise, $\mathbf{1}_{\{|Y-a|\ge\varepsilon\}}\le (Y-a)^2/\varepsilon^2$. Taking expectations preserves the inequality. The expectation on the left is the probability of the event, and the expectation on the right is $\mathbb{E}[(Y-a)^2]/\varepsilon^2=v/\varepsilon^2$. This proves the stated bound. $\square$

To use the inequality for a sample average, we calculate its mean and variance. Linearity of expectation gives $\mathbb{E}[\overline{X}_n]=\mu$. For the variance, write each centered summand as $X_k-\mu$. Independence implies that distinct centered summands have covariance zero: for $i\ne j$, $\mathbb{E}[(X_i-\mu)(X_j-\mu)]=\mathbb{E}[X_i-\mu]\mathbb{E}[X_j-\mu]=0$. These expectations and products are well-defined because the variables have finite second moments. Thus the variance of the sum is the sum of its variances.

\operatorname{Var}(\overline{X}_n) =\frac{1}{n^2}\operatorname{Var}\!\left(\sum_{k=1}^n X_k\right) =\frac{1}{n^2}\sum_{k=1}^n\operatorname{Var}(X_k) =\frac{\sigma^2}{n}.

Theorem (Weak law by Chebyshev’s inequality): If $(X_k)_{k\ge1}$ are iid with finite mean $\mu$ and finite variance $\sigma^2$, then $\overline{X}_n\xrightarrow{\mathbb{P}}\mu$. More precisely, for every $\varepsilon>0$, $$ \mathbb{P}(|\overline{X}_n-\mu|>\varepsilon) \le \frac{\sigma^2}{n\varepsilon^2}. $$

Proof. If $\sigma^2=0$, then $\mathbb{E}[(X_k-\mu)^2]=0$, so $X_k=\mu$ almost surely for each $k$. For each $n$, the average is therefore $\mu$ almost surely, and the claimed convergence holds. Now suppose $\sigma^2>0$. The variance calculation above gives $\mathbb{E}[\overline{X}_n]=\mu$ and $\operatorname{Var}(\overline{X}_n)=\sigma^2/n$. Since $\{|\overline{X}_n-\mu|>\varepsilon\}$ is contained in $\{|\overline{X}_n-\mu|\ge\varepsilon\}$, Chebyshev’s inequality gives $\mathbb{P}(|\overline{X}_n-\mu|>\varepsilon)\le\sigma^2/(n\varepsilon^2)$. For fixed $\varepsilon>0$, the right side tends to zero as $n$ tends to infinity. This is exactly convergence in probability to $\mu$. $\square$

The bound is a guarantee, not usually an exact probability. Its value can exceed one for small $n$, in which case it is still a valid but uninformative upper bound; probabilities themselves never exceed one. A useful bound arises when $n$ is large enough, or the tolerance is sufficiently wide, to make $\sigma^2/(n\varepsilon^2)$ small.

Worked Example: A Bernoulli Sample Average

Let $X_1,\ldots,X_n$ be iid Bernoulli random variables with success probability $1/3$. Find a Chebyshev bound for the probability that the sample proportion differs from $1/3$ by at least $1/6$, when $n=40$.

For a Bernoulli variable, $X_k$ equals $1$ with probability $1/3$ and $0$ with probability $2/3$. Its mean is $\mu=1\cdot\frac13+0\cdot\frac23=\frac13$. Its second moment is $\mathbb{E}[X_k^2]=1^2\cdot\frac13+0^2\cdot\frac23=\frac13$, so its variance is $\sigma^2=\frac13-(\frac13)^2=\frac29$. The sample proportion is $\overline{X}_{40}$.

Substituting $n=40$, $\varepsilon=\frac16$, and $\sigma^2=\frac29$ into the bound gives

\mathbb{P}\!\left(\left|\overline{X}_{40}-\tfrac13\right|\ge\tfrac16\right) \le \frac{2/9}{40(1/6)^2} =\frac{2/9}{40/36} =\frac{2}{9}\cdot\frac{36}{40} =\frac{72}{360} =\frac15.

Check: The sample-average variance is $(2/9)/40=1/180$, and dividing it by $(1/6)^2=1/36$ gives $(1/180)/(1/36)=36/180=1/5$, as above. Thus the probability is at most $1/5$; the calculation does not claim it equals $1/5$.

Worked Example: A Two-Point Distribution

Let each $X_k$ equal $0$ or $2$, each with probability $1/2$, independently across $k$. Bound the probability that the average of $100$ observations differs from its mean by at least $1/2$.

The mean is $\mu=0\cdot\frac12+2\cdot\frac12=1$. The second moment is $\mathbb{E}[X_k^2]=0^2\cdot\frac12+2^2\cdot\frac12=2$, so $\sigma^2=2-1^2=1$. Using $n=100$ and $\varepsilon=1/2$,

\mathbb{P}\!\left(|\overline{X}_{100}-1|\ge\tfrac12\right) \le\frac{1}{100(1/2)^2} =\frac{1}{100/4} =\frac{4}{100} =\frac{1}{25}.

Check: The variance of the average is $1/100$. Dividing by the squared tolerance, $(1/2)^2=1/4$, gives $(1/100)/(1/4)=4/100=1/25$. This agrees with the direct substitution.

Worked Example: A Mean-Zero Sample Average

Suppose $X_k=-1$ with probability $2/3$ and $X_k=2$ with probability $1/3$, independently across $k$. Find a bound for $\mathbb{P}(|\overline{X}_{32}|\ge1/2)$.

The mean is $\mu=(-1)(2/3)+2(1/3)=-2/3+2/3=0$. The second moment is $\mathbb{E}[X_k^2]=(-1)^2(2/3)+2^2(1/3)=2/3+4/3=2$. Therefore $\sigma^2=2-0^2=2$. Substitution into Chebyshev’s bound yields

\mathbb{P}\!\left(|\overline{X}_{32}|\ge\tfrac12\right) \le\frac{2}{32(1/2)^2} =\frac{2}{32/4} =\frac{2}{8} =\frac14.

Check: The average has variance $2/32=1/16$. Dividing by the squared tolerance $1/4$ gives $(1/16)/(1/4)=4/16=1/4$, confirming the bound.

Why Independence Matters

The factor $1/n$ in the variance of the sample average comes from the disappearance of cross-covariances. Identical distributions and finite variances alone do not guarantee that this calculation works. In particular, a collection of observations that are perfectly dependent can keep its average random no matter how many terms are included.

Worked Example: A Failure Without Independence

Let $Z$ equal $1$ or $-1$, each with probability $1/2$, and set $X_k=Z$ for every $k$. Each $X_k$ has mean $0$ and variance $1$, but the sequence is not independent. For every $n$, $\overline{X}_n=Z$, because the average of $n$ copies of $Z$ is $Z$. Hence, for $\varepsilon=1/2$,

\mathbb{P}(|\overline{X}_n-0|>\tfrac12) =\mathbb{P}(|Z|>\tfrac12) =1

for every $n$. The averages do not converge in probability to $0$. The failed variance calculation is visible directly: $\operatorname{Var}(\overline{X}_n)=\operatorname{Var}(Z)=1$, not $1/n$. The $n$ summands have nonzero covariance, so their variances cannot simply be added as if they were independent.

Common mistake: treating the variance of an average as unchanged. In the Bernoulli example, one might incorrectly use $\operatorname{Var}(\overline{X}_{40})=2/9$, the variance of one observation. That would give the purported bound $(2/9)/(1/6)^2=8$. The correct variance is $(2/9)/40=1/180$, so the bound is $(1/180)/(1/36)=1/5$. The missing factor $1/40$ comes from averaging 40 independent observations. The bound $8$ is not a useful probability estimate, and it does not express the reduction in variance provided by averaging.

Chebyshev’s inequality applies to events with $\ge\varepsilon$, while convergence in probability is commonly expressed using the strict event $>\varepsilon$. The strict event is a subset of the non-strict event, so its probability is bounded by the same Chebyshev estimate. This distinction does not change the limit. The finite-variance iid assumptions give a sufficient route to the weak law; they are not a claim that every sequence satisfying a weak law must have finite variance.

Check Your Understanding

For each question, identify the assumptions being used and show the variance or probability calculation that supports your answer.

If iid observations have variance $9$, what is the variance of their average based on $n$ observations?
For iid variables with mean $\mu$ and variance $4$, write the Chebyshev bound for $\mathbb{P}(|\overline{X}_n-\mu|>2)$.
Why does the proof use independence when computing the variance of the sum?
In the dependent example, what is $\operatorname{Var}(\overline{X}_n)$ for each $n$, and why does the sequence fail to satisfy the weak law?
Does an upper bound of $3/2$ imply that an event has probability $3/2$? Explain what such a bound tells you.

The Weak Law of Large Numbers by Chebyshev's Inequality

What You'll Learn