Chebyshev's Weak Law for IID Finite-Variance Samples

Why a Sample Average Settles Near Its Mean

A sample average is random: a new sample can produce a different value. The weak law of large numbers gives a precise sense in which this variability decreases as the sample grows. Under suitable assumptions, the probability that the average differs from its target by more than any fixed positive amount tends to zero.

Here the assumptions are especially transparent. The observations are independent and identically distributed, and their variance is finite. Independence makes the variance of the average shrink in proportion to $1/n$. Chebyshev’s inequality then converts that variance into a bound on the probability of a substantial error.

Setup and Definitions

Let $X_1,X_2,\ldots$ be independent and identically distributed random variables with $E[X_i]=\mu$ and $\operatorname{Var}(X_i)=\sigma^2<\infty$. For each positive integer $n$, define the sample average

\overline{X}_n=\frac{1}{n}\sum_{i=1}^n X_i.

Definition: The sequence $\overline{X}_n$ converges in probability to $\mu$, written $\overline{X}_n\xrightarrow{p}\mu$, if for every fixed $\varepsilon>0$, $$ \lim_{n\to\infty}P\bigl(|\overline{X}_n-\mu|>\varepsilon\bigr)=0. $$ The tolerance $\varepsilon$ is fixed while $n$ increases.

The definition asks about the probability of an error larger than each chosen positive tolerance. It does not require that every realized sequence of averages eventually stay within that tolerance. That stronger, path-by-path question belongs to almost-sure convergence.

Two Ingredients

First, the variance of a sum includes covariance terms. For finite-variance random variables $Y_1,\ldots,Y_n$,

\operatorname{Var}\left(\sum_{i=1}^nY_i\right) =\sum_{i=1}^n\operatorname{Var}(Y_i) +2\sum_{1\leq i<j\leq n}\operatorname{Cov}(Y_i,Y_j).

This identity follows by centering each variable and expanding the square in the definition of variance. For independent variables, distinct pairs have zero covariance: independence gives $E[Y_iY_j]=E[Y_i]E[Y_j]$, so $\operatorname{Cov}(Y_i,Y_j)=0$. The expectations in this calculation exist because finite second moments imply finite first moments, and the product is integrable by Cauchy–Schwarz.

Theorem: If $X_1,\ldots,X_n$ are independent with common mean $\mu$ and finite common variance $\sigma^2$, then $$ E[\overline{X}_n]=\mu, \qquad \operatorname{Var}(\overline{X}_n)=\frac{\sigma^2}{n}. $$

Linearity of expectation gives $E[\overline{X}_n]=n\mu/n=\mu$. For the variance, independence removes all covariance terms:

\operatorname{Var}(\overline{X}_n) =\frac{1}{n^2}\operatorname{Var}\left(\sum_{i=1}^nX_i\right) =\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(X_i) =\frac{n\sigma^2}{n^2} =\frac{\sigma^2}{n}.

Second, Chebyshev’s inequality bounds deviations using only a finite variance. For any random variable $Y$ with finite mean and variance, and every $t>0$,

P\bigl(|Y-E[Y]|\geq t\bigr) \leq \frac{\operatorname{Var}(Y)}{t^2}.

To see this, apply Markov’s inequality to the nonnegative random variable $Z=(Y-E[Y])^2$. For any $a>0$, $E[Z]\geq aP(Z\geq a)$, because $Z$ is at least $a$ on the event $\{Z\geq a\}$. Taking $a=t^2$ gives the displayed inequality.

Chebyshev’s Weak Law

Theorem: If $X_1,X_2,\ldots$ are IID with $E[X_i]=\mu$ and $\operatorname{Var}(X_i)=\sigma^2<\infty$, then for every $\varepsilon>0$ and every $n\geq1$, $$ P\bigl(|\overline{X}_n-\mu|\geq\varepsilon\bigr) \leq \frac{\sigma^2}{n\varepsilon^2}. $$ Consequently, $\overline{X}_n\xrightarrow{p}\mu$.

Proof. The variance calculation above gives $E[\overline{X}_n]=\mu$ and $\operatorname{Var}(\overline{X}_n)=\sigma^2/n$. Apply Chebyshev’s inequality to $Y=\overline{X}_n$ with $t=\varepsilon$:

P\bigl(|\overline{X}_n-\mu|\geq\varepsilon\bigr) \leq \frac{\operatorname{Var}(\overline{X}_n)}{\varepsilon^2} =\frac{\sigma^2}{n\varepsilon^2}.

For any fixed $\varepsilon>0$, the right-hand side tends to zero as $n\to\infty$. Since the event with strict deviation, $\{|\overline{X}_n-\mu|>\varepsilon\}$, is contained in the event with non-strict deviation, its probability also tends to zero. This is precisely convergence in probability. If $\sigma^2=0$, the bound is zero; indeed each $X_i=\mu$ almost surely, so the conclusion holds directly as well. $\square$

The conclusion is a finite-sample guarantee followed by a limit: for a fixed error threshold, the probability of exceeding it is at most a constant divided by $n$. The guarantee can be conservative, but it requires no particular distributional shape.

Worked Applications

Worked Example: A Bernoulli Sample Proportion

Suppose $X_i$ indicates whether patient $i$ experiences a specified outcome, with $P(X_i=1)=0.3$ and $P(X_i=0)=0.7$. Assume the indicators are IID, and let $\overline{X}_n$ be the observed proportion with the outcome. For $n=100$, bound the probability that this proportion differs from $0.3$ by at least $0.1$.

A Bernoulli variable with success probability $p$ has mean $p$ and variance $p(1-p)$. Thus $\mu=0.3$ and $\sigma^2=0.3(0.7)=0.21$. Chebyshev’s bound gives

P\bigl(|\overline{X}_{100}-0.3|\geq0.1\bigr) \leq\frac{0.21}{100(0.1)^2} =\frac{0.21}{1} =0.21.

Therefore the probability is at most $0.21$, or 21%. This is an upper bound, not an estimate of the actual probability. The calculation uses the sampling assumptions and variance, not a normal approximation.

Worked Example: A Normal Sample Average

Let $X_1,\ldots,X_{100}$ be IID $N(5,4)$, where the second parameter is the variance. Bound the probability that $\overline{X}_{100}$ differs from $5$ by at least $0.8$, and compare the bound with the exact probability.

Here $\mu=5$ and $\sigma^2=4$. Chebyshev’s inequality yields

P\bigl(|\overline{X}_{100}-5|\geq0.8\bigr) \leq \frac{4}{100(0.8)^2} =\frac{4}{64} =0.0625.

Because an average of independent normal variables is normal, $\overline{X}_{100}\sim N(5,4/100)=N(5,0.04)$. Its standard deviation is $0.2$, so a deviation of $0.8$ is four standard deviations. If $\Phi$ denotes the standard normal CDF, the exact probability is

P\bigl(|\overline{X}_{100}-5|\geq0.8\bigr) =2\Phi(-4)\approx0.0000633.

The exact probability is much smaller than the distribution-free bound. Chebyshev’s inequality remains valid without normality; knowing the normal model simply provides additional information for a sharper calculation.

Worked Example: Choosing a Sample Size for Exponential Data

Suppose $X_i$ are IID exponential random variables with rate $2$. Their mean is $1/2$ and variance is $1/4$. How large should $n$ be to ensure, using Chebyshev’s inequality, that the probability of an error of at least $0.1$ in the sample average is no more than $0.05$?

The theorem gives

P\bigl(|\overline{X}_n-0.5|\geq0.1\bigr) \leq\frac{1/4}{n(0.1)^2} =\frac{0.25}{0.01n} =\frac{25}{n}.

To make this upper bound at most $0.05$, require $25/n\leq0.05$. Since $n$ is positive, this is equivalent to $n\geq25/0.05=500$. Thus $n=500$ suffices, and substitution verifies the guarantee:

\frac{25}{500}=0.05.

This is a sufficient sample size according to the bound, not necessarily the smallest sample size that achieves the probability target under the exponential model.

What the Assumptions Do—and Do Not—Say

Identical distribution gives every observation the same mean and variance; independence eliminates the covariance terms that would otherwise enter the variance of the sum. Finite variance makes Chebyshev’s inequality available and supplies the explicit rate $\sigma^2/(n\varepsilon^2)$. Each assumption has a role in this particular proof.

The conclusion is robust to the form of the common distribution. The observations need not be normal, symmetric, or bounded. But the theorem does not say that every observation is close to $\mu$, nor does the probability bound itself establish almost-sure convergence. It controls the probability of a fixed-size deviation of the average.

Common Mistakes

Forgetting the square on the tolerance. The bound is $\sigma^2/(n\varepsilon^2)$, not $\sigma^2/(n\varepsilon)$. The square comes from applying Markov’s inequality to the squared deviation.
Using $\sigma^2/n$ without checking dependence. This variance formula follows here because independent observations have zero pairwise covariance. In general, the variance of a sum includes covariance terms.
Treating an upper bound as an exact probability. Chebyshev’s inequality guarantees that the probability is no larger than the stated number; it need not be close to that number.
Letting the tolerance vary without justification. Convergence in probability checks each fixed $\varepsilon>0$. A bound for a tolerance that changes with $n$ is a different claim and must be analyzed separately.

Key takeaway: For IID observations with finite variance, the average has variance $\sigma^2/n$. Chebyshev’s inequality then bounds every fixed-threshold error probability by $\sigma^2/(n\varepsilon^2)$, which tends to zero and proves convergence in probability to the common mean.

Check Your Understanding

Use the assumptions carefully when answering.

If $\sigma^2=9$, $n=225$, and $\varepsilon=0.2$, what upper bound does Chebyshev’s inequality give for $P(|\overline{X}_n-\mu|\geq\varepsilon)$?
Which step in the variance calculation uses independence?
For IID Bernoulli observations with success probability $p$, write the Chebyshev bound for the sample proportion to differ from $p$ by at least $\varepsilon$.
Why does a small Chebyshev upper bound not generally equal the true deviation probability?
What is held fixed when using the theorem to conclude convergence in probability?