The Weak Law Of Large Numbers With Finite Variance

Why the Sample Mean Settles Near Its Target

In econometric arguments, a sample mean often serves as an estimator of a population mean. A basic large-sample question is whether that estimator gets arbitrarily close to its target as the sample grows. The weak law of large numbers answers yes under a simple and useful set of assumptions: the observations are independent and identically distributed, and their variance is finite.

The result is about convergence in probability, not a claim that the sample mean must get closer for every realized sample as observations are added. Instead, for any fixed positive tolerance, the probability that the sample mean misses the population mean by at least that tolerance tends to zero.

Definition: Let $X_1,X_2,\ldots$ be independent and identically distributed random variables with $E[X_i]=\mu$ and $\operatorname{Var}(X_i)=\sigma^2<\infty$. Their sample mean is $\bar{X}_n=n^{-1}\sum_{i=1}^n X_i$. The weak law of large numbers states that $\bar{X}_n\xrightarrow{p}\mu$, meaning that for every $\varepsilon>0$, $P(|\bar{X}_n-\mu|\geq\varepsilon)\to 0$ as $n\to\infty$.

The assumptions apply to the data-generating random variables, not just to the observed values in one sample. Identical distribution gives every observation the same mean and variance; independence ensures that covariances between distinct observations are zero. Finite variance means each observation has a well-defined variance $\sigma^2$ that is not infinite. The theorem does not require the $X_i$ to be normally distributed.

Here $\bar{X}_n$ denotes the random sample mean before the observations are realized. For a particular observed sample, its numerical value can be written $\bar{x}_n$. The convergence statement concerns the sampling distribution of $\bar{X}_n$, rather than a single observed value.

Deriving the Probability Bound

The key calculation is the variance of the sample mean. Linearity of expectation gives $E[\bar{X}_n]=\mu$. Independence and identical variance imply that the variance of the sum is the sum of the variances. Thus,

E[\bar{X}_n] =\frac{1}{n}\sum_{i=1}^{n}E[X_i] =\mu, \qquad \operatorname{Var}(\bar{X}_n) =\frac{1}{n^2}\sum_{i=1}^{n}\operatorname{Var}(X_i) =\frac{\sigma^2}{n}.

For a random variable $Y$ with finite variance, Chebyshev’s inequality states that $P(|Y-E[Y]|\geq\varepsilon)\leq\operatorname{Var}(Y)/\varepsilon^2$ for every $\varepsilon>0$. Apply it to $Y=\bar{X}_n$, whose expectation and variance were just calculated:

P\left(|\bar{X}_n-\mu|\geq\varepsilon\right) \leq \frac{\operatorname{Var}(\bar{X}_n)}{\varepsilon^2} = \frac{\sigma^2}{n\varepsilon^2}.

This inequality provides both the proof of the weak law and a usable finite-sample bound. For a fixed $\varepsilon>0$, the numerator $\sigma^2$ and the tolerance $\varepsilon$ do not change with $n$, while the denominator increases without bound. Therefore the upper bound tends to zero. Since the probability is nonnegative and no larger than this vanishing bound, it too tends to zero. This is precisely the definition of $\bar{X}_n\xrightarrow{p}\mu$.

Theorem: If $X_1,X_2,\ldots$ are iid with finite mean $\mu$ and finite variance $\sigma^2$, then for every $\varepsilon>0$, $$ P\left(|\bar{X}_n-\mu|\geq\varepsilon\right) \leq\frac{\sigma^2}{n\varepsilon^2}, $$ and consequently $\bar{X}_n\xrightarrow{p}\mu$.

The assumptions have distinct roles in this argument. Finite variance makes Chebyshev’s inequality applicable and supplies a finite numerator. Independence removes covariance terms from the variance of the sum. Identical distribution ensures a common mean $\mu$ and variance $\sigma^2$, so the sample mean targets that same $\mu$ and its variance is $\sigma^2/n$. If these conditions fail, this proof does not establish the stated bound.

Using the Bound in Concrete Settings

A Chebyshev bound is often conservative: it guarantees that the probability is no greater than a stated number, but does not usually calculate the exact probability. In each example, the distributional assumptions are stated before the bound is used.

Worked Example: A Deviation Bound for a Sample of 25

Suppose $X_1,\ldots,X_{25}$ are iid, with each $X_i$ equal to 0 or 2 with probability $1/2$ each. Find a Chebyshev upper bound on the probability that the sample mean differs from its population mean by at least 0.40.

Check the assumptions and moments. The observations are specified as independent and identically distributed. For each observation, $E[X_i]=(0+2)/2=1$, and $\operatorname{Var}(X_i)=\{(0-1)^2+(2-1)^2\}/2=1$. Thus $\mu=1$, $\sigma^2=1$, and the variance is finite.

Use the bound with $n=25$ and $\varepsilon=0.40$:

P\left(|\bar{X}_{25}-1|\geq 0.40\right) \leq \frac{1}{25(0.40)^2} = \frac{1}{25(0.16)} = \frac{1}{4} = 0.25.

The probability of a deviation of at least 0.40 is therefore no greater than 0.25 according to this bound. The calculation does not show that the probability equals 0.25; the actual probability may be smaller.

Worked Example: Bounding Error in a Mean Measurement

Suppose hypothetical iid measurements $X_i$ have population mean 50 minutes and variance 100 minutes squared. For a sample of $n=100$ measurements, bound the probability that the sample mean is at least 2 minutes from the population mean.

Check the assumptions. The stipulated observations are independent, identically distributed, and have finite variance $100\text{ minutes}^2$. The common mean is $\mu=50$ minutes, so the theorem applies. Use $\varepsilon=2$ minutes:

P\left(|\bar{X}_{100}-50|\geq 2\text{ minutes}\right) \leq \frac{100\text{ minutes}^2} {100(2\text{ minutes})^2} = \frac{100}{400} = 0.25.

The units cancel because both the variance and squared tolerance are measured in minutes squared. The result guarantees an upper bound of 0.25 on the probability of a sample-mean error of 2 minutes or more. It is a bound for the sample mean, not for the deviation of an individual measurement.

Worked Example: Choosing a Sample Size for a Target Bound

Suppose iid observations have known population variance $9$ units squared. How large must the sample be for the Chebyshev upper bound on $P(|\bar{X}_n-\mu|\geq 0.5\text{ units})$ to be at most 0.05?

Check the assumptions. Assume the observations are independent and identically distributed with a common finite mean and variance $\sigma^2=9\text{ units}^2$. The target tolerance is $\varepsilon=0.5$ units. Requiring the bound to be no more than 0.05 gives:

\frac{9}{n(0.5)^2}\leq 0.05 \quad\Longleftrightarrow\quad \frac{9}{0.25n}\leq 0.05 \quad\Longleftrightarrow\quad n\geq\frac{9}{0.05(0.25)} =720.

Thus $n=720$ is the smallest integer sample size meeting this Chebyshev guarantee. At that size, the bound is $9/[720(0.25)]=9/180=0.05$. At $n=719$, it is $9/[719(0.25)]=9/179.75\approx0.05007$, which exceeds 0.05. The guarantee concerns the bound, not necessarily the actual probability.

What the Weak Law Does—and Does Not—Say

The weak law is a statement about how the distribution of a sample mean behaves as sample size grows. For every fixed positive tolerance, the chance of being outside that tolerance becomes small. It does not assert that every larger sample produces a realized mean closer to $\mu$ than every smaller sample did. Nor does it say that a particular finite-sample error probability is exactly $\sigma^2/(n\varepsilon^2)$.

The bound also makes the roles of the inputs transparent. A larger variance raises the bound; a larger sample size lowers it in proportion to $1/n$; and a larger tolerance lowers it in proportion to $1/\varepsilon^2$. These are properties of the guarantee from Chebyshev’s inequality, not necessarily exact descriptions of how the actual tail probability changes in every possible distribution.

Identify the target and assumptions.
Specify the common mean $\mu$, common finite variance $\sigma^2$, and verify that the observations are iid.

Choose the tolerance.
Write the event of interest as $|\bar{X}_n-\mu|\geq\varepsilon$, keeping $\varepsilon$ in the units of the observations.

Apply Chebyshev’s inequality.
Substitute into $\sigma^2/(n\varepsilon^2)$ and calculate carefully, including squaring the tolerance.

Interpret the result as an upper bound.
State that the probability is no greater than the computed value; do not call the bound the exact probability.

Common Mistake: Treating a Bound as an Exact Probability

A frequent error is to calculate $\sigma^2/(n\varepsilon^2)$ and report, “The probability of this deviation is 0.25.” Chebyshev’s inequality only establishes $P(|\bar{X}_n-\mu|\geq\varepsilon)\leq0.25$; unless a separate calculation establishes equality, the actual probability is not known from this result. A second practical error is to omit the square on $\varepsilon$. For example, when $\varepsilon=0.40$, its square is $0.16$, and with $n=25$, $25(0.16)=4$. Keeping the parentheses visible helps prevent arithmetic errors.

Key takeaway: For iid observations with common finite variance, the sample mean has variance $\sigma^2/n$. Chebyshev’s inequality then bounds its probability of missing $\mu$ by at least $\varepsilon$ as $\sigma^2/(n\varepsilon^2)$, which tends to zero for each fixed positive $\varepsilon$.

Check Your Understanding

Answer each question using the assumptions and meanings of the bound carefully.

State the assumptions of the finite-variance weak law and write its convergence-in-probability definition.
If $\sigma^2=4$, $n=80$, and $\varepsilon=0.5$, what is the Chebyshev upper bound on $P(|\bar{X}_n-\mu|\geq0.5)$?
Why does the variance of the iid sample mean equal $\sigma^2/n$? Identify the role of independence.
If a Chebyshev calculation gives an upper bound of 0.10, what can and cannot be concluded about the actual probability?
With variance 16 and tolerance 1, what sample size makes the Chebyshev upper bound no greater than 0.02?