Quantitative Weak-Law Calculations for Sample Means

From a Weak Law to a Sample-Size Calculation

A weak law says that a sample mean gets close to its target in probability as the sample size increases. In applications, the next question is quantitative: how large must the sample be to make a specified error probability small enough? A variance calculation gives a direct, generally conservative answer. When the sampling distribution is known, as for Gaussian observations, we can also calculate the deviation probability more exactly.

Work on a probability space \((\Omega,\mathcal{F},\mathbb{P})\). Let \(X_1,X_2,\ldots\) be independent and identically distributed random variables with mean \(\mu=\mathbb{E}[X_1]\) and finite variance \(\sigma^2=\operatorname{Var}(X_1)\). The sample mean based on \(n\) observations is \(\overline{X}_n=n^{-1}\sum_{i=1}^nX_i\). We will ask for the probability that its absolute error exceeds a tolerance \(\varepsilon>0\), and compare that probability with a target \(\alpha\), where \(0<\alpha<1\).

Proof. Independence gives \(\operatorname{Var}(\overline{X}_n)=n^{-2}\sum_{i=1}^n\operatorname{Var}(X_i)=\sigma^2/n\). For any random variable \(Y\) with finite variance and any \(\varepsilon>0\), apply Markov's inequality to the nonnegative variable \((Y-\mathbb{E}Y)^2\): the event \(\{|Y-\mathbb{E}Y|>\varepsilon\}\) is contained in \(\{(Y-\mathbb{E}Y)^2\ge\varepsilon^2\}\), so its probability is at most \(\mathbb{E}[(Y-\mathbb{E}Y)^2]/\varepsilon^2=\operatorname{Var}(Y)/\varepsilon^2\). Taking \(Y=\overline{X}_n\) proves the bound. To make the bound at most \(\alpha\), it suffices and is necessary for this bound that \(n\ge\sigma^2/(\alpha\varepsilon^2)\). If the ratio is positive, the smallest positive integer satisfying this inequality is its ceiling. If \(\sigma^2=0\), the ratio is zero and every positive integer satisfies the inequality; the smallest permitted sample size is then \(1\). These cases give the stated maximum. \(\square\)

In particular, with \(\sigma^2\) finite and fixed, the bound tends to zero as \(n\) increases. This proves convergence in probability of \(\overline{X}_n\) to \(\mu\), the weak law in this setting. For a design calculation, however, the useful step is to insert the desired \(\varepsilon\) and \(\alpha\) into the formula. The resulting sample size guarantees the requested probability by this variance bound; it need not be the smallest sample size that would work according to the true probability.

Bernoulli Sample Means

If \(X_i\) is Bernoulli with success probability \(p\), then \(X_i\) equals \(1\) with probability \(p\) and \(0\) with probability \(1-p\). Its mean is \(p\), and its variance is \(p(1-p)\). The sample mean \(\overline{X}_n\) is the observed success proportion. The variance bound is therefore \(\mathbb{P}(|\overline{X}_n-p|>\varepsilon)\le p(1-p)/(n\varepsilon^2)\). This uses the actual value of \(p\); if it is unknown, a conservative bound can use \(p(1-p)\le1/4\), which follows from \((p-\tfrac12)^2\ge0\).

For Bernoulli data, the exact probability can also be found by counting successes. If \(K_n=\sum_{i=1}^nX_i\), independence gives \(\mathbb{P}(K_n=k)=\binom{n}{k}p^k(1-p)^{n-k}\) for \(k=0,\ldots,n\). Since \(\overline{X}_n=K_n/n\), the strict deviation event uses the integer values \(k\) satisfying \(|k/n-p|>\varepsilon\). The strict inequality matters: a value exactly on the tolerance boundary is not included.

Bounded Observations

Sometimes the variance is not known, but the observations are known to lie in a fixed interval. A variance bound then converts the range information into a sample-size guarantee. The following inequality is useful because it is valid for any distribution supported on the interval, not only a uniform distribution.

Proof. If \(a=b\), then \(X=a\) almost surely and its variance is zero, so the inequality holds. Suppose \(a<b\), and write \(m=\mathbb{E}[X]\), which lies in \([a,b]\). Since \((X-a)(b-X)\ge0\) almost surely, taking expectations and expanding gives \(\mathbb{E}[X^2]\le(a+b)m-ab\). Consequently, \(\operatorname{Var}(X)=\mathbb{E}[X^2]-m^2\le(a+b)m-ab-m^2=(m-a)(b-m)\). The two nonnegative factors on the right sum to \(b-a\). For any nonnegative \(u,v\) with \(u+v=b-a\), \((u-v)^2\ge0\) implies \(4uv\le(u+v)^2=(b-a)^2\). Apply this with \(u=m-a\) and \(v=b-m\) to obtain \((m-a)(b-m)\le(b-a)^2/4\). This proves the claim. \(\square\)

If independent, identically distributed observations lie in \([a,b]\), use \((b-a)^2/4\) in place of \(\sigma^2\) in the sample-size formula. If their actual variance is known, using it can give a smaller guarantee. The range bound applies to more general independent observations as well when they share a common mean and each has variance at most a known common bound: the variance of their average is then at most that bound divided by \(n\).

Gaussian Sample Means

For Gaussian observations, the sample mean has a known distribution, allowing an exact probability calculation in terms of the standard normal cumulative distribution function \(\Phi(x)=\mathbb{P}(Z\le x)\), where \(Z\) has the standard normal distribution. If the independent observations have distribution \(N(\mu,\sigma^2)\) with \(\sigma>0\), then \(\overline{X}_n\) has distribution \(N(\mu,\sigma^2/n)\). One way to verify this is to use the Gaussian characteristic function: for one observation it is \(\exp(i\mu t-\sigma^2t^2/2)\); independence makes the characteristic function of the sum the product of the \(n\) characteristic functions, \(\exp(in\mu t-n\sigma^2t^2/2)\). Rescaling the sum by \(1/n\) gives the characteristic function \(\exp(i\mu t-\sigma^2t^2/(2n))\), that of \(N(\mu,\sigma^2/n)\). The uniqueness of characteristic functions identifies the distribution.

It follows that the exact two-sided probability is \(2\Phi(-\varepsilon\sqrt{n}/\sigma)\). To meet a target \(\alpha\), one may compare this quantity directly with \(\alpha\); the variance bound instead gives the sufficient condition \(n\ge\sigma^2/(\alpha\varepsilon^2)\). The exact calculation can be substantially sharper, but it requires Gaussian sampling assumptions and evaluation of a normal tail.

What the Bound Does—and Does Not—Guarantee

The variance calculation depends on independence: it is what allows the variance of the sum to equal the sum of the variances. Identical distribution is a convenient way to ensure a common mean and variance, but the calculation also works for independent variables with a common mean and a suitable bound on their average variance. The bound does not promise that a particular observed sample mean will be within tolerance, nor does it generally identify the true minimum sample size. It gives a guaranteed upper bound on a probability and a sample size sufficient to make that upper bound no larger than the chosen target.

Common mistake. A frequent error is to treat the ceiling of the sample-size ratio as the answer even when the ratio is zero. For example, if \(X_i=\mu\) almost surely, then \(\sigma^2=0\), and \(\lceil\sigma^2/(\alpha\varepsilon^2)\rceil=0\). But the sample size is required to be a positive integer: no sample mean with \(n=0\) is defined. In this case every positive sample size has zero deviation probability, so the smallest allowed sample size is \(1\). The corrected general expression is \(\max\{1,\lceil\sigma^2/(\alpha\varepsilon^2)\rceil\}\).

Independence is not merely a technical detail. If \(Z\) is a single Bernoulli variable with success probability \(\tfrac12\), and every \(X_i\) is set equal to that same \(Z\), then each observation has variance \(\tfrac14\), but \(\overline{X}_n=Z\) for every \(n\). For \(\varepsilon=\tfrac14\), the deviation probability from the mean \(\tfrac12\) is \(1\) for all \(n\), not a quantity tending to zero. The independent-sample variance calculation cannot be applied to this dependent sequence.