Convergence Of Empirical Frequencies And Proportions

From Repeated Outcomes to Empirical Proportions

In repeated sampling, a sample proportion is random: its value depends on which observations happen to be drawn. The central result here is that, under independent and identically distributed sampling, the proportion of observations satisfying a specified condition converges in probability to the population probability of that condition. This is a direct application of the weak law of large numbers.

The result applies to binary outcomes such as participation or nonparticipation, and it also applies to the frequency of any one category in a finite-valued variable. Its practical meaning is that large samples make large discrepancies between an empirical frequency and its population probability increasingly unlikely. It does not say that a particular finite-sample proportion must equal the population probability.

Definition: Let $X_1,\ldots,X_n$ be observations with the same distribution, and let $A$ be an event that can be evaluated for each observation. The empirical frequency of $A$ is the fraction of observations for which $A$ occurs: $$ \hat p_A=\frac{1}{n}\sum_{i=1}^n \mathbf{1}\{X_i\in A\}. $$ Here, $\mathbf{1}\{X_i\in A\}$ equals 1 when $X_i\in A$ and 0 otherwise. The corresponding population probability is $p_A=P(X_i\in A)$.

When $A$ is the event that a binary outcome equals 1, $\hat p_A$ is the usual sample proportion of ones. In general, an empirical frequency is also a sample proportion: it is the proportion of observations that meet the stated condition.

Why the Weak Law Applies

Assume $X_1,\ldots,X_n$ are independent and identically distributed. For a fixed event $A$, the indicators $I_i=\mathbf{1}\{X_i\in A\}$ are also independent and identically distributed. Each indicator has expectation $E[I_i]=p_A$. It is bounded between 0 and 1, so it has finite mean; in fact, its variance is $p_A(1-p_A)$. The weak law therefore applies to their sample average.

Theorem: Under independent and identically distributed sampling, for any fixed event $A$, $$ \hat p_A=\frac{1}{n}\sum_{i=1}^n \mathbf{1}\{X_i\in A\} \xrightarrow{p} p_A. $$ Equivalently, for every $\varepsilon>0$, $P(|\hat p_A-p_A|>\varepsilon)\to 0$ as $n\to\infty$.

This conclusion is convergence in probability, as defined in the earlier material on convergence in probability. It concerns the probability of a deviation larger than a fixed positive tolerance. It does not assert that the sequence of sample proportions is equal to $p_A$, or that every successive sample proportion is closer than the last one.

A Finite-Sample Justification

For a single event, the indicator variance gives a useful justification and a finite-sample bound. Independence makes the variance of the average equal to the variance of one indicator divided by $n$:

E[\hat p_A]=p_A,\qquad \operatorname{Var}(\hat p_A)=\frac{p_A(1-p_A)}{n}.

Applying Chebyshev's inequality, which bounds a deviation probability by variance divided by squared deviation size, gives

P\left(|\hat p_A-p_A|>\varepsilon\right) \leq \frac{p_A(1-p_A)}{n\varepsilon^2}.

For fixed $\varepsilon>0$, the numerator is at most $1/4$, since $p_A(1-p_A)\leq 1/4$. Thus the upper bound tends to zero as $n$ increases. This verifies the convergence claim directly for empirical frequencies. The bound need not be sharp; it is a guarantee, not an exact calculation of the deviation probability.

Worked Example: A Bernoulli Participation Proportion

Suppose, hypothetically, that each sampled eligible person participates with probability $p=0.40$, independently of the others. Let $Y_i=1$ for participation and $Y_i=0$ otherwise. For $n=600$, the sample participation proportion is $\hat p=600^{-1}\sum_{i=1}^{600}Y_i$. Find an upper bound on the probability that it differs from 0.40 by more than 0.04.

Model and conditions. The $Y_i$ are independent and identically distributed Bernoulli random variables with $P(Y_i=1)=0.40$. Thus $E[Y_i]=0.40$ and $\operatorname{Var}(Y_i)=0.40(0.60)=0.24$, so the assumptions for the bound hold.

Calculation. The variance of the sample proportion is $0.24/600=0.0004$. Chebyshev's inequality gives

P(|\hat p-0.40|>0.04) \leq \frac{0.40(0.60)}{600(0.04)^2} =\frac{0.24}{600(0.0016)} =\frac{0.24}{0.96} =0.25.

Under the stated model, the probability that the sample proportion is more than 0.04 away from 0.40 is at most 0.25. This bound is not an assertion that the probability equals 0.25. As the sample size grows, the same bound for a fixed tolerance decreases in proportion to $1/n$, which illustrates convergence in probability.

Frequencies for Several Categories

Suppose each observation takes one of a finite number $K$ of mutually exclusive and exhaustive categories, with population probabilities $p_1,\ldots,p_K$. For category $j$, define its sample frequency by

\hat p_j=\frac{1}{n}\sum_{i=1}^n \mathbf{1}\{X_i=j\}.

For each fixed $j$, this is an empirical frequency for the event $A_j=\{j\}$. The one-event result therefore gives $\hat p_j\xrightarrow{p}p_j$. With a finite number of categories, all their frequencies converge simultaneously in probability as well. To see why, for any $\varepsilon>0$, the probability that at least one frequency deviates by more than $\varepsilon$ is no greater than the sum of the individual deviation probabilities:

P\left(\max_{1\leq j\leq K}|\hat p_j-p_j|>\varepsilon\right) \leq \sum_{j=1}^{K}P(|\hat p_j-p_j|>\varepsilon).

Each term on the right tends to zero by the one-event result, and there are only finitely many terms. Their sum therefore tends to zero. This argument uses the union bound; it does not require the category frequencies within a sample to be independent of one another. In fact, because the frequencies sum to 1, they are generally dependent.

Worked Example: Frequencies Across Four Categories

Suppose, hypothetically, that each observation belongs to exactly one of four categories, with probabilities $p_1=0.20$, $p_2=0.35$, $p_3=0.30$, and $p_4=0.15$. The observations are independent and identically distributed. In one hypothetical sample of $n=1000$, the category counts are 203, 346, 301, and 150. Calculate the observed frequencies and bound the probability that any category frequency differs from its population probability by more than 0.03.

Observed frequencies. Dividing each count by 1000 gives $\hat p_1=0.203$, $\hat p_2=0.346$, $\hat p_3=0.301$, and $\hat p_4=0.150$. The counts sum to 1000, and the frequencies sum to 1. These realized values are close to, but not all exactly equal to, the specified probabilities.

Assumptions and bound. Each category indicator is Bernoulli with probability $p_j$, and independent sampling makes each category's sample frequency an average of independent indicators. Apply Chebyshev's inequality to each frequency, then the union bound:

P\left(\max_{1\leq j\leq 4}|\hat p_j-p_j|>0.03\right) \leq \sum_{j=1}^{4}\frac{p_j(1-p_j)}{1000(0.03)^2}.

The sum of the four variance numerators is $0.20(0.80)+0.35(0.65)+0.30(0.70)+0.15(0.85)=0.725$. Hence

P\left(\max_{1\leq j\leq 4}|\hat p_j-p_j|>0.03\right) \leq \frac{0.725}{1000(0.0009)} =\frac{0.725}{0.9} \approx 0.806.

The bound guarantees only that this probability is no greater than approximately 0.806; it is quite loose and does not describe the observed sample as a random event. The convergence result is still informative asymptotically: with these four fixed category probabilities, the probability of any frequency being more than 0.03 away tends to zero as the sample size grows.

Common Mistake: Treating Convergence as Exactness

A common error is to reason, “The sample proportion converges to the population probability, so with a large sample the two must be equal.” That conclusion does not follow. Convergence in probability says that the chance of a discrepancy exceeding any fixed positive tolerance becomes small as $n$ increases. It allows finite-sample discrepancies and does not require the observed proportion to move steadily toward the population probability as observations are added.

For example, a hypothetical sample with 21 participants out of 50 has $\hat p=21/50=0.42$. If the population participation probability is 0.40, the sample proportion differs from it by 0.02. This is a possible realization under the Bernoulli model, not a contradiction of the weak law. The theorem describes the behavior of the sampling distribution across repeated samples as sample size increases, rather than guaranteeing an exact value for one sample.

Key takeaway: An empirical frequency is an average of indicator variables. Under independent and identically distributed sampling, the weak law implies that this average converges in probability to the corresponding population probability. For a finite set of categories, every category frequency converges, and the frequencies converge simultaneously.

Check Your Understanding

Answer each question using the definitions and results above.

For independent and identically distributed observations, define the empirical frequency of an event $A$ using indicator variables.
If $p_A=0.25$ and $n=200$, what are the expectation and variance of $\hat p_A$?
State in words what $\hat p_A\xrightarrow{p}p_A$ means for a fixed positive tolerance.
Why can the weak law apply to an event indicator even when little is known about the distribution of the original observation $X_i$?
For a finite number of categories, how can one justify that the chance of any category frequency exceeding a fixed tolerance tends to zero?
Explain why a sample frequency different from its population probability does not contradict convergence in probability.

Convergence Of Empirical Frequencies And Proportions

What You'll Learn