From Observations to Proportions
Many summaries of a sample ask what fraction of observations meet a condition: the fraction of patients who respond to treatment, the fraction of measurements below a threshold, or the fraction of draws falling in a specified category. Each such fraction is an average of zero-one variables. That simple representation makes the laws of large numbers directly applicable.
Let \(X_1,X_2,\ldots\) be independent and identically distributed (iid) observations with distribution \(F\), and let \(A\) be a measurable set of possible observations. Define \(I_i=\mathbf{1}\{X_i\in A\}\), where \(\mathbf{1}\) equals one when its condition holds and zero otherwise. The sample proportion in \(A\) among the first \(n\) observations is \(\hat p_n(A)=n^{-1}\sum_{i=1}^n I_i\). The indicators are iid Bernoulli variables with success probability \(p=P(X_1\in A)\).
The expectation of each indicator is \(p\), and its variance is \(p(1-p)\). Since the indicators are independent across observations, the sample proportion has expectation \(p\) and variance \(p(1-p)/n\). These identities give a direct weak law and help quantify finite-sample uncertainty.
Weak and Strong Laws for a Sample Proportion
The weak law says that the proportion is close to its target with probability tending to one. The strong law gives the more powerful pathwise conclusion that, with probability one, the observed proportions eventually converge to the target.
Proof of weak convergence. The variables \(I_i=\mathbf{1}\{X_i\in A\}\) are iid Bernoulli with mean \(p\) and variance \(p(1-p)\). Therefore $$ E[\hat p_n(A)]=p,\qquad \operatorname{Var}(\hat p_n(A))=\frac{p(1-p)}{n}. $$ For each \(\varepsilon>0\), Chebyshev’s inequality gives $$ P\left(\left|\hat p_n(A)-p\right|\geq\varepsilon\right) \leq \frac{p(1-p)}{n\varepsilon^2}\longrightarrow 0. $$ This is convergence in probability.
Proof of almost-sure convergence. Write \(Y_i=I_i-p\) and \(S_n=\sum_{i=1}^nY_i\). The \(Y_i\) are independent, have mean zero, and satisfy \(|Y_i|\leq1\). In the expansion of \(E[S_n^4]\), any term containing an index exactly once has expectation zero, by independence and \(E[Y_i]=0\). The remaining terms are the single-index fourth powers and the paired terms, so $$ E[S_n^4]=\sum_{i=1}^n E[Y_i^4]+6\sum_{1\leq i<j\leq n}E[Y_i^2]E[Y_j^2] \leq n+3n(n-1)\leq n+3n^2. $$ Here \(|Y_i|\leq1\) implies \(E[Y_i^4]\leq1\) and \(E[Y_i^2]\leq1\). Markov’s inequality applied to \(S_n^4\) now yields, for every \(\varepsilon>0\), $$ P\left(\left|\hat p_n(A)-p\right|>\varepsilon\right) =P(|S_n|>n\varepsilon) \leq \frac{n+3n^2}{n^4\varepsilon^4}. $$ The right-hand side is summable over \(n\), since it is a constant multiple of \(n^{-3}+3n^{-2}\). By the first Borel–Cantelli lemma, the event \(\{|\hat p_n(A)-p|>\varepsilon\}\) occurs only finitely often with probability one. Apply this conclusion to \(\varepsilon=1/m\) for each positive integer \(m\). The intersection of these countably many probability-one events still has probability one, and on it \(\hat p_n(A)\to p\). \(\square\)
Worked Example: Estimating a Response Probability
Suppose each patient independently responds to a treatment with probability \(p=0.72\). Let \(\hat p_{400}\) be the response proportion among \(n=400\) patients. Find its mean and variance, and bound the probability that it differs from \(0.72\) by at least \(0.05\).
Each response indicator is Bernoulli with mean \(0.72\) and variance \(0.72(1-0.72)=0.2016\). Consequently, $$ E[\hat p_{400}]=0.72,\qquad \operatorname{Var}(\hat p_{400})=\frac{0.2016}{400}=0.000504. $$ By Chebyshev’s inequality, $$ P(|\hat p_{400}-0.72|\geq0.05) \leq\frac{0.000504}{(0.05)^2} =\frac{0.000504}{0.0025}=0.2016. $$ Thus the bound is \(0.2016\). It is an upper bound, not an exact probability. As the number of independent patients grows, the weak and strong laws both guarantee convergence of the response proportion to \(0.72\).
Frequency Vectors for Finite Categories
Now suppose each observation belongs to exactly one of \(K\) categories, where \(K\) is a fixed finite positive integer. Write \(p_j=P(X_1\text{ is in category }j)\), for \(j=1,\ldots,K\), so that \(\sum_{j=1}^Kp_j=1\). The count in category \(j\) is \(N_{n,j}\), and its frequency is \(\hat p_{n,j}=N_{n,j}/n\). Collecting the frequencies gives the vector \(\hat{\boldsymbol p}_n=(\hat p_{n,1},\ldots,\hat p_{n,K})\).
For each fixed category \(j\), its indicator across observations is an iid Bernoulli sequence with success probability \(p_j\). The one-category theorem therefore applies to every coordinate. Since there are only finitely many coordinates, their convergence can be combined into convergence of the whole vector.
Proof of vector convergence and covariance formulas. For each \(j\), define \(I_{i,j}\) to be one if observation \(i\) is in category \(j\), and zero otherwise. Then \(\hat p_{n,j}=n^{-1}\sum_{i=1}^n I_{i,j}\). For any \(\varepsilon>0\), the union bound and the coordinate weak law give $$ P\left(\max_{1\leq j\leq K}|\hat p_{n,j}-p_j|>\varepsilon\right) \leq \sum_{j=1}^K P(|\hat p_{n,j}-p_j|>\varepsilon)\longrightarrow0. $$ For almost-sure convergence, each coordinate converges almost surely by the Bernoulli strong law. The intersection of the \(K\) coordinatewise probability-one events has probability one, since \(K\) is finite. On that intersection every coordinate converges, and hence the vector converges in any norm on \(\mathbb R^K\).
For a single observation, \(I_{i,j}\) has variance \(p_j(1-p_j)\). If \(j\ne k\), the categories cannot both occur, so \(I_{i,j}I_{i,k}=0\) and $$ \operatorname{Cov}(I_{i,j},I_{i,k}) =E[I_{i,j}I_{i,k}]-E[I_{i,j}]E[I_{i,k}] =0-p_jp_k=-p_jp_k. $$ Indicators from different observations are independent. Summing the within-observation variances or covariances over \(n\) observations and dividing by \(n^2\) gives the stated formulas. \(\square\)
The negative covariance is not a contradiction: increasing one category’s frequency leaves less of the fixed total frequency available for the others. Indeed, \(\sum_{j=1}^K\hat p_{n,j}=1\) exactly for every sample, just as \(\sum_{j=1}^Kp_j=1\). The count vector has the multinomial distribution with sample size \(n\) and category probabilities \(\boldsymbol p\).
Worked Example: Three Multinomial Categories
A sample of \(n=100\) independent observations falls into categories A, B, and C with probabilities \(0.50\), \(0.30\), and \(0.20\). Find the expected counts, the variances of the counts, and the covariance between the counts for A and B.
The expected count in category \(j\) is \(np_j\). Thus the expected counts are \(100(0.50)=50\), \(100(0.30)=30\), and \(100(0.20)=20\). For each category the count variance is \(np_j(1-p_j)\), giving $$ \operatorname{Var}(N_{100,A})=100(0.50)(0.50)=25, $$ $$ \operatorname{Var}(N_{100,B})=100(0.30)(0.70)=21,\qquad \operatorname{Var}(N_{100,C})=100(0.20)(0.80)=16. $$ For distinct categories, the count covariance is \(-np_jp_k\). Therefore $$ \operatorname{Cov}(N_{100,A},N_{100,B}) =-100(0.50)(0.30)=-15. $$ Dividing counts by \(100\) divides their variances and covariances by \(100^2\). For example, \(\operatorname{Cov}(\hat p_{100,A},\hat p_{100,B})=-15/10000=-0.0015\). The category frequencies always sum to one, not merely on average.
Worked Example: A Proportion Defined by a Measurement Threshold
Let \(X_1,\ldots,X_n\) be iid measurements from an exponential distribution with rate \(2\), so \(P(X_i>t)=e^{-2t}\) for \(t\geq0\). Consider the proportion of measurements exceeding \(1\). Find its limiting value and, for \(n=500\), give a Chebyshev bound for an error of at least \(0.05\).
Here \(A=(1,\infty)\), so the event probability is $$ p=P(X_i>1)=e^{-2}\approx0.135335. $$ The sample proportion is an average of Bernoulli indicators, so it converges in probability and almost surely to \(e^{-2}\). Its variance for \(n=500\) is $$ \frac{p(1-p)}{500} \approx\frac{(0.135335)(0.864665)}{500} \approx\frac{0.117019}{500} \approx0.00023404. $$ Chebyshev’s inequality gives $$ P(|\hat p_{500}(A)-e^{-2}|\geq0.05) \leq\frac{0.00023404}{0.0025}\approx0.09362. $$ The event proportion targets the probability of exceeding the threshold; it does not target the mean of the measurement \(X_i\).
Common Pitfall: Categories Within One Observation Are Not Independent
It is tempting to treat the category indicators for a single observation as independent because each one is an indicator variable. They are not independent when categories are mutually exclusive: if an observation is in category A, its indicator for category B must equal zero. Their covariance is \(-p_jp_k\) for distinct categories. The independence used in the laws of large numbers is instead the independence of observations across sample indices. For each fixed category, the indicators across \(i\) are independent, which is exactly what the Bernoulli argument requires.
A second useful distinction is between a finite partition and an unlimited collection of categories. The vector argument above uses a fixed finite \(K\), including the finite-intersection step for almost-sure convergence. It does not claim uniform convergence over an infinite collection of categories. For a finite partition, however, each frequency converges to its category probability, and the entire frequency vector converges coordinate by coordinate.
Check Your Understanding
Answer each question using the indicator representation where appropriate.
- For iid observations \(X_i\) and a measurable set \(A\), define the indicator whose average is the sample proportion in \(A\), and state its mean and variance.
- Why does Chebyshev’s inequality imply that the sample proportion converges in probability to \(P(X_1\in A)\)?
- For a finite partition into \(K\) categories, why can coordinatewise almost-sure convergence be combined to obtain almost-sure convergence of the frequency vector?
- If two distinct categories have probabilities \(p_j\) and \(p_k\), what is the covariance between their sample frequencies, and why is it negative?
- A sample records the fraction of measurements above a fixed threshold. Which probability does that fraction converge to, and what assumptions on the observations are used here?