Two Ways a Sample Average Can Settle Down
A law of large numbers describes the long-run behavior of averages of random observations. The word “settle” has more than one precise meaning: an average may approach a constant in probability, or it may approach that constant for almost every realized sequence of observations. These are different guarantees, and the distinction matters when interpreting what a law actually says.
Let \(X_1,X_2,\ldots\) be random variables on one probability space, and define their partial sums and sample averages by
The weak and strong laws are not definitions of a particular average; they are convergence claims about a sequence such as \(\overline{X}_n\). Each claim needs a target and assumptions on the observations. In the classical iid setting, the target is the common mean, when that mean exists.
The strong law is stronger: it controls the full sequence of averages along almost every outcome, not just the probability of a large error at each individual sample size. In particular, strong convergence implies weak convergence. The reverse implication does not hold in general.
Centering, Normalization, and the Target
A sample average is normalized by \(n\): it is a sum divided by the number of observations. If the observations have finite expectations, write
Subtracting \(m_n\) centers the average around its own expectation. This distinguishes random fluctuation from any systematic movement in the expected values. For identically distributed observations with finite mean \(\mu=E[X_1]\), \(m_n=\mu\), so the centered average is \(\overline{X}_n-\mu\). For nonidentically distributed observations, the expected average need not be constant.
To verify the probability version, fix \(\varepsilon>0\). Since \(m_n\to\mu\), for all sufficiently large \(n\), \(|m_n-\mu|<\varepsilon/2\). The identity \(\overline{X}_n-\mu=(\overline{X}_n-m_n)+(m_n-\mu)\) and the triangle inequality then give
For the almost-sure version, on every outcome where \(\overline{X}_n-m_n\to0\), adding the deterministic sequence \(m_n-\mu\to0\) gives \(\overline{X}_n-\mu\to0\). This decomposition shows why both centering and normalization should be stated explicitly: the centered random part and the average of the expectations play separate roles.
Classical iid Formulations
A standard pair of laws gives a clean comparison. If \(X_1,X_2,\ldots\) are independent and identically distributed with \(E[|X_1|]<\infty\), then their common mean \(\mu=E[X_1]\) is finite. The weak law states that \(\overline{X}_n\xrightarrow{p}\mu\), and the strong law states that \(\overline{X}_n\xrightarrow{a.s.}\mu\). These are statements about the same normalized sums and target, but the convergence modes differ.
A finite second moment, \(E[X_1^2]<\infty\), is a familiar stronger sufficient condition for both conclusions. It is not part of the definition of either law, and it is not necessary for the iid finite-mean versions just stated. Outside the iid setting, the hypotheses must be matched to the structure of the observations: identical distributions, independence, and moment conditions cannot be silently assumed.
Worked Example: Bernoulli Sample Proportions
Let \(X_1,X_2,\ldots\) be iid Bernoulli random variables with \(P(X_i=1)=p\) and \(P(X_i=0)=1-p\), where \(0\leq p\leq1\). The average \(\overline{X}_n\) is the proportion of successes in the first \(n\) trials.
The mean is \(E[X_i]=p\), because \(1\cdot p+0\cdot(1-p)=p\). Also, \(X_i^2=X_i\), so
Thus the observations are iid with finite absolute mean, and the classical iid laws apply: \(\overline{X}_n\xrightarrow{p}p\) and \(\overline{X}_n\xrightarrow{a.s.}p\). The variance calculation also confirms the finite-second-moment condition, since \(E[X_i^2]=p<\infty\). The two conclusions concern the same success proportion; the strong statement additionally asserts convergence along almost every infinite sequence of trials.
Why Strong Convergence Implies Weak Convergence
This implication does not require independence, identical distributions, or finite moments. It follows directly from the definitions, provided both modes refer to random variables on the same probability space.
Proof. Fix \(\varepsilon>0\), and for each positive integer \(N\) define
The events \(A_N\) decrease as \(N\) increases. Their intersection is the event that \(|Y_n-Y|>\varepsilon\) occurs infinitely often. Almost-sure convergence makes this event have probability zero. By continuity of probability for decreasing events, \(P(A_N)\to0\). For each \(n\geq N\), \(\{|Y_n-Y|>\varepsilon\}\subseteq A_N\), and therefore \(P(|Y_n-Y|>\varepsilon)\leq P(A_N)\). Taking \(n\to\infty\) proves convergence in probability. \(\square\)
Worked Example: Alternating Expectations with a Convergent Average
Let \(X_i\) be deterministic, with \(X_i=0\) for odd \(i\) and \(X_i=2\) for even \(i\). The observations are not identically distributed, but every sample average can be calculated exactly.
For \(n=2k\), there are \(k\) zeros and \(k\) twos, so \(S_{2k}=2k\) and \(\overline{X}_{2k}=1\). For \(n=2k+1\), the additional observation is zero, so \(S_{2k+1}=2k\) and
The even and odd subsequences both converge to 1, so the full sequence does too. Because the sequence is deterministic, this convergence is both in probability and almost surely. Meanwhile, \(E[X_i]\) alternates between 0 and 2; the target 1 is the limit of the averages of those expectations, not the expectation of every observation.
Why Weak Convergence Need Not Be Strong
Convergence in probability says that the error at a specified large index is unlikely to exceed a fixed tolerance. It does not, by itself, rule out infinitely many errors along a realized sequence. The following construction makes that distinction explicit for sample averages. It is deliberately not an iid example: it shows why the assumptions of a standard law cannot be dropped.
Worked Example: Sample Averages Converging Weakly but Not Strongly
Take the probability space \([0,1)\) with uniform probability. At level \(j=1,2,\ldots\), partition \([0,1)\) into the \(2^j\) half-open intervals
List all these intervals in order of increasing level, and in any fixed order within each level. For the \(n\)th listed interval \(I_n\), define \(Y_n(\omega)=1\) if \(\omega\in I_n\), and \(Y_n(\omega)=0\) otherwise. If \(I_n\) is at level \(j\), then \(P(Y_n=1)=2^{-j}\). The level tends to infinity as \(n\to\infty\), so \(P(|Y_n|>1/2)\to0\). Hence \(Y_n\xrightarrow{p}0\).
However, every \(\omega\in[0,1)\) lies in exactly one interval at every level. Thus \(Y_n(\omega)=1\) infinitely often. It is also zero infinitely often, since at each level \(2^j-1\) of the intervals exclude that \(\omega\). Consequently \(Y_n(\omega)\) does not converge to zero—or to any other value—for any \(\omega\).
Now define observations by \(X_1=Y_1\) and, for \(n\geq2\), \(X_n=nY_n-(n-1)Y_{n-1}\). Their partial sums telescope:
Therefore these sample averages converge in probability to zero but fail to converge almost surely. The construction uses dependent, nonidentically distributed observations; it is not a counterexample to the iid strong law.
Worked Example: A Shared Random Effect Prevents a Constant Target
Let \(Z\) be Bernoulli with \(P(Z=1)=P(Z=0)=1/2\), and set \(X_i=Z\) for every \(i\). Every observation has mean \(1/2\), but the observations are completely dependent: once \(Z\) is realized, all observations have that same value.
For every \(n\), \(S_n=nZ\) and \(\overline{X}_n=Z\). The average does not approach its mean \(1/2\), since
for every \(n\). In fact, the averages converge almost surely to the random variable \(Z\), not to a deterministic constant. This example shows why a shared expectation alone does not establish a law of large numbers: dependence can prevent averaging from reducing variability.
Common Mistakes and Practical Interpretation
- Treating “weak” as a weaker numerical target. The weak and strong laws can have the same target and normalization. “Weak” refers to convergence in probability, not to a different mean.
- Assuming weak convergence automatically gives almost-sure convergence. It does not. The typewriter construction gives averages converging in probability while failing to converge along every realized sequence.
- Forgetting to center nonidentically distributed observations. The natural centered quantity is \(\overline{X}_n-m_n\), where \(m_n=n^{-1}\sum_{i=1}^n E[X_i]\). A common mean for every \(X_i\) is a special case.
- Quoting a law without its assumptions. A conclusion for iid observations does not automatically apply to observations sharing a common random effect, or to an arbitrary dependent sequence.
When applying a law of large numbers, identify the normalized sum, its target, and whether the claim is in probability or almost surely. Then check the assumptions that support that particular conclusion. This keeps a statement about averages from being mistaken for a guarantee about every realized sequence.
Check Your Understanding
Try these questions without looking back.
- State the difference between convergence in probability and almost-sure convergence for a sample average.
- If the expectations \(E[X_i]\) vary with \(i\), what quantity centers \(\overline{X}_n\)?
- Why does almost-sure convergence imply convergence in probability, and which assumptions on the \(X_i\) are needed for this implication?
- In the alternating deterministic example, why is the limiting average 1 even though the individual expectations do not converge?
- What feature of the shared-random-effect example prevents its averages from approaching the constant \(E[X_i]\)?