When Dependence Breaks a Law of Large Numbers

Why Marginal Similarity Is Not Enough

A law of large numbers connects an average of many observations to a fixed mean. But repeated observations provide less and less new information: if every observation is the same random quantity, taking a longer average does not make that quantity less random. The key issue is not merely whether observations have the same distribution or a finite mean. It is how their dependence affects the fluctuations of the average.

Write $\bar X_n=n^{-1}\sum_{i=1}^n X_i$ for the average of the first $n$ terms of a sequence. In this tutorial, the comparison target is the common marginal mean $\mu=E[X_i]$, when it exists and is the same for every $i$. Convergence in probability to $\mu$ requires $P(|\bar X_n-\mu|>\varepsilon)\to 0$ for every $\varepsilon>0$; almost-sure convergence requires $\bar X_n\to\mu$ with probability one.

Definition: A sequence $(X_i)$ has a common marginal distribution if every $X_i$ has the same distribution. This condition does not assert that the variables are independent. A law of large numbers for the sequence is a convergence statement about $\bar X_n$, often convergence in probability or almost surely to the common marginal mean.

The distinction matters because $\bar X_n$ is built from the joint behavior of $X_1,\ldots,X_n$, while a marginal distribution describes one variable at a time. Dependence can make many terms move together, preventing averaging from reducing their shared variation.

Where Dependence Enters the Variance

Suppose $X_1,\ldots,X_n$ have finite second moments and a common mean $\mu$. Expanding the square of the centered average gives the exact identity

\operatorname{Var}(\bar X_n) =\frac{1}{n^2}\left\{\sum_{i=1}^n\operatorname{Var}(X_i) +2\sum_{1\leq i<j\leq n}\operatorname{Cov}(X_i,X_j)\right\}.

The diagonal terms are the individual variances; every pair of observations contributes a covariance term. If the variables are independent, those terms are zero, and for a common variance $\sigma^2$ the result is $\operatorname{Var}(\bar X_n)=\sigma^2/n$. Chebyshev's inequality then gives $P(|\bar X_n-\mu|\geq\varepsilon)\leq \sigma^2/(n\varepsilon^2)$, which tends to zero.

For dependent observations, the same variance expansion remains valid, but the covariance terms cannot simply be dropped. There are $n(n-1)/2$ pairs, so even covariances that do not grow with $n$ can affect the order of the variance. Here is the derivation.

Result: If $X_1,\ldots,X_n$ have finite second moments and $E[X_i]=\mu$ for all $i$, then the displayed variance identity holds. If, in addition, each variance is $\sigma^2$ and every distinct pair has covariance $c$, then $$ \operatorname{Var}(\bar X_n)=\frac{\sigma^2}{n}+\frac{n-1}{n}c. $$

To obtain the second formula, there are $n$ diagonal terms equal to $\sigma^2$ and $n(n-1)/2$ distinct pairs, each counted twice in the variance expansion. Thus the numerator is $n\sigma^2+n(n-1)c$; dividing by $n^2$ gives the stated expression. When $c>0$, the variance does not tend to zero but instead tends to $c$. This identifies a possible obstruction, though a variance calculation by itself is not a universal test for convergence in probability.

Worked Example: Perfectly Repeated Observations

Let $Z$ be Bernoulli with $P(Z=1)=P(Z=0)=1/2$, and set $X_i=Z$ for every $i$. Each $X_i$ is bounded, has the same distribution, and has mean $\mu=1/2$. However, $$ \bar X_n=\frac{1}{n}\sum_{i=1}^n Z=Z $$ for every $n$. Therefore $P(|\bar X_n-\mu|>1/4)=1$: whether $Z=0$ or $Z=1$, its distance from $1/2$ is $1/2$. The averages do not converge in probability to $1/2$. They also do not converge almost surely to $1/2$; they equal either zero or one on every outcome.

The covariance calculation shows the same failure from another angle. Since $\operatorname{Var}(Z)=1/4$ and $\operatorname{Cov}(X_i,X_j)=1/4$ for every pair, the formula gives $\operatorname{Var}(\bar X_n)=1/(4n)+(n-1)/(4n)=1/4$. Averaging has not reduced the variance at all.

Dependence Does Not Automatically Prevent Convergence

Perfect repetition is a dramatic form of dependence, not a general model for all dependent sequences. Some dependent sequences still obey a law of large numbers because the dependence is limited in a way that allows the information in the average to accumulate.

Worked Example: Repeating Each Observation Twice

Let $Y_1,Y_2,\ldots$ be independent Bernoulli variables with success probability $p$, and define $X_{2j-1}=X_{2j}=Y_j$. Each $X_i$ is Bernoulli with mean $p$; adjacent pairs are perfectly dependent. For $n=2m$, $$ \bar X_{2m}=\frac{2\sum_{j=1}^mY_j}{2m}=\frac{1}{m}\sum_{j=1}^mY_j. $$ The strong law for the independent $Y_j$'s gives $\bar X_{2m}\to p$ almost surely.

For an odd sample size, $$ \bar X_{2m+1}=\frac{2\sum_{j=1}^mY_j+Y_{m+1}}{2m+1}. $$ Subtract $p$ and rearrange: $$ \bar X_{2m+1}-p =\frac{2m}{2m+1}\left(\frac{1}{m}\sum_{j=1}^mY_j-p\right) +\frac{Y_{m+1}-p}{2m+1}. $$ The first term tends to zero almost surely by the same strong law. Since $Y_{m+1}$ and $p$ lie in $[0,1]$, the absolute value of the second term is at most $1/(2m+1)$, which tends to zero. The even and odd subsequences therefore both converge to $p$ almost surely, so the full sequence of averages does too.

This example is dependent at every duplicated pair, yet each underlying $Y_j$ contributes a new independent value as $j$ increases. The duplication changes how much information a given sample size contains; it does not create one shared random value that persists through the entire sequence.

A Shared Random Environment Can Set the Limiting Average

A subtler failure occurs when observations are independent only after conditioning on a shared random variable. In that case, the average can settle near a conditional mean that varies between realizations of the shared environment. The marginal mean averages over environments, but an individual long sequence experiences only one of them.

Worked Example: Conditional Bernoulli Observations

Let $H$ be Bernoulli with success probability $1/2$, and let $U_1,U_2,\ldots$ be independent uniform random variables on $[0,1]$, independent of $H$. Define $$ X_i=\mathbf{1}\{U_i\leq p_H\},\qquad p_0=\frac14,\quad p_1=\frac34. $$ Each $X_i$ is Bernoulli. Its marginal success probability is $$ P(X_i=1)=P(H=0)\frac14+P(H=1)\frac34 =\frac12\cdot\frac14+\frac12\cdot\frac34=\frac12, $$ so the common marginal mean is $1/2$.

Conditional on $H=0$, the $X_i$'s are independent Bernoulli variables with success probability $1/4$; conditional on $H=1$, they are independent Bernoulli variables with success probability $3/4$. The strong law applied under each conditional distribution gives $\bar X_n\to 1/4$ almost surely on $H=0$, and $\bar X_n\to 3/4$ almost surely on $H=1$. Both events have probability $1/2$, so altogether $\bar X_n\to p_H$ almost surely.

The limiting average is not the marginal mean: its distance from $1/2$ is $1/4$ in either environment. In particular, almost surely $|\bar X_n-1/2|>1/8$ for all sufficiently large $n$. Consequently $P(|\bar X_n-1/2|>1/8)\to1$, rather than zero. The observations share dependence through $H$, even though the $U_i$'s are independent.

Which Independence-Based Steps Fail?

The usual variance proof of the weak law does not fail because its algebra changes: the variance expansion is still correct. The failed step is replacing the variance of a sum by the sum of its variances without justification. That simplification uses zero covariances, which independence guarantees (when the relevant moments exist), but dependence need not provide.

In the perfectly repeated example, every covariance equals the full individual variance. The pair terms exactly prevent the variance from shrinking. In the shared-environment example, the same common source of variation affects many observations. By contrast, the duplicated-pair example has dependence within pairs but still has an increasing number of independent underlying values. These examples show why a label such as “dependent” does not, by itself, determine the behavior of the averages.

Common pitfall: “The $X_i$'s have the same bounded distribution and finite mean, so their average converges to that mean.” Identical marginal distributions and boundedness do not supply the needed control over dependence. In the perfectly repeated example, all those conditions hold, but $\bar X_n=Z$ for every $n$. To justify a law of large numbers, check the actual assumptions that control joint behavior; do not infer independence from identical distributions.

This distinction is useful whenever observations may share a subject, cluster, time period, or latent environment. Before applying an independence-based argument, identify which terms are independent and which covariance contributions remain. A dependence structure may preserve convergence, change the limit, or prevent convergence to the marginal mean. The conclusion must come from the structure of the sequence, not from its one-variable distributions alone.

Check Your Understanding

For each question, focus on the joint behavior of the observations as well as their marginal distributions.

If $X_i=Z$ for every $i$, where $Z$ has finite variance, what is $\bar X_n$, and why does its variance fail to shrink?
In the variance formula for $\bar X_n$, which terms are discarded in the usual independent-sample calculation?
In the duplicated-pair example, why does the sequence of averages still converge despite dependence within each pair?
In the shared-environment example, what is the almost-sure limit of $\bar X_n$, and how does it differ from the marginal mean?
Why does having identical bounded marginal distributions not, by itself, establish a law of large numbers?