Worked Theorem-Selection Problems for IID and Independent Samples

A Practical Choice Among Laws of Large Numbers

In a law-of-large-numbers problem, the first question is not simply whether an average converges. It is which theorem is justified by the assumptions, and what kind of convergence does it provide? A finite-variance calculation may give convergence in probability quickly. An IID sequence with a finite first absolute moment has a strong-law conclusion even when its variance is infinite. For independent variables that are not identically distributed, a summability condition on their variances can provide another strong-law route.

These methods have different assumptions and conclusions. The variance condition for a weak law asks whether the variance of the average tends to zero. A common independent-sequence strong-law condition instead asks whether a weighted series of variances is summable. The finite-mean IID strong law uses neither requirement: its essential moment assumption is only integrability.

Three Theorem Choices

Variance criterion for a weak law: Suppose \(X_1,X_2,\ldots\) are independent, have finite means and finite variances, and let \(m_n=n^{-1}\sum_{i=1}^n E[X_i]\). If \(\sum_{i=1}^n\operatorname{Var}(X_i)/n^2\to 0\), then \(n^{-1}\sum_{i=1}^n X_i-m_n\xrightarrow{p}0\). In particular, if \(m_n\to m\), then \(n^{-1}\sum_{i=1}^n X_i\xrightarrow{p}m\).

Finite-mean IID strong law: If \(X_1,X_2,\ldots\) are IID and \(E[|X_1|]<\infty\), then \(n^{-1}\sum_{i=1}^nX_i\xrightarrow{a.s.}E[X_1]\). A finite variance is not required.

Kolmogorov strong-law criterion: Suppose \(X_1,X_2,\ldots\) are independent, \(E[|X_i|]<\infty\), and \(\operatorname{Var}(X_i)<\infty\) for each \(i\). If \(\sum_{i=1}^{\infty}\operatorname{Var}(X_i)/i^2<\infty\), then \(n^{-1}\sum_{i=1}^n(X_i-E[X_i])\xrightarrow{a.s.}0\). If in addition \(n^{-1}\sum_{i=1}^nE[X_i]\to m\), then \(n^{-1}\sum_{i=1}^nX_i\xrightarrow{a.s.}m\).

The last criterion does not require identical distributions, but it does require independence and finite variances. Its variance condition is stronger than the weak-law condition: one asks for an infinite sum to be finite, while the other asks for a sequence of ratios to tend to zero.

Deriving the Weak-Law Variance Criterion

Let \(\bar X_n=n^{-1}\sum_{i=1}^nX_i\). Independence and finite variances give \(\operatorname{Var}(\bar X_n)=n^{-2}\sum_{i=1}^n\operatorname{Var}(X_i)\). Chebyshev’s inequality then bounds the probability that \(\bar X_n\) differs from its expectation by more than any fixed positive amount.

P\left(\left|\bar X_n-m_n\right|>\varepsilon\right) \leq \frac{\operatorname{Var}(\bar X_n)}{\varepsilon^2} =\frac{1}{\varepsilon^2 n^2}\sum_{i=1}^n\operatorname{Var}(X_i) \longrightarrow 0.

This proves \(\bar X_n-m_n\xrightarrow{p}0\). If \(m_n\to m\), then for any \(\varepsilon>0\), eventually \(|m_n-m|\leq\varepsilon/2\), so \(P(|\bar X_n-m|>\varepsilon)\leq P(|\bar X_n-m_n|>\varepsilon/2)\to0\). Thus the average converges in probability to \(m\). The deterministic convergence of the average means is part of the conclusion: without it, the variance calculation alone does not identify a fixed limit. \(\square\)

Worked Example: A Weak Law for Bernoulli Observations

Let \(X_1,\ldots,X_n\) be IID Bernoulli variables with success probability \(p=0.3\). Use the variance criterion to bound the probability that the sample proportion differs from \(0.3\) by more than \(0.1\). Then find a sufficient sample size for this bound to be at most \(0.05\).

Each \(X_i\) has \(E[X_i]=0.3\) and \(\operatorname{Var}(X_i)=0.3(0.7)=0.21\). Thus \(E[\bar X_n]=0.3\) and \(\operatorname{Var}(\bar X_n)=0.21/n\). Chebyshev’s inequality gives

P(|\bar X_n-0.3|>0.1) \leq \frac{0.21/n}{(0.1)^2} =\frac{21}{n}.

For the bound to be at most \(0.05\), require \(21/n\leq0.05\), or \(n\geq420\). This is a sufficient sample size for the stated bound, not a claim that 420 is the smallest sample size for which the actual probability is at most \(0.05\). The variance criterion also shows directly that \(\bar X_n\xrightarrow{p}0.3\).

Deriving a Strong Law for Independent Sequences

Here is a useful strong-law argument when the variables are independent but need not have the same distribution. Write \(Y_i=X_i-E[X_i]\), and assume \(\sum_i\operatorname{Var}(X_i)/i^2<\infty\). The variables \(Y_i/i\) are independent, have mean zero, and their variances have finite sum. Kolmogorov’s convergence criterion for independent, mean-zero variables therefore gives convergence almost surely of \(\sum_iY_i/i\).

To pass from this convergent series to averages, set \(a_i=Y_i/i\) and \(s_j=\sum_{i=1}^j a_i\). Summation by parts gives

\frac{1}{n}\sum_{i=1}^nY_i =s_n-\frac{1}{n}\sum_{j=1}^{n-1}s_j.

On every outcome for which \(s_j\) converges, its arithmetic averages \(n^{-1}\sum_{j=1}^{n-1}s_j\) converge to the same limit as \(s_n\). Their difference therefore tends to zero. Hence \(n^{-1}\sum_{i=1}^nY_i\xrightarrow{a.s.}0\). If the averages of the expectations tend to \(m\), adding those deterministic averages proves the stated conclusion for \(X_i\). \(\square\)

This criterion is a sufficient condition, not a test that must hold whenever a strong law is true. In particular, failure of its variance summability condition does not by itself show that the averages fail to converge.

Worked Example: A Strong Law With Changing Means and Variances

Let \(\varepsilon_i\) be independent Rademacher variables, with \(P(\varepsilon_i=1)=P(\varepsilon_i=-1)=1/2\), and define \(X_i=1+i^{-1/2}+i^{-1/2}\varepsilon_i\). Find the almost-sure limit of the averages using the independent-sequence criterion.

The mean and variance are \(E[X_i]=1+i^{-1/2}\) and \(\operatorname{Var}(X_i)=i^{-1}\). The variance series in the criterion is finite because

\sum_{i=1}^{\infty}\frac{\operatorname{Var}(X_i)}{i^2} =\sum_{i=1}^{\infty}\frac{1}{i^3}<\infty.

The averages of the means satisfy

\frac{1}{n}\sum_{i=1}^nE[X_i] =1+\frac{1}{n}\sum_{i=1}^ni^{-1/2} \longrightarrow 1,

since \(\sum_{i=1}^ni^{-1/2}\leq 1+\int_1^n x^{-1/2}\,dx=2\sqrt n-1\), and \((2\sqrt n-1)/n\to0\). The criterion therefore yields \(n^{-1}\sum_{i=1}^nX_i\xrightarrow{a.s.}1\). The variables are independent but not identically distributed; their changing means have been handled separately from their centered fluctuations.

When the First Moment Is Enough

For an IID sample, do not reject a strong-law conclusion merely because a variance is infinite. The finite-mean truncation strong law applies whenever \(E[|X_1|]<\infty\). It is particularly useful when the distribution has a heavy tail: the weak-law variance calculation and the independent-sequence variance criterion may both be unavailable, while the IID finite-mean theorem still applies.

Worked Example: A Pareto Sample With Infinite Variance

Let \(X_1,X_2,\ldots\) be IID with density \(f(x)=\tfrac32x^{-5/2}\) for \(x\geq1\), and zero otherwise. Determine whether the sample average converges, and identify its limit.

First verify that the density integrates to one: \(\int_1^\infty \tfrac32x^{-5/2}\,dx=[-x^{-3/2}]_1^\infty=1\). Its mean is

E[X_1]=\int_1^\infty x\frac{3}{2}x^{-5/2}\,dx =\frac{3}{2}\int_1^\infty x^{-3/2}\,dx=3.

The second moment is infinite: \(\int_1^\infty x^2\frac32x^{-5/2}\,dx=\frac32\int_1^\infty x^{-1/2}\,dx=\infty\). Thus the variance is infinite, and the finite-variance Chebyshev argument does not apply. But \(X_1\geq0\) and \(E[X_1]=3<\infty\), so \(E[|X_1|]<\infty\). The IID finite-mean strong law gives \(n^{-1}\sum_{i=1}^nX_i\xrightarrow{a.s.}3\), and therefore also convergence in probability to 3.

A Selection Procedure and a Common Pitfall

Identify the dependence and sampling structure.
Check whether the variables are IID, independent but not identically distributed, or merely dependent. Do not apply an independent-sequence theorem without independence.

Match the required conclusion.
If convergence in probability is enough, check the variance-of-the-average condition. If almost-sure convergence is needed, consider the finite-mean IID theorem or a suitable independent strong-law criterion.

Verify every hypothesis and the proposed limit.
Check moment conditions, calculate the relevant variance or series, and—in a nonidentical sequence—show that the averages of the expectations converge to the claimed value.

A common mistake is to treat \(\sum_{i=1}^n\operatorname{Var}(X_i)/n^2\to0\) as if it automatically proved almost-sure convergence. That condition supports the variance-based weak law: it bounds a probability tending to zero. Almost-sure convergence is a different conclusion. One available independent-sequence sufficient condition is the stronger requirement \(\sum_{i=1}^{\infty}\operatorname{Var}(X_i)/i^2<\infty\), together with its stated assumptions. Another route is the IID finite-mean strong law. Always name the theorem that supports the type of convergence you claim.

The practical value of theorem selection is that it prevents unnecessary assumptions and unjustified conclusions. A finite-variance weak law is quick when its conditions hold. Truncation extends the IID strong law to integrable heavy-tailed variables. A summability criterion handles many independent sequences with changing distributions. The right choice depends on the structure of the sample, the available moments, and whether the problem asks for convergence in probability or almost surely.

Check Your Understanding

For each question, identify the theorem or condition that is appropriate and state what conclusion it supports.

For IID Bernoulli observations, why does the variance criterion imply convergence in probability of the sample proportion?
An IID distribution has a finite absolute first moment but infinite variance. Which strong-law result can still apply?
For independent, nonidentically distributed variables, what additional deterministic limit must be checked when applying the Kolmogorov strong-law criterion to obtain convergence to a constant?
Does failure of the condition \(\sum_i\operatorname{Var}(X_i)/i^2<\infty\) prove that an independent sequence has no strong law? Explain.
Explain why \(\sum_{i=1}^n\operatorname{Var}(X_i)/n^2\to0\) supports a weak-law conclusion but is not, by itself, the strong-law criterion stated here.