Laws of Large Numbers for Bounded Functions of IID Observations

Why Transformations Have Their Own Laws of Large Numbers

A statistic is often an average of quantities calculated from observations rather than an average of the observations themselves. An event count, for instance, averages indicator variables; a bounded score or transformed measurement averages values $g(X_i)$. The relevant question is whether that transformed average settles near a fixed number as the sample grows.

For IID observations, a single condition makes many such applications straightforward: the function being averaged is measurable and bounded, at least almost surely. The observations themselves need not be bounded, and they need not even have a finite mean. What matters here is that the transformed observations have a finite expectation and enough control of their tails to apply the laws.

Definition: Let $X_1,X_2,\ldots$ be IID random variables with common distribution $F$, and let $g:\mathbb{R}\to\mathbb{R}$ be Borel measurable. Suppose there is a finite constant $M$ such that $P(|g(X_1)|\le M)=1$. Define $$ \mu_g=E[g(X_1)]=\int_{\mathbb{R}} g(x)\,dF(x),\qquad A_n(g)=\frac{1}{n}\sum_{i=1}^n g(X_i). $$ The bounded-functions law of large numbers concerns the convergence of $A_n(g)$ to $\mu_g$. The bound is required only almost surely under $F$; $g$ need not be bounded at points that $X_1$ reaches with probability zero.

Since $g$ is measurable, $g(X_i)$ is a random variable. Applying the same function separately to IID observations gives IID transformed variables. In particular, their common expectation is $\mu_g$, and the boundedness assumption guarantees that this expectation and their variance are finite. No continuity of $g$ is needed.

The Weak Law for Bounded Transformations

The weak law states that, for every $\varepsilon>0$, the probability that $A_n(g)$ differs from $\mu_g$ by more than $\varepsilon$ tends to zero. Here the variance of an average decreases at rate $1/n$, so Chebyshev’s inequality gives the result directly.

Theorem (weak law for bounded IID functions): Under the definition above, $$ A_n(g)\xrightarrow{p}\mu_g. $$ In fact, for every $\varepsilon>0$, $$ P\bigl(|A_n(g)-\mu_g|>\varepsilon\bigr) \le \frac{\operatorname{Var}(g(X_1))}{n\varepsilon^2} \le \frac{M^2}{n\varepsilon^2}. $$

Proof. Write $Z_i=g(X_i)$. The $Z_i$ are IID, and $|Z_i|\le M$ almost surely, so $\operatorname{Var}(Z_i)\le E[Z_i^2]\le M^2$. Independence gives $$ \operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^n Z_i\right) =\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(Z_i) =\frac{\operatorname{Var}(Z_1)}{n}. $$ Applying Chebyshev’s inequality to $A_n(g)$, whose expectation is $\mu_g$, yields the first bound. The second follows from $\operatorname{Var}(Z_1)\le M^2$, and the bound tends to zero as $n$ tends to infinity. This is convergence in probability. $\square$

Worked Example: A Threshold Event Under an Exponential Model

Let $X_1,X_2,\ldots$ be IID exponential random variables with rate $1$, so $P(X_i>t)=e^{-t}$ for $t\ge 0$. Take $g(x)=\mathbf{1}\{x>\ln 2\}$. Find the limit of the average of these indicators and give a weak-law bound.

The indicator function is measurable and takes only the values $0$ and $1$, so it is bounded by $M=1$. Its expectation is the probability of the threshold event: $$ \mu_g=E[\mathbf{1}\{X_1>\ln 2\}] =P(X_1>\ln 2) =e^{-\ln 2} =\frac{1}{2}. $$

For a Bernoulli random variable with success probability $1/2$, the variance is $(1/2)(1-1/2)=1/4$. Therefore, for every $\varepsilon>0$, $$ P\left(\left|\frac{1}{n}\sum_{i=1}^n\mathbf{1}\{X_i>\ln 2\}-\frac{1}{2}\right|>\varepsilon\right) \le \frac{1}{4n\varepsilon^2}. $$ The bound tends to zero, so the fraction of observations exceeding $\ln 2$ converges in probability to $1/2$.

The Strong Law for Bounded Transformations

For bounded IID transformations, the almost-sure conclusion can also be obtained with an elementary fourth-moment argument. Boundedness makes the fourth moment finite, and the resulting probability bounds are summable. The first Borel–Cantelli lemma then turns those bounds into almost-sure convergence.

Theorem (strong law for bounded IID functions): Under the definition above, $$ A_n(g)\xrightarrow{a.s.}\mu_g. $$

Proof. Put $U_i=g(X_i)-\mu_g$. These variables are IID, have mean zero, and satisfy $|U_i|\le 2M$ almost surely, since $|\mu_g|\le E|g(X_i)|\le M$. If $M=0$, every $U_i=0$ almost surely and the claim follows immediately. Suppose $M>0$.

Expanding the fourth power of $\sum_{i=1}^n U_i$ and taking expectations, any term with an index appearing exactly once has expectation zero: independence factors out the mean of that variable. The remaining terms either use one index four times or two distinct indices twice each. There are six arrangements for each pair of distinct indices. Consequently, $$ E\left[\left(\sum_{i=1}^n U_i\right)^4\right] =\sum_{i=1}^n E[U_i^4] +6\sum_{1\le i<j\le n}E[U_i^2]E[U_j^2]. $$ Because $|U_i|\le 2M$, both $E[U_i^4]\le 16M^4$ and $E[U_i^2]\le 4M^2$. Hence $$ E\left[\left(\sum_{i=1}^n U_i\right)^4\right] \le 16nM^4+96\binom{n}{2}M^4 =16nM^4+48n(n-1)M^4 \le 48n^2M^4. $$ Markov’s inequality applied to the nonnegative fourth power now gives, for every $\varepsilon>0$, $$ P\bigl(|A_n(g)-\mu_g|>\varepsilon\bigr) =P\left(\left|\sum_{i=1}^n U_i\right|>n\varepsilon\right) \le \frac{48M^4}{n^2\varepsilon^4}. $$ For a fixed $\varepsilon>0$, the sum of these bounds over $n$ is finite. By the first Borel–Cantelli lemma, the event $|A_n(g)-\mu_g|>\varepsilon$ occurs only finitely often, almost surely. Apply this conclusion to the countable sequence $\varepsilon=1/k$, $k=1,2,\ldots$. The intersection of the resulting probability-one events still has probability one. On that intersection, for every $k$, eventually $|A_n(g)-\mu_g|\le 1/k$. This is exactly $A_n(g)\to\mu_g$, proving the strong law. $\square$

The argument also shows why the IID assumption matters: independence is used both to discard mixed terms in the fourth-moment expansion and to factor the terms that remain. Boundedness by itself does not supply those steps.

Worked Example: A Bounded Nonlinear Transformation of a Uniform Variable

Let $X_1,X_2,\ldots$ be IID uniform random variables on $[0,1]$, and set $g(x)=1/(1+x)$ on this interval. Find the limiting value of the transformed average and verify that the bounded-function laws apply.

The function is continuous, hence measurable, and on $[0,1]$ it satisfies $1/2\le g(x)\le 1$. Its expectation is $$ \mu_g=E\left[\frac{1}{1+X_1}\right] =\int_0^1\frac{1}{1+x}\,dx =\bigl[\ln(1+x)\bigr]_0^1 =\ln 2. $$

Thus both the weak and strong laws apply: $$ \frac{1}{n}\sum_{i=1}^n\frac{1}{1+X_i} \xrightarrow{p}\ln 2 \qquad\text{and}\qquad \frac{1}{n}\sum_{i=1}^n\frac{1}{1+X_i} \xrightarrow{a.s.}\ln 2. $$ For an explicit weak-law bound, $g(X_1)^2\le 1$ implies $\operatorname{Var}(g(X_1))\le 1$, so the probability of an error greater than $\varepsilon$ is at most $1/(n\varepsilon^2)$.

Worked Example: A Trigonometric Transformation

Let $X_1,X_2,\ldots$ be IID uniform random variables on $[0,2\pi]$, and let $g(x)=\cos x$. Determine the limit of their transformed average, and calculate its variance-based weak-law bound.

The function is measurable and bounded in absolute value by $1$. Its expectation is $$ \mu_g=\frac{1}{2\pi}\int_0^{2\pi}\cos x\,dx =\frac{1}{2\pi}[\sin x]_0^{2\pi}=0. $$ Also, using $\cos^2 x=(1+\cos 2x)/2$, $$ E[g(X_1)^2] =\frac{1}{2\pi}\int_0^{2\pi}\cos^2 x\,dx =\frac{1}{2}, \qquad \operatorname{Var}(g(X_1))=\frac{1}{2}-0^2=\frac{1}{2}. $$ Therefore, $$ P\left(\left|\frac{1}{n}\sum_{i=1}^n\cos X_i\right|>\varepsilon\right) \le \frac{1}{2n\varepsilon^2}, $$ and the strong law also gives $n^{-1}\sum_{i=1}^n\cos X_i\to 0$ almost surely.

Interpreting the Assumptions and Avoiding Pitfalls

The limit is $E[g(X_1)]$, not generally $g(E[X_1])$. For the uniform example, $E[g(X_1)]=\ln 2$, whereas $g(E[X_1])=g(1/2)=2/3$; these are different numbers. An average of transformed observations targets the expectation of the transformed observation.

Another mistake is to treat boundedness as a substitute for independence. For example, let $Z$ be a single Bernoulli random variable with success probability $1/2$, and set $X_i=Z$ for every $i$. These observations are bounded and identically distributed, but they are completely dependent. With $g(x)=x$, every average equals $Z$, so it does not converge in probability to $E[Z]=1/2$: the probability of an error greater than $1/4$ is $1$ for every $n$. The IID hypothesis in the theorems is substantive.

The value of these results is their breadth. Once a measurable transformation is bounded, its average has a well-defined limit even if the original observations have no finite mean. The task in an application is to identify the actual function being averaged, check its assumptions under the common distribution, and compute its expectation.

Key takeaway: If $X_i$ are IID and $g(X_i)$ is measurable and bounded almost surely, then its average converges in probability and almost surely to $E[g(X_1)]$. The transformation determines the target; independence and boundedness justify the convergence.

Check Your Understanding

For each question, identify the transformed variable and use its expectation as the candidate limit.

If $X_i$ are IID exponential with rate $2$, what is $E[\mathbf{1}\{X_i>1\}]$, and what does the weak law say about the average of these indicators?
Why is Borel measurability of $g$ needed when defining $g(X_i)$?
For IID uniform observations on $[0,1]$, find $E[1/(2+X_i)]$ and state the almost-sure limit of its average.
Which part of the fourth-moment proof uses independence?
Can the bounded-function laws be applied if $g$ is unbounded only on a set having probability zero under $F$? Explain.

Laws of Large Numbers for Bounded Functions of IID Observations

What You'll Learn