From Averages to Empirical Moments
An empirical moment is an average of a transformed observation. The sample mean averages \(X_i\); the empirical second raw moment averages \(X_i^2\); and the empirical \(k\)th raw moment averages \(X_i^k\). The strong law applies to each of these averages, but its integrability condition must be checked for the transformed variable, not merely for \(X_i\).
This distinction matters in practice. A sample variance is built from squared observations, so convergence of the sample mean alone does not establish convergence of the sample variance. For that, the second absolute moment must be finite.
The Strong Law for Empirical Raw Moments
The finite-mean strong law states that if \(Y_1,Y_2,\ldots\) are iid and \(E[|Y_1|]<\infty\), then \(n^{-1}\sum_{i=1}^n Y_i\xrightarrow{a.s.}E[Y_1]\). Apply it with \(Y_i=X_i^k\). Since \(|X_i^k|=|X_i|^k\), the condition required is the finite absolute moment of order \(k\).
The justification is direct: the variables \(X_i^k\) are iid, and \(E[|X_i^k|]=E[|X_1|^k]<\infty\). The finite-mean strong law therefore gives the stated limit. If the assumption holds for each \(k\) in a finite or countable set of positive integers, convergence holds simultaneously for all those orders on an event of probability one: intersect the corresponding probability-one events. A countable intersection of probability-one events still has probability one.
For odd \(k\), positive and negative values of \(X_1^k\) can cancel in a formal calculation, but that cancellation does not replace the integrability condition. A formal or conditionally interpreted value for \(E[X_1^k]\) is not enough; the strong law requires \(E[|X_1|^k]<\infty\). In standard Lebesgue terminology, a finite \(E[X_1^k]\) already entails this absolute integrability.
Worked Example: Raw Moments of an Exponential Sample
Let \(X_1,X_2,\ldots\) be iid exponential random variables with rate \(2\), so their density is \(f(x)=2e^{-2x}\) for \(x\geq0\). Find the almost-sure limits of the first three empirical raw moments, and use them to find the limit of the sample variance with divisor \(n\).
For any integer \(k\geq1\), integration by parts gives $$ E[X_1^k]=2\int_0^\infty x^k e^{-2x}\,dx =\frac{k}{2}\,2\int_0^\infty x^{k-1}e^{-2x}\,dx =\frac{k}{2}E[X_1^{k-1}], $$ where the boundary term vanishes. Since \(E[X_1^0]=1\), this yields \(E[X_1]=1/2\), \(E[X_1^2]=1/2\), and \(E[X_1^3]=3/4\). All three absolute moments are finite because \(X_1\geq0\) and these integrals are finite.
The theorem consequently gives \(m_{n,1}\xrightarrow{a.s.}1/2\), \(m_{n,2}\xrightarrow{a.s.}1/2\), and \(m_{n,3}\xrightarrow{a.s.}3/4\). The sample variance with divisor \(n\), denoted here by \(v_n\), satisfies $$ v_n=\frac{1}{n}\sum_{i=1}^n(X_i-\bar X_n)^2 =m_{n,2}-m_{n,1}^2 \xrightarrow{a.s.}\frac12-\left(\frac12\right)^2=\frac14, $$ where \(\bar X_n=m_{n,1}\). The limit is the population variance. For \(n\geq2\), the usual sample variance with divisor \(n-1\) is \(s_n^2=\frac{n}{n-1}v_n\); since \(n/(n-1)\to1\), it has the same almost-sure limit.
Sample Variances and Centered Moments
The raw-moment result gives sample variance convergence through an exact algebraic identity. It also gives convergence of higher centered sample moments. For the latter, define the empirical \(r\)th centered moment as \(c_{n,r}=n^{-1}\sum_{i=1}^n(X_i-\bar X_n)^r\), where \(\bar X_n=m_{n,1}\).
To verify the centered-moment claim, expand each power using the binomial theorem and average: $$ c_{n,r}=\sum_{j=0}^r {r\choose j}(-\bar X_n)^{r-j}m_{n,j}. $$ The moment assumption of order \(r\) implies the absolute-moment assumptions for every \(j\leq r\), because \(|x|^j\leq 1+|x|^r\). Thus, on an event of probability one, \(\bar X_n\to\mu=E[X_1]\) and \(m_{n,j}\to E[X_1^j]\) for all \(0\leq j\leq r\). The right-hand side is a finite sum of products, so its limit is $$ \sum_{j=0}^r {r\choose j}(-\mu)^{r-j}E[X_1^j] =E\left[\sum_{j=0}^r {r\choose j}(-\mu)^{r-j}X_1^j\right] =E[(X_1-\mu)^r]. $$ The expectations in this calculation exist: \(E[|X_1-\mu|^r]<\infty\), for example by \(|X_1-\mu|^r\leq 2^{r-1}(|X_1|^r+|\mu|^r)\). For \(r=2\), the identity reduces to \(c_{n,2}=m_{n,2}-m_{n,1}^2\), with limit \(E[X_1^2]-(E[X_1])^2=\operatorname{Var}(X_1)\). Since \(s_n^2=\frac{n}{n-1}c_{n,2}\) and \(n/(n-1)\to1\), the corrected sample variance has the same limit. \(\square\)
Worked Example: Centered Moments for a Uniform Sample
Suppose \(X_i\) are iid uniform on \([-1,1]\). Compute the almost-sure limits of the first, second, and fourth raw empirical moments, then obtain the limiting empirical fourth centered moment.
For an integer \(k\geq0\), symmetry makes \(E[X_1^k]=0\) when \(k\) is odd. For even \(k\), direct integration gives $$ E[X_1^k]=\frac12\int_{-1}^1 x^k\,dx=\frac{1}{k+1}. $$ In particular, \(E[X_1]=0\), \(E[X_1^2]=1/3\), \(E[X_1^3]=0\), and \(E[X_1^4]=1/5\). The finite fourth absolute moment verifies the needed assumptions for every order through four. Therefore \(m_{n,1}\to0\), \(m_{n,2}\to1/3\), \(m_{n,3}\to0\), and \(m_{n,4}\to1/5\), all almost surely.
Expanding around the sample mean gives $$ c_{n,4}=m_{n,4}-4\bar X_n m_{n,3}+6\bar X_n^2m_{n,2}-4\bar X_n^3m_{n,1}+\bar X_n^4. $$ Since \(\bar X_n=m_{n,1}\to0\) almost surely, every term except \(m_{n,4}\) tends to zero. Hence \(c_{n,4}\xrightarrow{a.s.}1/5\), which agrees with \(E[X_1^4]\) because the population mean is zero.
When a Required Moment Is Infinite
The strong law for empirical moments is an order-by-order statement. Finiteness of a lower-order moment does not supply the integrability needed at a higher order. The following example shows both convergence at available orders and divergence at an order whose expectation is infinite.
Worked Example: A Pareto Sample with an Infinite Third Moment
Let \(X_i\) be iid with density \(f(x)=3x^{-4}\) for \(x\geq1\), and zero otherwise. For \(k\geq0\), $$ E[X_1^k]=3\int_1^\infty x^{k-4}\,dx. $$ For \(k<3\), this integral equals \(3/(3-k)\). Thus \(E[X_1]=3/2\) and \(E[X_1^2]=3\), so \(m_{n,1}\xrightarrow{a.s.}3/2\) and \(m_{n,2}\xrightarrow{a.s.}3\). The sample variance with divisor \(n\) consequently tends to \(3-(3/2)^2=3/4\) almost surely.
For \(k=3\), the integral is \(3\int_1^\infty x^{-1}\,dx=\infty\); in fact \(E[X_1^3]=\infty\). To see what happens to the empirical third moment, set \(Y_i=X_i^3\). These are iid, nonnegative, and \(E[Y_i]=\infty\). For each positive integer \(M\), the truncated variables \(Z_i^{(M)}=\min(Y_i,M)\) are iid and bounded. The finite-mean strong law gives \(n^{-1}\sum_i Z_i^{(M)}\to E[Z_1^{(M)}]\) almost surely. Also \(m_{n,3}=n^{-1}\sum_iY_i\geq n^{-1}\sum_iZ_i^{(M)}\). Intersecting the probability-one events over all integer \(M\), we have \(\liminf_n m_{n,3}\geq E[Z_1^{(M)}]\) for every \(M\). By monotone convergence, \(E[Z_1^{(M)}]\uparrow E[Y_1]=\infty\). Therefore \(m_{n,3}\to\infty\) almost surely. A finite second moment does not imply a finite third moment or a finite limit for empirical third moments.
Common Mistake and Practical Checks
A frequent error is to apply the strong law to \(X_i^k\) after checking only that \(E[|X_i|]<\infty\). That verifies the sample-mean result, but for the \(k\)th empirical raw moment the transformed variables are \(X_i^k\), and the necessary finite-mean condition is \(E[|X_i^k|]=E[|X_i|^k]<\infty\). The Pareto example shows why the distinction is substantive.
Before claiming convergence of an empirical moment, identify its order \(k\), verify the corresponding absolute moment, and then apply the strong law to \(X_i^k\). To infer convergence of a sample variance, check order two; to infer convergence of a centered moment of order \(r\), check order \(r\). If all absolute moments are finite, the countable-intersection argument yields simultaneous almost-sure convergence for every integer order.
Check Your Understanding
Use the order of the moment to decide which integrability condition is needed.
- For iid observations with \(E[|X_1|^4]<\infty\), what is the almost-sure limit of \(m_{n,4}\)?
- Why does \(E[|X_1|]<\infty\) alone not establish convergence of \(m_{n,2}\) to a finite population second moment?
- Write the identity relating the sample variance with divisor \(n\) to \(m_{n,1}\) and \(m_{n,2}\), and state its almost-sure limit under a finite second absolute moment.
- If \(E[|X_1|^3]<\infty\), which lower-order raw empirical moments also converge almost surely, and why?
- In the Pareto example, why does \(m_{n,3}\to\infty\) almost surely even though \(m_{n,2}\) converges to a finite value?