Sample Moments and the Empirical Distribution

Why Summarize a Sample Empirically?

A sample gives a finite record of observed values, but we often need more than a list. Moments summarize features such as the sample's average and spread; the empirical distribution records what fraction of the observations fall at or below any chosen threshold. These summaries describe the data actually collected, while also being random quantities before the sample is observed.

Assume $X_1,\ldots,X_n$ are independent and identically distributed (IID) observations from a population distribution with cumulative distribution function $F_X(x)=P(X_1\le x)$. A sample summary is a function of these random observations. Once the sample is observed, the summary has a numerical value; across repeated samples, it can take different values.

Definition: The rth empirical raw moment is $M'_r=\frac{1}{n}\sum_{i=1}^n X_i^r$. The rth empirical central moment is $M_r=\frac{1}{n}\sum_{i=1}^n(X_i-\bar X)^r$, where $\bar X=\frac{1}{n}\sum_{i=1}^n X_i$. The empirical first raw moment is the sample mean, $M'_1=\bar X$. The empirical second central moment uses divisor $n$; the usual sample variance is $S^2=\frac{1}{n-1}\sum_{i=1}^n(X_i-\bar X)^2=\frac{n}{n-1}M_2$, for $n>1$.

The word empirical indicates that the calculation uses the observed sample, not the whole population. In particular, $M_2$ and $S^2$ are related but not the same statistic. Both are computed from the sample; the divisor distinguishes them.

The Empirical Distribution Function

For any real number $x$, the empirical distribution function (EDF) is the proportion of sample observations that are less than or equal to $x$. Write $\mathbf{1}\{X_i\le x\}=1$ when $X_i\le x$, and $0$ otherwise.

Definition: The empirical distribution function of $X_1,\ldots,X_n$ is $$ F_n(x)=\frac{1}{n}\sum_{i=1}^n \mathbf{1}\{X_i\le x\}. $$ It is a nondecreasing step function, equals $0$ below all observed values, and equals $1$ at or above the largest observed value. Each observation contributes a jump of $1/n$; if values are tied, their jumps occur at the same point and add together.

Thus, $F_n(x)$ is the sample proportion at or below $x$, not the population probability $F_X(x)$. For a set $A$, the empirical proportion in that set is $P_n(A)=\frac{1}{n}\sum_{i=1}^n\mathbf{1}\{X_i\in A\}$. The EDF is the special case $P_n((-\infty,x])$. Its use of “less than or equal to” matters at observed values: the jump at $x$ is already included in $F_n(x)$.

Worked Examples

Worked Example: Moments and the EDF From a Small Sample

A hypothetical quality audit records four component lifetimes, in hundreds of operating hours: $1,2,2,5$. Calculate the sample mean, the empirical second raw moment, the empirical second central moment, the usual sample variance, and selected values of the EDF.

Method. Use the four observations as the complete realized sample, retaining both copies of the tied value $2$. The sample size is $n=4$, and the sum is $1+2+2+5=10$, so $$ \bar X=\frac{10}{4}=2.5. $$

The sum of squared observations is $1^2+2^2+2^2+5^2=1+4+4+25=34$. Therefore the second raw moment is $$ M'_2=\frac{34}{4}=8.5. $$ The squared deviations from $2.5$ are $2.25,0.25,0.25,6.25$, whose sum is $9$. Hence $$ M_2=\frac{9}{4}=2.25,\qquad S^2=\frac{9}{4-1}=3. $$ As a check, $M'_2-\bar X^2=8.5-2.5^2=8.5-6.25=2.25=M_2$.

For the EDF, one of four observations is at or below $1$, while three are at or below $2$. Thus $F_4(1)=1/4=0.25$ and $F_4(2)=3/4=0.75$. In particular, the two tied observations at $2$ produce a jump of $2/4=0.50$ there. The proportion in $(1,2]$ is $F_4(2)-F_4(1)=0.75-0.25=0.50$.

Common mistake avoided: using $n-1$ in the definition of the empirical second central moment would give $3$, which is the sample variance, not $M_2$. The calculated values also satisfy $S^2=(4/3)M_2=(4/3)(2.25)=3$.

Worked Example: An Empirical Moment Across Repeated Samples

Suppose a hypothetical sensor reading $X$ is $1$ with probability $1/4$ and $3$ with probability $3/4$. For an IID sample of size $2$, find the distribution, expectation, and variance of the empirical second raw moment $M'_2$.

Method. Let $K$ be the number of readings equal to $3$. Independence gives $K\sim\operatorname{Binomial}(2,3/4)$, so $$ P(K=0)=\frac{1}{16},\qquad P(K=1)=2\left(\frac{3}{4}\right)\left(\frac{1}{4}\right)=\frac{6}{16},\qquad P(K=2)=\frac{9}{16}. $$ Each reading contributes $1$ to the sum of squares if it is $1$, and $9$ if it is $3$. Consequently, $$ M'_2=\frac{1}{2}\sum_{i=1}^2X_i^2 =\frac{(2-K)(1)+K(9)}{2} =1+4K. $$

The possible values and probabilities are therefore $1$ with probability $1/16$, $5$ with probability $6/16$, and $9$ with probability $9/16$. Its expectation is $$ E[M'_2]=1\left(\frac{1}{16}\right)+5\left(\frac{6}{16}\right)+9\left(\frac{9}{16}\right) =\frac{1+30+81}{16}=7. $$ Its variance follows from the same distribution: $$ E[(M'_2)^2]=1^2\left(\frac{1}{16}\right)+5^2\left(\frac{6}{16}\right)+9^2\left(\frac{9}{16}\right) =\frac{1+150+729}{16}=55, $$ $$ \operatorname{Var}(M'_2)=55-7^2=6. $$ As a check, the population second raw moment is $E[X^2]=1(1/4)+9(3/4)=7$, agreeing with the computed expectation. The empirical moment is still random: its three possible values are not all $7$.

Common mistake avoided: treating the empirical moment as if it must equal the population moment in each sample. Here it equals $7$ only on average over repeated samples; a particular sample gives $1$, $5$, or $9$.

Worked Example: Reconstructing Summaries From Frequencies

A hypothetical monitoring study reports four readings with this frequency table: value $10$ occurs once, value $20$ occurs twice, and value $40$ occurs once. Find the sample mean, $M'_2$, $M_2$, and $F_4(20)$. Also find the empirical proportion in $(20,40]$.

Method. The frequencies sum to $1+2+1=4$, so $n=4$. The sum of the readings is $10(1)+20(2)+40(1)=90$, and the sum of squares is $10^2(1)+20^2(2)+40^2(1)=100+800+1600=2500$. Hence $$ \bar X=\frac{90}{4}=22.5,\qquad M'_2=\frac{2500}{4}=625. $$

Use the identity $M_2=M'_2-\bar X^2$, or calculate the deviations directly. The identity gives $$ M_2=625-(22.5)^2=625-506.25=118.75. $$ Directly, the squared deviations are $156.25,6.25,6.25,306.25$; their sum is $475$, and $475/4=118.75$, confirming the result.

Three of the four readings are at or below $20$, so $F_4(20)=3/4=0.75$. Exactly one observation, the value $40$, lies in $(20,40]$, giving empirical proportion $1/4=0.25$. This agrees with $F_4(40)-F_4(20)=1-0.75=0.25$.

Common mistake avoided: the value $20$ is included in $F_4(20)$, but excluded from the interval $(20,40]$. The stated interval contains only the observation equal to $40$.

Why Empirical Summaries Are Random

Before the observations are drawn, each $X_i$ is random. Any formula built from them, including $M'_r$, $M_r$, and $F_n(x)$, is therefore a statistic and is random as well. After the data are observed, the statistic is evaluated at those values. The distinction is useful: the empirical distribution describes one sample, while the population distribution describes the model from which repeated samples are drawn.

The relationship between the EDF and the population distribution can be derived directly. Fix a real $x$, and define $I_i=\mathbf{1}\{X_i\le x\}$. Under the IID model, each $I_i$ is Bernoulli with success probability $F_X(x)$, and the indicators are independent. Since $F_n(x)=n^{-1}\sum I_i$, linearity of expectation gives

E[F_n(x)]=\frac{1}{n}\sum_{i=1}^nE[I_i] =\frac{1}{n}\,nF_X(x)=F_X(x).

Independence also makes the variance of the sum equal to the sum of the variances. A Bernoulli variable with success probability $F_X(x)$ has variance $F_X(x)[1-F_X(x)]$, so

\operatorname{Var}(F_n(x)) =\frac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(I_i) =\frac{F_X(x)[1-F_X(x)]}{n}.

Thus, at each fixed $x$, the EDF is an unbiased estimate of $F_X(x)$, and its variability depends on both the sample size and the population probability at that threshold. These statements require an IID sample; dependence or differing population distributions can change the formulas.

A parallel expectation result holds for empirical raw moments. If $E[|X_1|^r]$ is finite, then linearity of expectation and identical distributions give

E[M'_r] =E\left[\frac{1}{n}\sum_{i=1}^n X_i^r\right] =\frac{1}{n}\sum_{i=1}^n E[X_i^r] =E[X_1^r].

This does not say that every observed empirical moment equals its population counterpart. It says that the average of that statistic over repeated samples equals the population raw moment, when the required expectation exists. Central empirical moments involve the random sample mean inside every term, so their expectation does not follow from this simple linear argument.

Interpreting and Checking the Summaries

The empirical distribution can be viewed as assigning probability $1/n$ to each observation, counting repeated observations separately. Under this finite distribution, the mean is $M'_1$, and the $r$th raw moment is $M'_r$. This gives a useful justification for the moment formulas: an average under the empirical distribution is just the equally weighted average over the observed sample.

For the second central moment, expanding the square gives a computational identity:

M_2=\frac{1}{n}\sum_{i=1}^n(X_i-\bar X)^2 =\frac{1}{n}\sum_{i=1}^n X_i^2-\bar X^2 =M'_2-\bar X^2.

To verify it, expand each squared term. The sum is $\sum X_i^2-2\bar X\sum X_i+n\bar X^2$. Since $\sum X_i=n\bar X$, this becomes $\sum X_i^2-n\bar X^2$; dividing by $n$ gives the identity. It is a convenient arithmetic check, but it does not change the divisor distinction between $M_2$ and $S^2$.

Exam trap: Confusing the sample proportion below $x$ with a population probability, or confusing $M_2$ with $S^2$. For example, the incorrect calculation $F_n(20)=\#\{X_i<20\}/n$ excludes observations equal to $20$. The EDF uses $\le$, so in the frequency example $F_4(20)=(1+2)/4=0.75$, not $1/4$. Likewise, if $M_2=2.25$ for $n=4$, the sample variance is $(4/3)(2.25)=3$, not $2.25$.

Empirical summaries are useful because they reduce a sample to interpretable quantities without claiming that the sample is the entire population. Moments condense information about magnitude and spread; the EDF preserves information about proportions at every threshold. Always identify whether a question asks for a population quantity or a sample-based statistic, and state the divisor and inequality convention used.

Check Your Understanding

For each problem, identify the sample-based calculation carefully; in particular, keep tied observations and endpoint conventions explicit.

A sample is $2,2,4,8$. Find $\bar X$, $M'_2$, $M_2$, and $S^2$. (Answer: $\bar X=4$, $M'_2=22$, $M_2=6$, $S^2=8$.)
For the sample $3,5,5,9$, find $F_4(5)$ and the empirical proportion in $(3,5]$. (Answer: $F_4(5)=0.75$; the proportion in $(3,5]$ is $0.50$.)
Suppose an IID sample has size $5$, and $F_X(7)=0.4$. Find $E[F_5(7)]$ and $\operatorname{Var}(F_5(7))$. (Answer: $E[F_5(7)]=0.4$, $\operatorname{Var}(F_5(7))=0.4(0.6)/5=0.048$.)
A hypothetical population gives $X=0$ with probability $1/2$ and $X=2$ with probability $1/2$. For an IID sample of size $2$, list the possible values and probabilities of $M'_2$. (Answer: $M'_2=0$ with probability $1/4$, $M'_2=2$ with probability $1/2$, and $M'_2=4$ with probability $1/4$.)
A sample of size $4$ has $1$ observation at $0$, $2$ at $3$, and $1$ at $6$. Find $M'_2$, $M_2$, and $F_4(3)$. (Answer: $M'_2=13.5$, $M_2=4.5$, and $F_4(3)=0.75$.)

What You'll Learn