Sample Mean Risk Under IID Sampling

Why the Sample Mean’s Risk Needs Careful Assumptions

The sample mean is often introduced as an estimator of a population mean, and its familiar squared-error risk is the population variance divided by the sample size. That formula is useful, but its assumptions matter: unbiasedness needs a finite first moment, while the finite risk formula needs a finite second moment and independence. Separating these claims prevents a common mistake—assuming that unbiasedness alone guarantees a finite risk.

Let \(X_1,\ldots,X_n\) be IID observations from a distribution with mean \(\theta\), and consider estimating \(\theta\) by their arithmetic average. No Gaussian assumption is needed for the derivations below.

Definition: Suppose \(X_1,\ldots,X_n\) are independent and identically distributed, with \(E[X_i]=\theta\). The sample mean is \(\bar X=n^{-1}\sum_{i=1}^n X_i\). Under squared-error loss \(L(\theta,a)=(a-\theta)^2\), its risk is \(R(\theta,\bar X)=E_\theta[(\bar X-\theta)^2]\), where the expectation is taken under the distribution of the sample at parameter value \(\theta\). The risk is allowed to be \(+\infty\).

Unbiasedness Requires a Finite Mean

An estimator is unbiased for \(\theta\) if its expectation exists and equals \(\theta\). For the sample mean, it is enough that each observation has a finite first absolute moment and mean \(\theta\). This assumption allows linearity of expectation; it does not require a finite variance.

Result: If \(E[|X_i|]<\infty\) and \(E[X_i]=\theta\), then \(E[\bar X]=\theta\). Thus the sample mean is unbiased under a finite-mean assumption, whether or not \(\operatorname{Var}(X_i)\) is finite.

To verify the result, each \(X_i\) is integrable, so the finite sum defining \(\bar X\) is integrable. Linearity of expectation gives

E[\bar X] =E\!\left[\frac{1}{n}\sum_{i=1}^n X_i\right] =\frac{1}{n}\sum_{i=1}^n E[X_i] =\frac{1}{n}\sum_{i=1}^n\theta =\theta.

Independence was not needed for this calculation. Identical distributions with a common finite mean suffice for unbiasedness; indeed, even identical distributions are not essential if every observation has mean \(\theta\). Independence becomes important for the variance calculation.

Worked Example: A Bernoulli Sample Mean

Let \(X_1,\ldots,X_n\) be IID Bernoulli observations with \(P(X_i=1)=p\) and \(P(X_i=0)=1-p\). The parameter is the success probability \(\theta=p\), and the sample mean is the observed proportion of successes.

For one observation, \(E[X_i]=1\cdot p+0\cdot(1-p)=p\), and \(E[X_i^2]=1^2p+0^2(1-p)=p\). Therefore

\operatorname{Var}(X_i)=E[X_i^2]-(E[X_i])^2 =p-p^2=p(1-p).

For \(n=25\) and \(p=0.2\), the expected sample proportion is \(0.2\). Its risk under squared-error loss is

R(0.2,\bar X)=\frac{p(1-p)}{n} =\frac{0.2(0.8)}{25} =\frac{0.16}{25} =0.0064.

The risk is the expected squared error of the estimated proportion, not the probability that the sample proportion is incorrect.

Deriving the Finite-Variance Risk Formula

Now assume, in addition to independence and a common mean \(\theta\), that each observation has finite variance \(\operatorname{Var}(X_i)=\sigma^2<\infty\). The sample mean is unbiased, so its squared-error risk is its variance. We can also derive the result directly from the centered observations; the derivation makes explicit where independence enters.

Theorem: If \(X_1,\ldots,X_n\) are independent, have common mean \(\theta\), and have common finite variance \(\sigma^2\), then \(R(\theta,\bar X)=\operatorname{Var}(\bar X)=\sigma^2/n\). This is a finite-sample identity and requires no approximation.

Set \(Y_i=X_i-\theta\). Then \(E[Y_i]=0\), \(\operatorname{Var}(Y_i)=\sigma^2\), and \(\bar X-\theta=n^{-1}\sum_iY_i\). Expand the squared error:

R(\theta,\bar X) =E\!\left[\left(\frac{1}{n}\sum_{i=1}^nY_i\right)^2\right] =\frac{1}{n^2}\left\{ \sum_{i=1}^nE[Y_i^2] +2\sum_{1\le i<j\le n}E[Y_iY_j] \right\}.

For \(i\ne j\), independence and integrability imply \(E[Y_iY_j]=E[Y_i]E[Y_j]=0\). Also \(E[Y_i^2]=\sigma^2\). Hence the diagonal terms contribute \(n\sigma^2\), and all cross terms vanish:

R(\theta,\bar X) =\frac{n\sigma^2}{n^2} =\frac{\sigma^2}{n}.

Equivalently, because the estimator is unbiased, the bias-variance identity reduces its risk to its variance. Independence gives \(\operatorname{Var}(\sum_iX_i)=\sum_i\operatorname{Var}(X_i)\), so \(\operatorname{Var}(\bar X)=n^{-2}(n\sigma^2)=\sigma^2/n\). Both derivations rely on finite variances; the centered expansion also displays why independence removes the cross terms. \(\square\)

Worked Example: Risk for an Exponential Model

Suppose \(X_1,\ldots,X_n\) are IID exponential observations with density \(f(x;\theta)=\theta^{-1}e^{-x/\theta}\) for \(x>0\), where \(\theta>0\) is the mean. For this parameterization, \(E[X_i]=\theta\) and \(\operatorname{Var}(X_i)=\theta^2\), both finite. Thus the sample mean is unbiased and its squared-error risk is \(\theta^2/n\).

For \(n=4\) and \(\theta=2\), the individual variance is \(2^2=4\), and therefore

E[\bar X]=2,\qquad R(2,\bar X)=\operatorname{Var}(\bar X) =\frac{4}{4}=1.

The standard deviation of the estimation error is consequently \(1\), while its mean squared error is \(1\) as well; the two happen to have the same numerical value here, but represent different quantities.

When the Mean Exists but the Variance Does Not

A finite mean guarantees unbiasedness, but it does not ensure that the squared-error risk is finite. For an IID sample with finite mean and infinite variance, the sample mean remains unbiased, yet its squared-error risk is infinite for every fixed positive sample size.

Result: If \(E[|X_i|]<\infty\), \(E[X_i]=\theta\), and \(E[(X_i-\theta)^2]=+\infty\), then for every finite \(n\geq1\), \(E[(\bar X-\theta)^2]=+\infty\).

To see why, write \(Z_i=X_i-\theta\). Then \(E[|Z_i|]<\infty\), \(E[Z_i]=0\), and \(E[Z_i^2]=+\infty\). For \(n=1\), the claim is immediate. For \(n>1\), condition on \(Z_2,\ldots,Z_n\), and let \(c=\sum_{i=2}^n Z_i\). These observations are finite real numbers with probability one. Independence means that, conditional on them, \(Z_1\) retains its original distribution. For every fixed finite \(c\),

E[(Z_1+c)^2] =E[Z_1^2]+2cE[Z_1]+c^2 =+\infty.

The expansion is valid as an extended expectation: \(E[|Z_1|]<\infty\) makes the cross-term integrable for fixed \(c\), while the nonnegative square has infinite expectation because \(E[Z_1^2]=+\infty\). Thus the conditional expectation of \((\sum_i Z_i)^2\) is infinite, and so its unconditional expectation is infinite as well. Since \(\bar X-\theta=n^{-1}\sum_iZ_i\), multiplying the squared sum by \(n^{-2}\) leaves the expectation infinite. \(\square\)

Worked Example: An Unbiased Mean with Infinite Risk

Let each \(X_i\) have the Pareto density \(f(x)=\frac{3}{2}x^{-5/2}\) for \(x\geq1\), and zero otherwise. The density integrates to one because

\int_1^\infty \frac{3}{2}x^{-5/2}\,dx =\frac{3}{2}\left[\frac{-2}{3}x^{-3/2}\right]_1^\infty =1.

Its mean is finite:

E[X_i]=\frac{3}{2}\int_1^\infty x^{-3/2}\,dx =\frac{3}{2}\left[-2x^{-1/2}\right]_1^\infty =3.

But its second moment diverges:

E[X_i^2]=\frac{3}{2}\int_1^\infty x^{-1/2}\,dx =+\infty.

Consequently \(E[\bar X]=3\) for every finite \(n\), so the sample mean is unbiased for \(\theta=3\). The preceding result gives \(R(3,\bar X)=E[(\bar X-3)^2]=+\infty\) for every such \(n\). The estimator is unbiased, but the formula \(\sigma^2/n\) cannot be used with a finite \(\sigma^2\), and the risk is not finite.

Why Independence and the Target Mean Matter

The phrase “IID sampling” carries two distinct contributions. Identical distributions provide the same mean and variance for each observation. Independence makes distinct observations uncorrelated, which is why the cross terms in the risk derivation vanish. If the observations are dependent but have finite variances, the general variance calculation includes covariance terms:

\operatorname{Var}(\bar X) =\frac{1}{n^2}\left\{ \sum_{i=1}^n\operatorname{Var}(X_i) +2\sum_{1\le i<j\le n}\operatorname{Cov}(X_i,X_j) \right\}.

Thus a common marginal variance alone does not establish the \(\sigma^2/n\) risk formula. Positive dependence can increase the variance of the average; negative dependence can reduce it.

The parameter being estimated also matters. The sample mean is unbiased for the common population mean \(\theta\), not automatically for every parameter that may describe the distribution. The risk calculation is therefore tied to both the target and the sampling assumptions.

Common Mistake

Wrong reasoning: “The sample mean is unbiased, so its squared-error risk is its variance, and therefore the risk must be \(\sigma^2/n\).”

The first step needs a finite expectation, and the identity between unbiased-estimator risk and variance is understood with finite second moment for a finite variance. The division by \(n\) additionally uses independence and a common finite variance. In the Pareto example, unbiasedness holds but the squared-error risk is infinite. A safe procedure is to check the first moment for unbiasedness, then separately check the second moment and dependence structure before using the variance formula.

Key takeaway: For IID observations with mean \(\theta\) and finite variance \(\sigma^2\), the sample mean is unbiased and has squared-error risk \(\sigma^2/n\). A finite mean alone establishes unbiasedness; it does not guarantee finite risk.

Check Your Understanding

Use the assumptions carefully in each answer.

Which moment condition ensures that the sample mean has expectation \(\theta\)? Is independence needed for that conclusion?
For IID observations with mean \(\theta\) and finite variance \(\sigma^2\), derive the risk of \(\bar X\) under squared-error loss.
What role does independence play in the finite-variance risk derivation?
An IID distribution has a finite mean but infinite variance. What can be concluded about the sample mean’s unbiasedness and squared-error risk?
Why is it not sufficient to know that every observation has variance \(\sigma^2\) when the observations may be dependent?