Estimator Sampling Distributions and Performance

Why an Estimator Has a Sampling Distribution

An estimator is calculated from a sample, so its value depends on which sample happened to be observed. To assess how well an estimator performs, consider the values it would produce across repeated samples drawn under the same model. That repeated-sample behavior is described by its sampling distribution.

This distinction is important: an estimator is a random variable before data are observed; its realized value is an estimate. The sampling distribution lets us summarize how the estimator is centered and how widely it varies. These summaries describe performance under a specified model and sample size, not certainty about the result from one particular sample.

Definition: A statistic is a function of observed random variables that does not depend on unknown model parameters. An estimator of a parameter $\theta$ is a statistic intended to estimate $\theta$, often denoted $\hat{\theta}$. Before observing the sample, $\hat{\theta}$ is random. Its probability distribution under the assumed sampling model is its sampling distribution. After observing data, the calculated value of $\hat{\theta}$ is an estimate.

For example, with a random sample $X_1,\ldots,X_n$, the sample mean $\bar X=(X_1+\cdots+X_n)/n$ is a statistic. If it is used to estimate a population mean $\mu$, it is an estimator of $\mu$. Once the sample values are known, $\bar X$ has a particular numerical value. Before they are known, it has a sampling distribution determined by the model for $X_1,\ldots,X_n$.

Center and Spread Describe Performance

Two basic features of a sampling distribution are its center and its spread. The expected value $E[\hat{\theta}]$ describes its probability-weighted center when that expectation exists. The variance $\operatorname{Var}(\hat{\theta})$, or its square root, the standard deviation, describes spread. A smaller variance means the estimator’s repeated-sample values are more tightly concentrated around their own center.

The location of the center relative to the target parameter also matters. An estimator may be tightly concentrated but centered away from the target, or more variable but centered close to it. Describing both location and spread avoids treating “precise” as equivalent to “close to the target.” A sampling distribution is always conditional on its assumptions: changing the model, dependence structure, or sample size can change the distribution and its performance measures.

Key formulas: If an estimator has finite second moment, its sampling variance is $$ \operatorname{Var}(\hat{\theta})=E\left[(\hat{\theta}-E[\hat{\theta}])^2\right] =E[\hat{\theta}^2]-\{E[\hat{\theta}]\}^2. $$ Its standard deviation is $\sqrt{\operatorname{Var}(\hat{\theta})}$, in the same units as the estimator and target.

The formulas are not a license to ignore the model. For instance, the familiar variance reduction for a sample mean requires independent observations with a common finite variance. If observations are dependent, the covariance terms must also be included.

Worked Sampling-Distribution Calculations

Worked Example: A Sample Mean From a Normal Population

Assume $X_1,\ldots,X_{16}$ are independent observations from a normal population $N(\mu,12^2)$, where $\mu$ is unknown and the population standard deviation is known to be 12. Use $\bar X$ to estimate $\mu$. Find its sampling distribution, center, spread, and the probability that it lies within 6 units of $\mu$.

Method. Use the exact distribution of a linear combination of independent normal random variables. Because each observation is normal, their average is normal: $$ \bar X\sim N\left(\mu,\frac{12^2}{16}\right)=N(\mu,9). $$ The second parameter of $N(\mu,\sigma^2)$ is the variance. Thus the sampling-distribution center is $\mu$, the variance is 9, and the standard deviation is $\sqrt{9}=3$.

To calculate the requested probability, standardize using the sampling standard deviation, not the population standard deviation: $$ P(|\bar X-\mu|\leq 6) =P\left(\left|\frac{\bar X-\mu}{3}\right|\leq\frac{6}{3}\right) =P(|Z|\leq 2), $$ where $Z\sim N(0,1)$. Therefore $$ P(|Z|\leq 2)=\Phi(2)-\Phi(-2)=0.97725-0.02275=0.9545. $$ The probability is 0.9545 to four decimal places.

Check. The standard error is $12/\sqrt{16}=3$, matching the standard deviation from the distribution. The interval of 6 units is exactly two standard errors on either side of the center. The common mistake avoided here is dividing by 12 when standardizing the sample mean; 12 is the spread of one observation, while the mean’s spread is 3.

Worked Example: A Sample Proportion From Bernoulli Trials

Suppose 20 independent policy applications are classified as either complete or incomplete. Under the model, each is complete with probability $p=0.30$, independently of the others. Let $K$ be the number classified complete, and let $\hat p=K/20$ estimate $p$. Find the sampling distribution, center, variance, and standard deviation of $\hat p$.

Method. Each classification is a Bernoulli random variable with success probability 0.30, so the count $K$ has a binomial distribution: $$ K\sim\operatorname{Binomial}(20,0.30),\qquad P(K=k)=\binom{20}{k}(0.30)^k(0.70)^{20-k},\quad k=0,\ldots,20. $$ Since $\hat p=K/20$, its sampling distribution assigns probability $P(K=k)$ to the value $k/20$. Its possible values are $0,0.05,0.10,\ldots,1$.

A binomial count has mean $np$ and variance $np(1-p)$. Consequently, $$ E[\hat p]=E[K/20]=\frac{20(0.30)}{20}=0.30, $$ and $$ \operatorname{Var}(\hat p)=\frac{\operatorname{Var}(K)}{20^2} =\frac{20(0.30)(0.70)}{400} =\frac{4.2}{400}=0.0105. $$ The standard deviation is $$ \sqrt{0.0105}=0.1025 $$ to four decimal places.

Check. The standard deviation is on the proportion scale, as are $\hat p$ and $p$. Equivalently, the usual formula gives $\sqrt{p(1-p)/n}=\sqrt{0.21/20}=\sqrt{0.0105}$. The common mistake avoided is reporting the count’s variance, 4.2, as the variance of the proportion; dividing the count by 20 divides its variance by $20^2$, not by 20.

Worked Example: The Maximum of a Uniform Sample

Let $X_1,\ldots,X_4$ be independent observations from a uniform distribution on $[0,\theta]$, with $\theta=10$. Consider $M=\max(X_1,\ldots,X_4)$, the sample maximum, as an estimator of the upper endpoint. Find its sampling distribution, center, variance, and standard deviation.

Method. For $0\leq m\leq 10$, the event $M\leq m$ occurs exactly when all four observations are at most $m$. Independence gives $$ P(M\leq m)=\prod_{j=1}^4P(X_j\leq m) =\left(\frac{m}{10}\right)^4. $$ The full cumulative distribution function is therefore 0 for $m<0$, $(m/10)^4$ for $0\leq m\leq10$, and 1 for $m>10$. Differentiating within the interval gives the density $$ f_M(m)=\frac{4m^3}{10^4},\qquad 0<m<10. $$

Use this density to calculate the first two moments: $$ E[M]=\int_0^{10}m\frac{4m^3}{10^4}\,dm =\frac{4}{10^4}\left[\frac{m^5}{5}\right]_0^{10} =8, $$ and $$ E[M^2]=\int_0^{10}m^2\frac{4m^3}{10^4}\,dm =\frac{4}{10^4}\left[\frac{m^6}{6}\right]_0^{10} =\frac{200}{3}. $$ Thus $$ \operatorname{Var}(M)=E[M^2]-\{E[M]\}^2 =\frac{200}{3}-64=\frac{8}{3}, $$ and the standard deviation is $\sqrt{8/3}=1.633$ to three decimal places.

Check. The maximum cannot exceed 10, and its calculated center of 8 lies within the possible range. Its standard deviation is approximately 1.633 endpoint units. The common mistake avoided is using the distribution of one uniform observation for the maximum: the fourth power in the cumulative distribution arises because all four independent observations must be at most $m$.

Why These Calculations Work

A useful general justification explains the behavior of a sample mean. Suppose $X_1,\ldots,X_n$ are independent observations from the same distribution, with finite mean $\mu$ and variance $\sigma^2$. Linearity of expectation gives $$ E[\bar X]=E\left[\frac{1}{n}\sum_{j=1}^nX_j\right] =\frac{1}{n}\sum_{j=1}^nE[X_j] =\frac{n\mu}{n}=\mu. $$

For the variance, scaling by $1/n$ multiplies variance by $1/n^2$. Independence means the variance of the sum is the sum of the variances, so $$ \operatorname{Var}(\bar X) =\frac{1}{n^2}\operatorname{Var}\left(\sum_{j=1}^nX_j\right) =\frac{1}{n^2}\sum_{j=1}^n\operatorname{Var}(X_j) =\frac{n\sigma^2}{n^2} =\frac{\sigma^2}{n}. $$ Thus its standard deviation is $\sigma/\sqrt n$. This derivation does not assert that $\bar X$ is normally distributed for every population. Exact normality follows when the observations are normal; otherwise, a normal approximation requires additional justification and is not being assumed here.

The independence assumption is substantive. Without it, the variance of the sum includes covariances: $$ \operatorname{Var}(\bar X)=\frac{1}{n^2}\left(\sum_{j=1}^n\operatorname{Var}(X_j) +2\sum_{j<k}\operatorname{Cov}(X_j,X_k)\right). $$ Positive dependence can increase the spread relative to the independent-sample formula. Always identify the sampling model before applying a standard error formula.

Comparing Estimators and Avoiding Traps

When two estimators target the same parameter, compare their sampling distributions rather than their formulas alone. Their centers show where repeated estimates cluster; their variances show how dispersed those estimates are. A smaller variance indicates less variability around the estimator’s own center, but does not by itself guarantee that estimates are closer to the target.

A frequent exam error is to treat the spread of individual observations as the spread of an estimator. For independent observations with variance $\sigma^2$, an individual observation has variance $\sigma^2$, while their sample mean has variance $\sigma^2/n$. Another error is to use a formula outside its assumptions: a binomial sampling distribution requires independent Bernoulli trials with the same success probability, and the uniform-maximum calculation requires independent observations from the stated uniform model.

Practical sequence: State the model and independence assumptions; define the statistic; derive or identify its sampling distribution; then calculate its center and spread using the estimator’s scale. Check that the support and units make sense before interpreting the result.

Sampling distributions matter because they turn a proposed estimator into an object that can be evaluated mathematically. They show how sample size and model assumptions affect repeated-sample behavior and provide the foundation for later comparisons of estimation procedures. The same discipline applies throughout: specify what is random, identify the distribution under the model, and keep the estimator’s target and units visible.

Check Your Understanding

For each problem, identify the estimator’s distribution before calculating the requested performance measures.

Independent Bernoulli trials have success probability $0.40$, and $n=25$. For the sample proportion $\hat p$, find its mean, variance, and standard deviation. (Answer: mean $0.40$, variance $0.0096$, standard deviation $0.0980$.)
Suppose $X_1,\ldots,X_9$ are independent $N(\mu,3^2)$ observations. Find the sampling distribution of $\bar X$ and $P(|\bar X-\mu|\leq2)$. (Answer: $\bar X\sim N(\mu,1)$; probability $0.9545$.)
Let $M$ be the maximum of three independent observations from a uniform distribution on $[0,12]$. Find its cumulative distribution function on $0\leq m\leq12$, mean, and variance. (Answer: $F_M(m)=(m/12)^3$; mean $9$; variance $5.4$.)
Five independent observations have a Poisson distribution with parameter $\lambda=2$. For $\bar X$, find its mean, variance, and probability of the specific value $\bar X=2$. (Answer: mean $2$, variance $0.4$, $P(\bar X=2)=e^{-10}10^{10}/10!=0.1251$.)
Suppose $X_1,\ldots,X_{10}$ are independent with mean $\mu$ and variance $25$. Compare the variances and standard deviations of $\bar X$ and $X_1$, when each is used as an estimator of $\mu$. (Answer: $\operatorname{Var}(\bar X)=2.5$, SD $1.5811$; $\operatorname{Var}(X_1)=25$, SD $5$.)

What You'll Learn