The CLT for Sampling Without Replacement

Why Sampling Without Replacement Changes the CLT

When a sample is drawn without replacement, selecting one population member changes which members remain available for later draws. The observations are therefore dependent. This matters for the variability of a sample mean or proportion, especially when the sample is a substantial fraction of the population.

The dependence does not make normal approximation impossible. Under suitable conditions, a sample mean or proportion can still have an approximately normal sampling distribution. The key adjustments are to use the correct standard error, which includes a finite population correction, and to check that the population and sample are not too small or dominated by a few unusual values.

Definition: In simple random sampling without replacement from a finite population of size $N$, every subset of $n$ distinct population members has the same probability of being selected. The population values are fixed; the sample and its statistics vary from one random sample to another.

For a population with values $y_1,\ldots,y_N$, let $\mu$ be the population mean and let $\sigma^2=\frac{1}{N}\sum_{i=1}^{N}(y_i-\mu)^2$ be the population variance. The sample mean is $\bar{x}$. For a binary population, let $K$ of the $N$ members have the characteristic of interest, so the population proportion is $p=K/N$; the sample proportion is $\hat{p}$.

The Exact Variance and the Finite Population Correction

A sample drawn with replacement, or from a very large population relative to $n$, has the familiar standard error $\sigma/\sqrt{n}$ for a sample mean. Without replacement, the exact variance is smaller. Define the finite population correction (FPC), for $n<N$, by

\mathrm{FPC}=\sqrt{\frac{N-n}{N-1}}.

Theorem: Under simple random sampling without replacement, the sample mean has $E(\bar{x})=\mu$ and $$ \operatorname{Var}(\bar{x})=\frac{\sigma^2}{n}\frac{N-n}{N-1}, \qquad SE(\bar{x})=\frac{\sigma}{\sqrt{n}}\sqrt{\frac{N-n}{N-1}}. $$ For a binary population with proportion $p$, the sample proportion has $E(\hat{p})=p$ and $$ \operatorname{Var}(\hat{p})=\frac{p(1-p)}{n}\frac{N-n}{N-1}, \qquad SE(\hat{p})=\sqrt{\frac{p(1-p)}{n}\frac{N-n}{N-1}}. $$

The correction is at most 1, so sampling without replacement reduces the standard error compared with the with-replacement calculation. If $n$ is small relative to $N$, the correction is close to 1 and has little effect. If $n=N$, the entire population has been observed and the sampling variance is zero.

Why the Correction Appears

A short derivation shows where the reduction comes from. Write $I_i=1$ if population member $i$ is selected and $I_i=0$ otherwise. Then $\bar{x}=\frac{1}{n}\sum_{i=1}^{N} I_i y_i$, and $E(I_i)=n/N$. For distinct members $i$ and $j$, the probability that both are selected is $n(n-1)/[N(N-1)]$. Consequently,

\operatorname{Cov}(I_i,I_j) =\frac{n(n-1)}{N(N-1)}-\left(\frac{n}{N}\right)^2 =-\frac{n(N-n)}{N^2(N-1)}.

The negative covariance expresses the sampling dependence: including one member slightly reduces the chance of including another. Expanding the variance of $\bar{x}$, the individual variance terms and these negative covariance terms combine to give the theorem’s formula. In particular, the identity $\sum_{i=1}^{N}(y_i-\mu)=0$ implies $\sum_{i\ne j}(y_i-\mu)(y_j-\mu)=-\sum_{i=1}^{N}(y_i-\mu)^2$. Substituting this identity into the variance expansion yields $\operatorname{Var}(\bar{x})=\frac{\sigma^2}{n}\frac{N-n}{N-1}$.

When Is a Normal Approximation Reasonable?

The variance formula is exact for every simple random sample without replacement, but a normal approximation is not. A finite-population CLT gives a normal approximation under regularity conditions: both the number sampled and the number left out should grow, and no small set of population values should dominate the total variation. One sufficient asymptotic condition, for a sequence of finite populations, is

\min(n,N-n)\longrightarrow\infty \quad\text{and}\quad \frac{\max_i (y_i-\mu)^2} {\min(n,N-n)\,S_F^2}\longrightarrow 0, \qquad S_F^2=\frac{1}{N-1}\sum_{i=1}^{N}(y_i-\mu)^2.

This condition formalizes two practical checks: the sample and the unsampled portion should not be tiny, and no individual population value should be extreme enough to control the result. It does not require the population itself to be normally distributed. As with the ordinary CLT, strong skewness, rare extreme values, or a very small effective sample can make a normal approximation poor.

For binary data, a useful practical check is that the expected numbers of successes and failures in both the sample and the unsampled portion are not small: $np,\ n(1-p),\ (N-n)p,$ and $(N-n)(1-p)$. Having each around 10 or more is a common rule of thumb, not a guarantee. For a general quantitative population, inspect its shape and whether extreme values are influential; do not rely on a fixed sample-size rule alone.

Normal approximation: When the finite-population CLT conditions are credible, use $$ \bar{x}\ \mathrel{\dot\sim}\ N\left(\mu,\frac{\sigma^2}{n}\frac{N-n}{N-1}\right), \qquad \hat{p}\ \mathrel{\dot\sim}\ N\left(p,\frac{p(1-p)}{n}\frac{N-n}{N-1}\right). $$ Here the second argument of $N(\cdot,\cdot)$ is the variance. These are approximations to the sampling distributions, not claims that the population values themselves are normally distributed.

Worked Examples

Worked Example: A Sample Mean From a Large Finite Population

Suppose a hypothetical finite population contains $N=500$ measurements, with $\mu=50$ units and $\sigma=10$ units. A simple random sample without replacement of $n=100$ measurements is drawn. Assume the population values are not strongly skewed and no few values dominate its spread. Approximate the probability that the sample mean exceeds 52 units.

The sample is 20% of the population, so dependence should be reflected in the standard error. There are also 400 population members left out, and the stated shape assumption supports using a normal approximation for this sample size.

Standard error. The FPC is $\sqrt{(500-100)/(500-1)}=\sqrt{400/499}\approx0.8957$. Thus

SE(\bar{x}) =\frac{10}{\sqrt{100}}\sqrt{\frac{400}{499}} =1(0.8957) =0.8957\text{ units}.

Probability. Standardize 52 units using the approximate sampling distribution:

z=\frac{52-50}{0.8957}\approx2.233, \qquad P(\bar{x}>52)\approx P(Z>2.233)\approx0.0128.

The estimated probability that this sample mean exceeds 52 units is about 0.0128. Without the FPC, the standard error would be 1 unit and the resulting upper-tail probability would be larger; that calculation would ignore the reduced variability from sampling a substantial fraction of the finite population.

Worked Example: A Sample Proportion Without Replacement

A hypothetical finite population has $N=1000$ people, of whom $K=300$ have a particular characteristic. A simple random sample without replacement of $n=200$ people is selected. Approximate the probability that the sample proportion is between 0.25 and 0.35.

The population proportion is $p=300/1000=0.30$. The expected sample counts are $np=60$ successes and $n(1-p)=140$ failures. The unsampled group has expected counts $800(0.30)=240$ successes and $800(0.70)=560$ failures. All four counts exceed 10, so the usual binary-data check supports a normal approximation.

Standard error.

SE(\hat{p}) =\sqrt{\frac{0.30(0.70)}{200}\frac{1000-200}{1000-1}} =\sqrt{0.00105\left(\frac{800}{999}\right)} \approx0.02900.

Probability. The endpoints are approximately $1.724$ standard errors below and above 0.30:

P(0.25\leq\hat{p}\leq0.35) \approx P\left(\frac{0.25-0.30}{0.02900}\leq Z\leq\frac{0.35-0.30}{0.02900}\right) =P(-1.724\leq Z\leq1.724) \approx0.915.

The normal approximation estimates about a 0.915 probability that the sample proportion is between 0.25 and 0.35. This probability describes repeated samples from this fixed population under the stated sampling design.

Worked Example: A Large Sampling Fraction

A hypothetical population has $N=100$ members, exactly $K=50$ of whom have a characteristic. A simple random sample without replacement of $n=80$ is taken. Approximate the probability of observing at least 45 successes. This example illustrates how substantially the FPC can change a tail calculation.

The sample has expected counts 40 successes and 40 failures. In the 20 people not sampled, the expected counts are 10 successes and 10 failures. These checks are at the common rule-of-thumb boundary, so the normal approximation is plausible but should be treated as an approximation, not an exact answer.

Let $X$ be the sample success count, so $\hat{p}=X/80$. Its mean is $E(X)=80(0.50)=40$, and its standard deviation is

SD(X) =80\sqrt{\frac{0.50(0.50)}{80}\frac{20}{99}} =\sqrt{20\left(\frac{20}{99}\right)} \approx2.010.

Since $X$ is discrete, use a continuity correction: $X\geq45$ is approximated by a continuous value above 44.5. Therefore

P(X\geq45) \approx P\left(Z\geq\frac{44.5-40}{2.010}\right) =P(Z\geq2.239) \approx0.0126.

The normal approximation is about 0.0126. If the FPC were ignored, the standard deviation would be $\sqrt{20}\approx4.472$, producing a much larger tail probability. Here that would misrepresent the sampling variability because 80 of the 100 population members are selected.

Common Mistake and Practical Use

A common mistake is to treat successive draws without replacement as independent simply because each sampled value is recorded once. For example, using $\sigma/\sqrt{n}$ for a sample mean when $n/N$ is substantial ignores the negative dependence and overstates the standard error. The correction is to use the FPC in the variance or standard error, then assess whether a normal approximation is justified.

A different mistake is to apply the correction mechanically and assume it makes any approximation accurate. The FPC corrects the variance; it does not repair severe skewness, a tiny sample, or a population dominated by rare extreme values. First identify the sampling design and the population; then calculate the adjusted standard error and check the normal-approximation conditions.

Describe the design.
Confirm that the sample is a simple random sample without replacement from a finite population of size $N$.

Calculate the adjusted standard error.
Use the appropriate formula for $\bar{x}$ or $\hat{p}$, including $\sqrt{(N-n)/(N-1)}$.

Check approximation conditions.
Assess sample and unsampled sizes, population shape, influential values, or the four expected binary counts.

Standardize and interpret.
Use the adjusted standard error to calculate the normal probability, and report it in the context of repeated samples from the finite population.

Check Your Understanding

Answer each question using the sampling design and quantities specified.

Why are observations from a simple random sample without replacement dependent?
A sample of 50 is drawn without replacement from a population of 500. Is the FPC closer to 1 or substantially below 1? What does this imply about its effect on the standard error?
For a binary population with $N=800$, $K=200$, and $n=400$, calculate the FPC and list the four expected counts relevant to a practical normal-approximation check.
Why is a correct finite population correction not, by itself, enough to justify a normal approximation?
A researcher samples every member of a finite population. What is the sampling variance of the sample mean, and why?