Sampling With and Without Replacement

Why the Sampling Design Matters

Imagine drawing several names from a fixed list. If each name is returned to the list before the next draw, every draw starts from the same population. If names are not returned, each draw changes what remains. That difference affects not only which samples are possible, but also how much sample statistics such as a sample proportion or sample mean vary from sample to sample.

A sampling distribution describes the possible values of a statistic over repeated samples taken according to a specified design. The design is therefore part of the sampling model: knowing the population alone is not enough to determine how samples behave. The key distinction here is whether the sampled unit is replaced before another draw is made.

Definition: In sampling with replacement, each sampled unit is returned to the population before the next draw. In sampling without replacement, sampled units are not returned. With replacement, successive draws from a fixed population are independent and have the same distribution. Without replacement, the draws are dependent because earlier selections change the units available for later draws.

In both designs, suppose each draw is made at random from the units currently available. Without replacement, this gives a simple random sample when every subset of the required size is equally likely. A unit cannot be selected twice without replacement, although two different units may share the same recorded value. With replacement, the same unit can be drawn more than once.

How the Draws Become Dependent

Consider a fixed population with \(N\) units, of which \(K\) have a particular attribute. Let \(p=K/N\) be the population proportion with that attribute. On the first draw, the probability of selecting a unit with the attribute is \(p\) under either design. The conditional probability on the second draw is where the designs differ.

With replacement, the population is restored after the first draw, so the probability of a success on the second draw remains \(K/N\), whatever happened on the first draw. Without replacement, a first-draw success leaves only \(K-1\) successes among \(N-1\) remaining units. A first-draw failure leaves all \(K\) successes among those \(N-1\) units. Thus the second-draw probability responds to the first result.

Worked Example: Conditional Probabilities From a Small Population

A fixed group contains 10 units, 4 of which have a specified attribute. Compare the probability of a second success after a first success with replacement and without replacement.

With replacement: After the first unit is returned, 4 of the 10 units have the attribute. Therefore,

P(\text{second success}\mid\text{first success})=\frac{4}{10}=0.4000.

Without replacement: The first success has been removed, leaving 3 successes among 9 units. Therefore,

P(\text{second success}\mid\text{first success})=\frac{3}{9}=0.3333.

If the first draw is a failure and there is no replacement, the second-draw probability is \(4/9=0.4444\), since all 4 successes remain among 9 units. These different conditional probabilities show that the draws are dependent without replacement. With replacement, the second-draw probability is \(0.4000\) after either first-draw result, consistent with independence.

The dependence without replacement is generally negative for numerical values: a draw that is higher than the population average tends to leave a slightly lower average among the remaining units. The following derivation makes that claim precise.

Why Dependence Changes the Variability of a Mean

Represent the fixed population by values \(a_1,\ldots,a_N\), with population mean \(\mu\) and population variance \(\sigma^2=(1/N)\sum_{i=1}^{N}(a_i-\mu)^2\). Let \(X_1\) and \(X_2\) be the values on the first two draws. With replacement, these draws are independent, so their covariance is zero. Without replacement, every ordered pair of distinct units is equally likely.

Derivation: For two draws without replacement, the average product over ordered pairs of distinct units is

E[X_1X_2] =\frac{\sum_{i\ne j}a_i a_j}{N(N-1)} =\frac{\left(\sum_i a_i\right)^2-\sum_i a_i^2}{N(N-1)}.

Since \(\sum_i a_i=N\mu\) and \(\sum_i a_i^2=N(\mu^2+\sigma^2)\), substitution gives

E[X_1X_2] =\frac{N^2\mu^2-N(\mu^2+\sigma^2)}{N(N-1)} =\mu^2-\frac{\sigma^2}{N-1}.

Both draw positions have mean \(\mu\), so their covariance is

\operatorname{Cov}(X_1,X_2) =E[X_1X_2]-E[X_1]E[X_2] =-\frac{\sigma^2}{N-1}.

For a sample of size \(n\), the sample mean is \(\bar{x}=(X_1+\cdots+X_n)/n\). Its variance is the sum of the individual variances and all pairwise covariances, divided by \(n^2\). Under simple random sampling without replacement, each draw has variance \(\sigma^2\), and each distinct pair has the covariance just derived. Therefore,

\operatorname{Var}(\bar{x}) =\frac{1}{n^2}\left[n\sigma^2+n(n-1)\left(-\frac{\sigma^2}{N-1}\right)\right].

The negative covariance reduces the variance compared with independent draws, for which the pairwise covariance terms are zero and \(\operatorname{Var}(\bar{x})=\sigma^2/n\). This reasoning applies to sample proportions as well: a proportion is an average of zero-one values indicating whether each unit has the attribute. When the sample is drawn without replacement, the dependence among those indicators affects the proportion's variability too.

Worked Example: Comparing Sample-Mean Variability

Consider a fixed population with values 2, 4, and 8 units. Draw a sample of size 2 and calculate the variance of its mean under each design. Assume draws are uniform among available units; with replacement, the two draws are independent.

The population mean is \(\mu=(2+4+8)/3=14/3\). Its variance is

\sigma^2 =\frac{(2-14/3)^2+(4-14/3)^2+(8-14/3)^2}{3} =\frac{56}{9}.

With replacement: Each draw has variance \(56/9\), and the covariance is zero. Thus,

\operatorname{Var}(\bar{x}) =\frac{1}{2^2}\left(\frac{56}{9}+\frac{56}{9}\right) =\frac{28}{9}\approx 3.111.

Without replacement: The three possible unordered samples are \(\{2,4\}\), \(\{2,8\}\), and \(\{4,8\}\), each with probability \(1/3\). Their means are 3, 5, and 6. The mean of this sampling distribution is \((3+5+6)/3=14/3\), and its variance is

\operatorname{Var}(\bar{x}) =\frac{(3-14/3)^2+(5-14/3)^2+(6-14/3)^2}{3} =\frac{14}{9}\approx 1.556.

For this population and sample size, the sample mean varies less without replacement: its variance is \(14/9\) square units rather than \(28/9\) square units. The lower variability follows from the negative dependence between the two draws.

Proportions Under the Two Designs

For a binary attribute, let each sampled value be 1 for a unit with the attribute and 0 otherwise. The sample proportion \(\hat{p}\) is the average of these indicators. Without replacement, selecting one success makes another success less likely on the next draw; selecting a failure makes another success more likely than it was after a success. These opposing conditional changes constrain how extreme the sample proportion can be compared with independent draws.

Worked Example: A Sample Proportion From Five Units

A fixed population contains 5 units, of which 3 have a specified attribute. A sample of size 3 is drawn. Compare the variance of \(\hat{p}\) with and without replacement. The proportion's units are a share of sampled units, so its variance is in squared proportion units.

Without replacement: There are \(\binom{5}{3}=10\) equally likely samples. If the sample contains \(j\) successes, the number of such samples is \(\binom{3}{j}\binom{2}{3-j}\). Thus \(j=1,2,3\) have counts \(3,6,1\), respectively. The corresponding proportions are \(1/3,2/3,1\), with probabilities \(0.3,0.6,0.1\). Their mean is

E[\hat{p}]=0.3(1/3)+0.6(2/3)+0.1(1)=0.6000.

The variance is

\operatorname{Var}(\hat{p}) =0.3(1/3-0.6)^2+0.6(2/3-0.6)^2+0.1(1-0.6)^2 =0.04000.

With replacement: Each draw is an independent success with probability \(p=3/5=0.6\). The sample proportion is the average of three independent zero-one indicators, each with variance \(p(1-p)=0.24\). Hence

\operatorname{Var}(\hat{p}) =\frac{0.24}{3} =0.08000.

The without-replacement variance is half the with-replacement variance in this example. Both designs have expected sample proportion \(0.6000\), but the dependence among draws changes the spread of possible sample proportions.

Choosing and Describing the Sampling Model

Sampling without replacement is natural when a sample consists of distinct units from a finite group, such as patients on a roster or items in a shipment. Sampling with replacement is often used as a mathematical model for repeated independent draws, or when the source population is so large relative to the sample that the depletion caused by sampling is negligible for the question at hand. That approximation should be justified by the setting rather than assumed automatically.

When the sample is a substantial part of a finite population, the no-replacement dependence can make a meaningful difference to variability. A census, in which every unit is observed, has no sampling variation in a statistic that summarizes that fixed population: there is only one possible full-population sample. At smaller sampling fractions, the difference between designs is less pronounced, though the exact amount depends on the population and the statistic.

A clear analysis states the design before describing a sampling distribution. Ask whether each selected unit can appear again, whether draws can be treated as independent, and whether the sample is a substantial fraction of the population. Those answers guide the variability calculation and its interpretation.

Common Mistakes

Calling draws without replacement independent because each unit is selected at random. Random selection does not guarantee independence. The first selection changes which units remain, so conditional probabilities for later draws can change.
Assuming the same statistic variability under both designs. The expected sample proportion or mean may be the same under both designs, but dependence changes the variability. Check how the sample was drawn before choosing a variance calculation.
Confusing units with values. Without replacement, the same unit cannot recur, but two different units may have equal measured values. For example, two patients can have the same recorded measurement.
Claiming sampling without replacement always makes every outcome less variable in every sense. The covariance result concerns the specified statistic under the stated random-sampling model. Describe the statistic and design rather than making a blanket claim about individual samples.

Key takeaway: With replacement, draws from a fixed population are independent. Without replacement, draws are dependent because the population composition changes after each selection. For common sample means and proportions, this dependence is negative and reduces sampling variability relative to independent draws.

Check Your Understanding

Use the sampling design, not just the population description, to reason through each question.

A box contains 6 red and 4 blue tokens. If a token is drawn, recorded, and returned before the next draw, what is the probability of red on the second draw given that the first draw was red? How does the answer change if the token is not returned?
Explain why two draws without replacement are dependent even when every unit has an equal chance of being selected first.
A sample of people is drawn without replacement from a fixed clinic roster. Can two sampled people have the same recorded blood pressure? Explain.
For two draws from a population with nonzero variance, why does the negative covariance without replacement tend to lower the variance of their sample mean?
Two designs produce the same expected sample proportion. Does that imply the sample proportions have the same variance? Explain.