Worked Comparisons of Sampling With and Without Replacement

Two Designs, Two Probability Models

A sample of fixed size can be drawn in two fundamentally different ways. With replacement, each draw returns the selected unit to the population, so the draws are independent and the same unit may appear more than once. In simple random sampling without replacement (SRSWOR), every subset of the specified size is equally likely, and a unit cannot be selected twice. The distinction affects not only the probabilities of particular outcomes, but also the support, variance, and dependence of statistics calculated from the sample.

For a finite population of \(N\) units with fixed values \(x_1,\ldots,x_N\), let \(Y_1,\ldots,Y_n\) denote the values drawn in order when sampling with replacement. Each \(Y_j\) is uniform over the \(N\) population units. Under SRSWOR, let \(I_i\) equal 1 if unit \(i\) is included in the sample and 0 otherwise. The sample mean in either design is the mean of the \(n\) observed values, though repeated values are possible only with replacement.

The phrase “with replacement” refers to units, not necessarily to their recorded values. Two distinct units may have the same value, and sampling the same value twice does not by itself prove that the same unit was drawn twice. Conversely, a particular unit can be selected again only under replacement.

Count Distributions and Their Supports

Suppose a population has categories \(1,\ldots,k\), and a with-replacement draw falls in category \(j\) with probability \(p_j\), where \(\sum_{j=1}^k p_j=1\). If \(C_j\) is the number of draws in category \(j\), then for nonnegative integers \(c_1,\ldots,c_k\) summing to \(n\),

The factor \(n!/(c_1!\cdots c_k!)\) counts the ordered sequences that produce those counts. If the population has \(K\) units in a designated category and \(N-K\) outside it, then a with-replacement sample has success probability \(p=K/N\) on every draw. Its success count is binomial. Under SRSWOR, the count \(X\) instead has

The numerator counts subsets containing \(x\) designated units and \(n-x\) other units; the denominator counts all samples. These support limits matter: the count cannot exceed the number of designated units in the population or the sample size, and it cannot require more other units than exist.

Means, Variances, and Dependence

For the two count models above, the expected success count is \(np\) in both designs. Their variances differ: with replacement, the count is a sum of independent Bernoulli variables; without replacement, the draws are negatively dependent. For a population of \(N\) units with \(K\) successes, the formulas are

The factor \((N-n)/(N-1)\) is the finite-population correction for the count variance. It is zero when \(n=N\), as a full census leaves no uncertainty about the count. For general finite-population values, the corresponding sample-mean variance comparison follows from the dependence among inclusion indicators.

Proof. Set \(z_i=x_i-\bar{x}_N\), so \(\sum_i z_i=0\), and write \(I_i\) for the inclusion indicator under SRSWOR. Uniform selection gives \(P(I_i=1)=n/N\). For distinct \(i,j\), counting samples containing both units gives \(P(I_i=1,I_j=1)=n(n-1)/(N(N-1))\). Thus

The centered sample mean is \(n^{-1}\sum_i I_i z_i\). Since \(\sum_{i\ne j}z_i z_j=-\sum_i z_i^2\), its variance is

For sampling with replacement, the \(Y_j\) are independent and each has variance \(N^{-1}\sum_i z_i^2=(N-1)S^2/N\). Therefore \(\operatorname{Var}(\bar{Y})=\operatorname{Var}(Y_1)/n=((N-1)/N)S^2/n\). When \(S^2>0\), division gives \(\operatorname{Var}_{\mathrm{WOR}}(\bar{X})/\operatorname{Var}_{\mathrm{WR}}(\bar{Y})=(N-n)/(N-1)\). When \(S^2=0\), all population values are equal, so each sample mean is constant and both variances vanish. \(\square\)

Multinomial Dependence and a Unit-Level Example

With replacement, category counts are not independent of one another: their sum is fixed at \(n\). For two distinct categories with probabilities \(p_a,p_b\), the covariance of their counts is \(-np_ap_b\). One way to see this is to write each count as a sum of draw-level indicators. On a single draw, the indicators for two distinct categories cannot both equal 1, so their covariance is \(0-p_ap_b=-p_ap_b\); indicators from different draws are independent. Summing the \(n\) same-draw covariances gives the stated result.

Under SRSWOR, inclusion indicators for distinct units also have negative covariance, as shown in the variance proof. But counts of categories now depend on the finite numbers of units of each kind. It is therefore important to specify whether a model concerns independent categorical draws or selection of distinct population units.

Common Pitfall and Why the Comparison Matters

• Assuming replacement changes only the variance. It can also change the support, as the total of 8 in the preceding example demonstrates.
• Using a binomial model for distinct sampled units. The binomial model assumes independent draws with a fixed success probability. For a fixed finite population sampled without replacement, use the hypergeometric law.
• Dividing variances when the population is constant. If \(S^2=0\), both sample-mean variances are zero, so their ratio is \(0/0\), not a finite-population correction. State the ratio only when \(S^2>0\).
• Confusing repeated values with repeated units. Equal recorded values may belong to different units; the design specifies whether a unit can be selected again.

These distinctions matter whenever a probability calculation is meant to represent an actual sampling plan. The multinomial and binomial models describe independent draws; hypergeometric probabilities and the finite-population correction describe selection of distinct units. Matching the model to the design is necessary for correct supports, probabilities, and uncertainty calculations.