More Responses Do Not Automatically Mean a Better Estimate
In “Unusual Sample Means and the Z-Score of x-bar,” we measured how far a sample mean is from the center of its sampling distribution. That comparison depends on knowing where the sampling distribution is centered. A larger sample can make sample means less variable, but it cannot guarantee that the sampling distribution is centered at the population mean. The way the sample is selected matters.
Recall from “Mean of the Sampling Distribution of x-bar” that, for random samples of a fixed size selected by simple random sampling, the mean of the sampling distribution of \(\bar{x}\) equals the population mean \(\mu\). That property is called unbiasedness. A poor sampling method can instead favor certain individuals or responses. In that case, the sample mean may tend to miss \(\mu\), even when the sample is large.
Keep three ideas separate. Bias describes a systematic shift in the center of estimates away from the target. Variability describes how much estimates differ from one sample to another. Accuracy describes how close an estimate is to the target. A method may produce tightly clustered estimates that are consistently off target: low variability does not ensure accuracy.
What a Larger Sample Changes
For independent observations from a population with standard deviation \(\sigma\), the standard deviation of the sampling distribution of \(\bar{x}\) is \(\sigma/\sqrt{n}\). As explained in “How Sample Size Affects Variability of x-bar,” increasing \(n\) makes this spread smaller, assuming the sampling method and other relevant conditions stay appropriate. The standard deviation describes variability; it does not tell us whether the center is correct.
For an unbiased method, the sampling distribution is centered at \(\mu\), so reducing its spread generally makes estimates cluster more closely around the target. But if the method systematically favors part of the population, the sampling distribution may be centered somewhere other than \(\mu\). A larger sample can make estimates cluster more tightly around that wrong center.
Why Voluntary-Response Samples Can Be Biased
In a voluntary-response sample, people choose whether to participate. Those who respond may differ in relevant ways from those who do not. For example, people with especially strong opinions or experiences may be more likely to complete a survey. If participation is related to the quantity being measured, the mean among respondents can differ systematically from the population mean.
The number of responses does not settle this issue. Collecting more voluntary responses may describe the group of people who volunteer with increasing precision, but that group need not represent the target population. The concern is not simply that a small sample happened to be unrepresentative. The selection process itself can repeatedly favor the same kinds of people.
A voluntary-response sample is different from a random sample. Random sampling gives members of the target population a chance to be selected through a chance process. In a voluntary-response sample, people decide whether to enter the sample. A very large response count does not make that self-selection random.
Worked Examples: Sample Size and Bias
Worked Example: More Precision With a Random Sample
Suppose the commute times of residents in a town have population mean \(\mu=20\) minutes and population standard deviation \(\sigma=10\) minutes. Compare the sampling distributions of the mean commute time for simple random samples of \(n=25\) and \(n=100\). Assume the town has 2,000 residents.
State. We want to compare the centers and variability of the sample means for two sample sizes, using the same random sampling method.
Plan. Because these are simple random samples, the sampling distribution of \(\bar{x}\) is centered at \(\mu=20\) minutes. Check independence using the 10% condition: \(25<0.10(2000)=200\) and \(100<200\). Thus, for both sample sizes, we can use \(\sigma/\sqrt{n}\) for the standard deviation of the sample mean. No Normal model is needed to compare the center and standard deviation.
Do. For \(n=25\), the standard deviation of the sample mean is
For \(n=100\), it is
The center remains 20 minutes for both sample sizes, while the standard deviation of the sample mean falls from 2 minutes to 1 minute. As a check, increasing the sample size by a factor of 4 divides the standard deviation by \(\sqrt{4}=2\).
Conclude. With the random sampling method, both sampling distributions are centered at the population mean, and the larger sample has less variability. This is the setting in which a larger sample helps sample means tend to be closer to the target. The calculation does not guarantee that every particular sample of 100 residents will have a mean closer to 20 than every sample of 25.
Worked Example: A Voluntary-Response Commute Poll
Consider a hypothetical town in which 80% of residents have a 15-minute commute and 20% have a 40-minute commute. The population mean commute time is
Suppose residents are invited to answer an online commute poll. For this invented example, assume that 10% of residents with 15-minute commutes volunteer, while 60% of residents with 40-minute commutes volunteer. What mean should we expect among the volunteers?
State. We are comparing the mean commute time among volunteers with the population mean of 20 minutes.
Plan. This is a voluntary-response sample, not a simple random sample. To find the expected mean among volunteers, calculate the proportion of all residents who volunteer from each commute group, then find each group’s share of the volunteer pool. This calculation describes the stated hypothetical response process; it does not make the volunteers a random sample of all residents.
Do. Of all residents, the proportion who are short-commute volunteers is \(0.80(0.10)=0.08\). The proportion who are long-commute volunteers is \(0.20(0.60)=0.12\). Thus, the total proportion volunteering is \(0.08+0.12=0.20\). Among volunteers, the short-commute share is \(0.08/0.20=0.40\), and the long-commute share is \(0.12/0.20=0.60\).
The expected volunteer mean is therefore
This is 10 minutes above the population mean: \(30-20=10\). If we calculate the standard deviation among volunteers, their mean is 30 and their variance is \(0.40(15-30)^2+0.60(40-30)^2=90+60=150\), so their standard deviation is \(\sqrt{150}\approx12.25\) minutes.
Suppose many volunteers are available and the poll takes a random sample of \(n=100\) from them. Treating those selections as independent, the standard deviation of the sample mean around the volunteer mean is approximately \(12.25/\sqrt{100}=1.225\) minutes. The center remains 30 minutes, not the population mean of 20 minutes.
Conclude. Under this illustrative response process, the mean among volunteers is systematically 10 minutes higher than the town mean. Taking more responses from the same volunteer pool could make the poll mean less variable around 30 minutes, but it would not make the method unbiased for the town’s mean commute time. The response-rate assumptions are invented to show how self-selection can shift the center.
Worked Example: A Tighter Cluster Can Still Miss the Target
A school wants to estimate the population mean number of hours students spend on homework per week. Suppose the true mean is 6 hours. A random sampling method produces sampling distributions centered at 6 hours, with standard deviation 0.8 hours for \(n=25\) and 0.4 hours for \(n=100\). Separately, suppose a voluntary online poll tends to attract students with heavier workloads, so its sampling distribution is centered at 8 hours, with standard deviation 1.2 hours for \(n=25\) and 0.6 hours for \(n=100\). These values are hypothetical. Which method and sample size give the clearest picture of the difference between variability and bias?
State. We will compare each method’s center with the target of 6 hours, and compare the stated variability for the two sample sizes.
Plan. The random method is centered at the population mean, so its bias is 0 in this model. The voluntary poll is centered 2 hours above the population mean, so it has a positive bias of \(8-6=2\) hours. For each method, compare the given standard deviations to assess variability. These statements use the hypothetical sampling distributions in the question; a small standard deviation does not establish that a method is unbiased.
Do. Increasing the random sample from 25 to 100 reduces its standard deviation from 0.8 to 0.4 hours. Its center stays at 6 hours. Increasing the voluntary-response sample from 25 to 100 reduces its standard deviation from 1.2 to 0.6 hours, but its center stays at 8 hours. For the larger voluntary sample, the center is still \(8-6=2\) hours above the target, even though the standard deviation is only 0.6 hour.
Conclude. In this example, the larger voluntary sample produces less variable estimates than the smaller voluntary sample, but those estimates still cluster around 8 rather than the true mean of 6 hours. The random sample of 100 has both a center at the target and less variability than the random sample of 25. The example shows why sample size and sampling method answer different questions: sample size affects variability, while the selection process can determine whether the center is biased.
Common Mistakes and AP Exam Tip
- “A bigger sample eliminates bias.” It does not. A larger \(n\) can reduce variability, but a systematic selection problem can continue to shift the center.
- “A large sample must represent the population.” The response count alone does not establish representativeness. Describe how people were selected, and consider whether the selection process favors some values or groups.
- Confusing bias with variability. Bias is about the center of the sampling distribution relative to \(\mu\). Variability is about the spread around that center. A method can have low variability and still be biased.
- Claiming every large sample is closer to the truth. A smaller standard deviation means sample means are less spread out over repeated samples. It does not guarantee that a particular observed sample mean is closer to \(\mu\).
- Calling a voluntary-response poll a random sample. People choose to respond, so the sample is subject to self-selection. More responses do not change the design into a random sample.
For full-credit communication, identify the population parameter, describe whether the sampling method is random or self-selected, and separate the method’s center from its variability. If the method is biased, explain which way estimates may tend to differ from the population mean and why. Avoid saying only that the sample is “too small” when the central problem is how responses were obtained.
Check Your Understanding
Use the distinction between bias and variability to answer these questions.
- A random sample of size 36 has a sampling distribution centered at \(\mu\) with standard deviation 3. If the sample size increases to 144 and the other conditions remain appropriate, what happens to the standard deviation of the sample mean?
- In a voluntary-response poll, people with especially long commutes are more likely to reply. Explain why collecting many more replies may reduce variability without removing bias.
- A method produces estimates centered at 12, while the population mean is 10. In your own words, describe the method’s bias and explain why smaller variability would not correct it.
- Why does a large number of responses not, by itself, justify calling a survey a random sample?
- A student says, “The larger sample is always more accurate.” Give a more careful statement about what a larger sample can do and what it cannot guarantee.