Sample Size and Precision

More Data Can Mean More Precision—but Not Less Bias

In Identifying the Source of Bias in a Scenario, you learned to trace how people are selected, who may be missing, and whether answers could be influenced. This tutorial connects those ideas to sample size. A larger random sample generally gives a statistic that varies less from sample to sample, making it more precise. But increasing the number of observations does not repair a biased selection process.

A sample statistic is calculated from the selected sample, while a parameter describes the population, as explained in Parameters and Statistics. Because a different random sample may produce a different statistic, one sample result is not guaranteed to match the population value. Here, precision refers to how little a statistic tends to vary across samples collected in the same way. It is not a promise that one particular statistic is close to the parameter.

Imagine repeatedly selecting random samples from the same population and calculating the same statistic each time. The statistics would not all be identical. With a larger sample, they generally cluster more closely together. This is the connection between sample size and sampling variability, introduced in Sources of Variability in Collected Data. The next tutorial will examine variability across repeated samples in more detail.

“Generally” matters. A larger sample does not guarantee that its statistic will be closer to the population value than the statistic from a smaller sample. Random variation can make a particular large-sample statistic farther away. The benefit of a larger sample is about the typical variation across samples, not a certainty about every result.

What a Larger Sample Changes

Suppose a town wants to estimate the proportion of residents who support a park renovation. If the town selects a random sample, the sample proportion \(\hat{p}\) may differ from the population proportion \(p\). A larger random sample usually gives more information about the population and reduces the amount that \(\hat{p}\) tends to fluctuate across repeated samples.

For a sample proportion, the possible values also become more finely spaced as the sample size increases. If \(n=20\), each additional person who answers “yes” changes \(\hat{p}\) by \(1/20=0.05\), or 5 percentage points. If \(n=100\), each additional “yes” changes it by \(1/100=0.01\), or 1 percentage point. Finer possible steps do not themselves guarantee accuracy, but they help explain why estimates from larger samples can be more precise.

The same broad idea applies to other statistics, such as a sample mean \(\bar{x}\): increasing the size of a random sample generally reduces the statistic’s sampling variability. The exact amount depends on the population and the sampling method. In this tutorial, focus on the direction of the effect rather than a formula for measuring that variability.

A fair comparison holds the rest of the sampling plan as constant as possible. Compare random samples from the same population, using the same selection process and measuring the same variable. If one study uses a different frame, a different question, or a different selection method, sample size alone cannot explain why its result differs.

Worked Example: Comparing Two Random Polls

Precision Does Not Guarantee a Close Result

A common misunderstanding is that a larger random sample must produce an estimate closer to the population value. Random selection does not guarantee this. A sample of 200 could, by chance, contain a higher or lower proportion of supporters than a sample of 50. The larger sample tends to vary less across repeated samples, but its result can still be on either side of the population value.

Another way to see the distinction is to consider the possible values of a sample proportion. With \(n=20\), the count of “yes” responses can only be a whole number, so \(\hat{p}\) changes in increments of \(1/20\). With \(n=100\), the increments are \(1/100\). For example, 7 “yes” responses out of 20 gives 35%, while 8 gives 40%. With 100 people, 35, 36, or 37 “yes” responses give 35%, 36%, or 37%. The larger sample allows more finely spaced proportions, but a finely spaced estimate can still be inaccurate.

Worked Example: More Possible Levels of Detail

Why a Bigger Sample Cannot Fix Bias

Sample size addresses sampling variability, not every weakness in data collection. If a sampling frame leaves out part of the population, selecting more people from that incomplete frame does not give the missing group a chance to be selected. If people with a particular opinion are more likely to answer, adding more responses through the same process may preserve that imbalance. These concerns connect to Undercoverage Bias, Nonresponse Bias, and Voluntary Response Bias.

A very large biased sample can produce a result that is stable but systematically different from the population value. In that case, the result may be precise in the sense that repeated use of the same flawed method produces similar statistics, yet still be inaccurate for the population of interest. Precision is not the same as freedom from bias.

Worked Example: A Large Sample From an Incomplete Frame

Choosing a Sample Size Is Only Part of Planning

When planning a study, first define the population and the variable, as in Defining the Population of Interest and Deciding Which Variables to Measure. Then choose a selection method that gives the intended population appropriate representation. Only after considering these design choices should you treat a larger sample as a way to reduce sampling variability.

A larger sample can also take more time, money, and staff effort. If the method has serious bias, collecting many more observations the same way may be a poor use of resources. Improving the sampling frame, reducing nonresponse, or changing how participants are recruited may matter more for the credibility of the result than simply increasing \(n\).

If the study is a census, it seeks information from everyone in the defined population, as discussed in Census Versus Sample. A census removes sampling variability for that population at that time, but it does not automatically remove nonresponse, response bias, measurement problems, or errors in defining the population. Collecting data from everyone is not a guarantee of perfect information.

Common Mistakes and AP Exam Tips

• Claiming that a larger sample is always closer. Say that larger random samples generally have less sampling variability. Do not claim that every larger sample must produce a more accurate statistic.
• Using “precision” to mean “unbiased.” Precision concerns how much statistics vary across samples. Bias concerns a systematic tendency to differ from the population value. A method can be consistent and still biased.
• Assuming sample size repairs undercoverage. A large random sample selected from a frame that omits part of the population still cannot include the omitted people. Identify and fix the frame problem.
• Ignoring the sampling method when comparing sample sizes. The general sample-size comparison assumes the same target population and a comparable random selection method. A larger convenience sample is not automatically better than a smaller random sample.
• Confusing one observed comparison with a general rule. If one larger sample is closer in an example, report that as what happened in those samples. Explain separately that the broader claim concerns typical variability across repeated random samples.
• Calling every large sample representative. A high response count does not show that respondents represent the target population. Consider selection, coverage, and response, as practiced in Identifying the Source of Bias in a Scenario.

A strong AP response names the sample size and method, distinguishes variability from bias, and limits its claim. For example: “Using a larger random sample from the same population generally reduces sampling variability, so the sample proportion tends to be more precise. It does not guarantee that this particular estimate is closer to the population proportion, and it does not correct undercoverage in the sampling frame.”