Sources of Variability in Collected Data

Why Collected Data Differ

In the previous tutorial, Choosing Between a Survey, Observational Study, and Experiment, we focused on matching a data collection method to a research question. Whatever method researchers choose, the resulting data can vary. Two reasonable samples may give different estimates, a measuring tool may not record a quantity exactly, and the people or objects being studied may genuinely differ.

These sources of variability matter because they affect what a data set can tell us. If researchers see a difference between two groups, for example, they should consider whether it reflects genuine differences, how the data were measured, or which observations happened to be selected. A numerical result by itself does not reveal the source.

These sources can occur together. A survey statistic can vary because a different random sample was selected, because respondents do not report an experience exactly, and because people truly have different experiences. Recognizing the sources helps us describe the limits of the data without assuming every difference is a mistake.

Sampling Variability: Different Samples, Different Statistics

A sample is only part of a population. If researchers take another sample from the same population, they will usually select at least some different individuals. The resulting statistic—such as a sample mean or sample proportion—may therefore differ. This variation among sample statistics is sampling variability.

Random selection helps prevent a researcher from deliberately or accidentally favoring certain members of the population. It does not guarantee that the sample will match the population perfectly. A random sample can, by chance, include more members of one kind than another. As discussed in Generalizing Results to a Population, random selection can support generalizing to the population from which the sample was selected; it does not make a sample statistic identical to the population parameter.

A larger sample will generally show less sampling variability than a smaller sample selected in the same way. More observations provide more information about the population. But increasing the sample size does not fix a biased selection method, and it does not automatically fix inaccurate measurements. A very large sample gathered from only volunteers, for instance, can still fail to represent the population of interest.

Measurement Error: The Recorded Value Is Not Always Exact

Measurement error occurs when the recorded value differs from the quantity the study intends to measure. A scale might be miscalibrated, a stopwatch might be started late, a survey respondent might misremember an event, or two observers might apply a category definition differently. Errors can be small or substantial, and they can occur in numerical or categorical data.

Some measurement error is random: readings vary unpredictably, sometimes above and sometimes below the intended value. Other error is systematic: a consistent problem pushes readings in one direction. A thermometer that always reads 0.4 degrees Celsius too high has a systematic offset. Repeating the measurement may reveal inconsistent readings, but repeating it with the same miscalibrated thermometer does not remove the offset.

When a trusted reference value is available, a measurement’s error can be calculated as recorded value minus reference value. In many studies, however, the exact value is not known. Researchers may be able to reduce potential error with clear variable definitions, suitable tools, consistent procedures, training, and careful recording. These steps improve measurement; they do not promise perfect accuracy.

Natural Variation: Real Differences in the World

Not every difference among data values is an error. Observational units may genuinely differ. For example, plants grown in the same garden can reach different heights because of biological differences and small differences in their growing conditions. The heights vary because the plants vary; a ruler can measure those real differences accurately.

Natural variation can also occur within one unit over time. A person’s pulse may change from one moment to another, even when the measurement procedure is consistent. A question about a person’s pulse therefore needs to specify when or under what conditions it is measured. Otherwise, readings taken at different times may reflect genuine changes in the person’s pulse as well as measurement error.

In practice, “natural variation” does not always mean that researchers know the exact cause of each difference. It means the values may reflect real differences in the units or real changes over time, rather than a flaw in the selection or measurement process. As with measurement error, careful definitions and consistent procedures help researchers interpret what the data represent.

Separating Sources and Improving a Data Collection Plan

A useful way to think about a data collection plan is to ask three questions. First, who or what will be selected? That points to sampling variability and possible selection problems. Second, how will each variable be measured? That points to measurement error. Third, how much might the observational units genuinely differ, or the variable change over time? That points to natural variation.

These are related but not interchangeable. A biased sample is not simply another name for sampling variability. Sampling variability describes how statistics differ across samples selected by the same method; bias is a systematic problem that makes the method tend to miss or misrepresent part of the population. Similarly, a wide spread in the data does not prove poor measurement. It may reflect real differences among units.

Consider an observational study of the time residents spend waiting for a bus. A random sample of residents may produce different average waiting times on different occasions. That is sampling variability. Residents may start their timers late or remember a past wait inaccurately; those are possible measurement errors. Buses may genuinely arrive at different intervals and residents may arrive at different points in the schedule; those differences contribute natural variation. A data collection plan can address each source differently: choose an appropriate sample, define and measure waiting time consistently, and record relevant context such as time of day.

The method should fit the question, as explained in the previous tutorial. A survey may be appropriate for a reportable experience, while direct timing may be better for an observed wait. Neither choice guarantees that all error disappears. Researchers should explain what they measured and recognize limits that remain.

Common Mistakes and AP Exam Tips

• Calling every difference “measurement error.” Real differences among people, objects, or times can produce different values. For full credit, say whether the difference could reflect natural variation, measurement error, or both, and connect the claim to the context.
• Assuming random selection produces the same statistic every time. Random selection avoids choosing only preferred cases, but different random samples can still produce different statistics. State that this difference is sampling variability.
• Assuming a larger sample fixes every problem. A larger sample generally reduces sampling variability, but it cannot automatically repair a biased sample or a systematic measuring-tool problem. Identify which source the proposed change addresses.
• Confusing a consistent offset with random fluctuation. Readings that repeatedly fall above a trusted reference suggest a possible systematic measurement error. Readings that vary around a reference may indicate random measurement variation, though the process should still be checked.
• Treating observed spread as proof of a flawed study. Spread can be an expected feature of real data. Describe what varies and avoid claiming a cause unless the data collection process provides evidence for it.

A strong AP response names the source and explains its role in context. For example: “Different random samples of residents could produce different sample mean waiting times, so the estimate has sampling variability. Inaccurate timing could add measurement error, while actual differences in bus arrival intervals could create natural variation.” That wording distinguishes the sources instead of treating all variability as a single problem.