Sampling Bias Versus Sampling Variability

Two Reasons a Sample Estimate Can Differ

Suppose two surveys estimate the proportion of students at a university who commute by bus. Their results differ. That difference could arise because each survey happened to select different students, or because the selection process repeatedly favors students who use the bus. The first is sampling variability; the second is sampling bias. Distinguishing them matters because collecting a larger sample can reduce random fluctuation, but it cannot necessarily correct a selection process that systematically misses part of the population.

A useful way to make the distinction is to imagine repeating the same sampling procedure many times, always targeting the same population and calculating the same estimate. The estimates would not usually be identical. Their spread describes the procedure’s sampling variability, while their long-run center indicates whether the procedure tends to estimate the population value accurately.

These definitions concern a procedure, not just one observed sample. An estimate can be far from the population value even when its procedure is unbiased; that difference may be a random fluctuation. Conversely, a biased procedure can sometimes produce an estimate close to the truth by chance. One result alone does not reveal which explanation applies.

Repeated Samples and the Long-Run Center

For a fixed population and procedure, imagine drawing repeated samples and calculating an estimate \(T\) each time. The resulting collection of possible estimates is the sampling distribution of \(T\). Its expected value \(E(T)\) is the long-run average estimate. If the target parameter is \(\theta\), the bias of \(T\) is

A positive bias means the procedure tends to estimate too high; a negative bias means it tends to estimate too low. If the bias is zero, individual estimates still vary unless the procedure always returns the same value. The standard deviation of the sampling distribution measures that variation. When this standard deviation is called the standard error, it refers to the standard deviation of a statistic’s sampling distribution under the stated model and assumptions, not to the standard deviation of individual observations.

For example, under simple random sampling without replacement from a finite population, every group of \(n\) distinct population members has the same chance of selection. If the population contains \(N\) members and a proportion \(p\) have a specified characteristic, the sample proportion \(\hat{p}\) has expected value \(p\). Its exact variance is

This result assumes a fixed population, a binary characteristic measured without error, and a simple random sample without replacement. The factor \((N-n)/(N-1)\) accounts for sampling a substantial fraction of the population: as the sample approaches the full population, there is less uncertainty from which members were selected. This formula describes variability around \(p\) for that probability sampling design. It should not be applied as if a convenience sample were a simple random sample.

How an Unrepresentative Process Shifts the Center

Bias arises when the sampling process systematically gives some parts of the target population less chance of representation than others, or otherwise selects a group that differs in a relevant way. The defining issue is not simply that the sample looks unusual. It is that the procedure’s long-run estimates are displaced from the target parameter.

For a nonprobability process, such as a convenience sample, there may be no known random mechanism from which to calculate a design-based expectation or standard error. We can still describe how a repeated, clearly specified process would behave, but we should not claim that a probability-sampling formula applies. The example below uses random sampling inside a restricted group so that the variation can be calculated separately from the shift caused by restricting who can be selected.

Why a Larger Sample Does Not Automatically Fix Bias

Increasing sample size often reduces sampling variability under a probability design, because the estimate averages information from more sampled units. But a larger sample from the same restricted group can estimate that group’s characteristic more precisely while remaining unrepresentative of the target population. More observations do not compensate for systematically excluding people who differ from those included.

A Useful Summary: Bias and Variability Together

Bias and variability describe different features of an estimator. Bias describes how far its long-run center is from the target. Variance describes how spread out its repeated estimates are around their own long-run center. A procedure can have little bias but high variability, or substantial bias but low variability. In practice, both can matter because an estimate can be inaccurate either by fluctuating widely or by being consistently shifted.

For any estimator \(T\) with finite variance and target \(\theta\), its mean squared error is the expected squared distance from the target. It combines variance and squared bias:

To verify the identity, add and subtract \(E(T)\) inside the squared difference:

The middle term is zero because \(E[T-E(T)]=0\). This decomposition clarifies why a narrow spread alone is not enough: a tightly clustered set of estimates can still be centered far from the target.

Common Mistake: Treating One Bad Estimate as Proof of Bias

Wrong reasoning: “This sample estimate differs from the population value, so the sampling method must be biased.” This confuses a single outcome with the long-run behavior of the procedure. Even an unbiased random sampling design will usually produce estimates that are not exactly equal to the population parameter.

Correction: Ask what would happen if the same procedure were repeated many times. Random differences among those results are sampling variability. A consistent long-run displacement from the target is bias. Also ask who could be selected and who might be systematically missed; a large sample size alone does not answer that question.

In evaluating a survey, it is therefore useful to describe both the selection process and the uncertainty it creates. A probability sample may have sampling variability even when it is unbiased for the intended parameter. A consistently unrepresentative process may have both variability and bias. Response rates or sample size alone do not establish that the sample’s selection process represents the target population.