The Repeated-Sampling Meaning of P-Values

What Repeated Sampling Adds to a P-Value

A p-value is calculated from a probability model, but its meaning becomes clearer when we imagine repeating the same study many times. Each repetition uses the same sampling plan and analysis, while the data vary from sample to sample. The p-value describes how often the resulting test statistic would be at least as extreme as the statistic observed, if the null model were true.

This is a statement about a collection of hypothetical repetitions under a specified model. It is not a prediction that a particular set of future studies must contain exactly that proportion of extreme results. Nor does the calculation by itself tell us whether the null model is true. The key is to keep the condition “under the null model” attached to the repeated-sampling interpretation.

Definition: For a specified null model and a chosen rule defining which test-statistic outcomes count as at least as extreme as the observed statistic, the p-value is the probability, under that null model, of obtaining an outcome in that extreme set. Equivalently, it is the long-run proportion of repetitions of the same sampling procedure that would produce an outcome in that set, when the null model is used to generate the data.

The procedure and extremeness rule matter. For a one-sided alternative, “at least as extreme” means at least as far in the direction specified by that alternative. For a two-sided alternative, it includes outcomes in either direction that meet the chosen two-sided extremeness rule. The repeated samples must be analyzed using the same rule each time; changing the rule after seeing the result would describe a different calculation.

Turning a Probability Into a Long-Run Proportion

Suppose a single repetition has probability \(p\) of producing a test statistic in the defined extreme set. In \(m\) independent repetitions, count how many meet that condition. Each repetition either meets it or does not, so the count is a sum of indicator variables, each equal to 1 for an extreme result and 0 otherwise.

Fix the model and procedure.
Specify the null model, sample size, data-collection plan, test statistic, and rule for deciding which results are at least as extreme as the observed result.

Repeat the whole procedure.
Generate a new sample under the same null model each time, then calculate the statistic using the same analysis.

Count qualifying results.
Record how many of the repeated statistics fall in the defined extreme set, and divide that count by the number of repetitions.

Here is a brief justification for the long-run interpretation. Let \(I_j=1\) if repetition \(j\) produces a statistic in the extreme set, and \(I_j=0\) otherwise. Under independent repetitions from the same null model, each \(I_j\) has probability \(p\) of being 1. The observed fraction is the average of these indicators.

\hat{p}_{\text{repeat}}=\frac{1}{m}\sum_{j=1}^{m} I_j,\qquad E(\hat{p}_{\text{repeat}})=p,\qquad SE(\hat{p}_{\text{repeat}})=\sqrt{\frac{p(1-p)}{m}}.

The expected fraction equals the single-repetition probability \(p\). Its standard error decreases as the number \(m\) of repetitions grows, so the observed fraction tends to be close to \(p\) over many repetitions. It need not equal \(p\) exactly in a finite batch. This is why a p-value is appropriately described as a long-run proportion, rather than as a guaranteed percentage of any particular batch.

Worked Examples

Worked Example: A Two-Sided Test of a Mean

Suppose a hypothetical process produces independent measurements from a normal population with known standard deviation \(\sigma=12\) units. A random sample of \(n=36\) measurements has \(\bar{x}=104.2\) units. Test \(H_0:\mu=100\) against the two-sided alternative \(H_a:\mu\ne100\). Interpret the p-value through repeated sampling.

Model and conditions. For this calculation, assume the measurements are an independent random sample from the stated normal population, and that the known value \(\sigma=12\) units is correct under the null model. Under \(H_0\), the sample mean has mean 100 units and standard error \(\sigma/\sqrt{n}\). These assumptions give the stated normal reference model for the test statistic.

Calculate the statistic. The standard error is \(12/\sqrt{36}=2\) units, so the standardized statistic is

z=\frac{\bar{x}-\mu_0}{\sigma/\sqrt{n}} =\frac{104.2-100}{12/\sqrt{36}} =\frac{4.2}{2}=2.10.

For the two-sided alternative, outcomes at least as extreme are those with \(|Z|\ge2.10\). By symmetry of the standard normal distribution, the p-value is twice its upper-tail probability:

p=P(|Z|\ge2.10) =2P(Z\ge2.10) =2(0.01786442) \approx0.03572884 \approx0.03573.

Thus, under the null model, about 3.573% of repetitions of this same sampling procedure would produce a sample mean at least 4.2 units above 100 or at least 4.2 units below 100. For example, among 10,000 repetitions, the expected number meeting that two-sided criterion is \(10{,}000(0.03572884)\approx357.3\), or about 357. The actual count in a particular batch of 10,000 could be higher or lower.

Worked Example: An Exact P-Value for Repeated Coin Tosses

Imagine a hypothetical test of whether a coin is biased toward heads. Under \(H_0\), each toss is independent and has probability \(0.5\) of landing heads. Ten tosses produce 9 heads, and the alternative is \(H_a:p>0.5\), where \(p\) is the probability of heads.

Model and extremeness rule. Under the null model, the number of heads \(X\) in ten tosses has a binomial distribution with 10 trials and success probability \(0.5\). Because the alternative is toward more heads, results at least as extreme as the observed result are \(X\ge9\).

Calculate the p-value. The two qualifying outcomes are 9 heads and 10 heads:

p=P(X\ge9) =P(X=9)+P(X=10) =\binom{10}{9}(0.5)^{10}+\binom{10}{10}(0.5)^{10} =\frac{10+1}{1024} \approx0.01074.

The repeated-sampling interpretation is that, if the coin really has a 0.5 chance of heads on each independent toss, about 1.074% of repetitions of ten tosses would produce 9 or 10 heads. Across 1,000 repetitions, the expected number meeting this criterion is \(1{,}000(11/1024)\approx10.74\), or about 11. This does not mean every batch of 1,000 repetitions will contain exactly 11 such outcomes.

Worked Example: A One-Sided Mean Test

In another hypothetical setting, a random sample of \(n=25\) independent measurements comes from a normal population with known \(\sigma=10\) units. The observed sample mean is \(\bar{x}=52.8\) units. Consider \(H_0:\mu=50\) against \(H_a:\mu>50\).

Model and conditions. Assume the measurements are independent, randomly sampled from the stated normal population, and have the known standard deviation of 10 units under the null model. The standard error of the sample mean is \(10/\sqrt{25}=2\) units.

Calculate the statistic and p-value.

z=\frac{52.8-50}{10/\sqrt{25}} =\frac{2.8}{2}=1.40,\qquad p=P(Z\ge1.40)\approx0.08076.

Because the alternative is toward larger means, the extreme set is the upper tail only. Under the null model, about 8.076% of repeated samples of 25 measurements would have a sample mean at least 52.8 units. Out of 10,000 repetitions, the expected count is \(10{,}000(0.0807567)\approx808\). The repeated-sampling statement is conditional on the specified null model and assumptions; it describes a frequency across repetitions, not a probability assigned to the hypothesis itself.

What Must Stay the Same Across Repetitions?

A meaningful repeated-sampling interpretation keeps the design and analysis fixed. In particular, the imagined repetitions use the same sample size, the same way of selecting or generating observations, the same null model, the same test statistic, and the same rule for defining extremeness. The data change; the procedure does not.

For a study involving people, this does not usually mean measuring the same participants again and again. It means imagining new samples drawn according to the same study design from the population represented by the null model. The conditions used to justify the probability calculation must also remain in force in those repetitions.

The p-value is a probability for the statistic’s extreme set, not necessarily for the observed statistic alone. In a continuous model, an individual exact value has probability zero, so the calculation concerns a tail or another region of possible values. In a discrete model, the extreme set may include several possible values, as in the coin-toss example.

Common Mistake: Treating the Long-Run Rate as a Promise

A tempting but incorrect statement is: “A p-value of 0.04 means exactly 4 of every 100 repeated studies will be at least as extreme.” The calculation gives a probability of 0.04 for any one repetition under the null model. In a batch of 100 independent repetitions, the expected number is 4, but the realized count can differ.

For instance, the standard deviation of the proportion in a batch of 100 repetitions when \(p=0.04\) is

\sqrt{\frac{0.04(1-0.04)}{100}} \approx0.0196.

That is about 1.96 percentage points, so visible variation around 4% is unsurprising in a batch of only 100. More repetitions generally make the observed proportion more stable, but no finite batch is required to match the p-value exactly.

A related wording pitfall is to omit the null-model condition. “About 4% of studies produce results this extreme” is incomplete: the precise statement is “About 4% of repetitions would produce results this extreme if the null model and the stated sampling assumptions were correct.” The condition is central to what the p-value means.

Key takeaway: A p-value is the null-model probability of the defined extreme set of test-statistic outcomes. In repeated sampling, it is the long-run proportion of repetitions that would fall in that set when the same design and analysis are used.

Check Your Understanding

Answer each question using the repeated-sampling interpretation.

A test has a p-value of 0.03. State what this means in terms of repeated samples under the null model, without claiming that exactly 3 out of every 100 repetitions must be extreme.
In a two-sided test, why can results on both sides of the null value count as at least as extreme as the observed result?
A student says, “A p-value of 0.08 means 8% of repeated samples will produce exactly the observed statistic.” What two parts of this statement need correction?
Name three features of a study or analysis that should remain fixed when describing repeated samples for a p-value.
If 10,000 repetitions are imagined for a test with p-value 0.02, what is the expected number of repetitions in the defined extreme set? Does that number have to be observed exactly?