Tutorials › Introduction to Statistics › Interpreting Evidence and Effect Size

Home  /  Tutorials  /  Tests for Proportions

Tests for Proportions · Tutorial 635 of 996

Interpreting Evidence and Effect Size

Distinguish how strongly a one-proportion test challenges its null value from how large and practically important the estimated difference may be.

Intermediate 9 min read

What You'll Learn

  • Define the estimated difference between a sample proportion and its null value.
  • Interpret a one-proportion test’s p-value as evidence under the null model.
  • Calculate and contextualize percentage-point differences and z-test p-values.
  • Explain why sample size can change a p-value without changing the estimated difference.
  • Separate statistical evidence from practical importance.
  • Check the assumptions behind a one-proportion z test.

Two Questions, Not One

A one-proportion test can answer whether sample data provide evidence against a proposed population proportion. But that is not the same as answering whether the difference is large enough to matter. These are separate questions: How incompatible are the data with the null model? and How large is the estimated difference, and is it important in this setting?

The p-value addresses the first question. The difference between the observed sample proportion and the null proportion helps address the second. Keeping these quantities distinct is essential: a small difference can produce a small p-value in a very large sample, while a potentially meaningful difference can remain uncertain in a small sample.

Definition: In a one-proportion test, \(p_0\) is the population proportion specified by the null hypothesis \(H_0\), and \(\hat{p}\) is the observed sample proportion. The estimated difference from the null value is \(\hat{p}-p_0\). It is measured in proportion units, or equivalently can be expressed as a difference in percentage points. The test's p-value is the probability, calculated under the null model, of obtaining a test statistic at least as extreme as the observed statistic in the direction or directions specified by \(H_a\).

A difference of \(0.05\) is a difference of 5 percentage points. It is not automatically a 5% relative increase: relative change depends on the starting value. For example, an increase from 50% to 55% is 5 percentage points, or 10% relative to 50%. In a test, report the absolute difference clearly first; whether relative change is also useful depends on the question and context.

What the Test Statistic Connects

For a one-proportion z test, the statistic compares the observed difference with the amount of sampling variation expected under \(H_0\). If the null hypothesis is \(p=p_0\), the null standard error is \(\sqrt{p_0(1-p_0)/n}\), and the test statistic is:

$$ z=\frac{\hat{p}-p_0}{\sqrt{p_0(1-p_0)/n}} $$

The numerator is the observed difference; the denominator is its standard error under the null model. Thus the z statistic expresses the difference in null standard-error units. The p-value is then obtained from the standard normal distribution using the direction of \(H_a\): an upper-tail alternative uses the probability at or above the observed z, a lower-tail alternative uses the probability at or below it, and a two-sided alternative counts results at least as far from zero in either direction.

The usual z-test calculation relies on a suitable one-proportion model: each outcome is binary, observations are independent under the sampling or assignment design, and the sample is obtained in a way that supports inference to the population of interest. For the normal approximation under the null, the expected numbers of successes and failures, \(np_0\) and \(n(1-p_0)\), must be sufficiently large. These conditions support the calculation; they do not make the p-value a measure of practical importance.

Key result: For a fixed observed difference \(\hat{p}-p_0\), increasing \(n\) reduces the null standard error and can increase \(|z|\), producing a smaller p-value. The difference itself does not change just because the sample is larger. Consequently, a p-value alone does not tell us the size or practical importance of the difference.

Worked Examples

Worked Example: Same Difference, Different Evidence

Suppose two hypothetical independent random samples are used to assess whether a population success proportion differs from \(p_0=0.50\). In the first sample, 220 of 400 people have the specified outcome. In the second, 55 of 100 people have it. For each sample, test \(H_0:p=0.50\) against \(H_a:p\ne0.50\) using a one-proportion z test, and distinguish the evidence from the estimated difference.

Check the model conditions. The outcome is binary, and each sample is described as an independent random sample. Under the null, the first sample has \(np_0=400(0.50)=200\) expected successes and \(n(1-p_0)=200\) expected failures. The second has 50 expected successes and 50 expected failures. These counts support the normal approximation for both tests.

First sample. The observed proportion is \(\hat{p}=220/400=0.55\), so the estimated difference is \(0.55-0.50=0.05\), or 5 percentage points. Its null standard error and z statistic are:

$$ SE_0=\sqrt{\frac{0.50(1-0.50)}{400}}=0.025,\qquad z=\frac{0.55-0.50}{0.025}=2.00 $$

For the two-sided alternative, the p-value is \(2P(Z\ge2.00)\), approximately \(0.0455\). The data provide evidence against \(p=0.50\) at the 0.05 significance level. The estimated difference, however, is still 5 percentage points; the p-value does not say that the difference is 4.55% or that it is practically important.

Second sample. Here \(\hat{p}=55/100=0.55\), again giving an estimated difference of 5 percentage points. The null standard error and statistic are:

$$ SE_0=\sqrt{\frac{0.50(1-0.50)}{100}}=0.05,\qquad z=\frac{0.55-0.50}{0.05}=1.00 $$

The two-sided p-value is \(2P(Z\ge1.00)\), approximately \(0.3173\). This sample does not provide strong evidence against the null at the 0.05 level. Yet its observed difference is the same 5 percentage points as in the first sample. The different p-values arise because the sample sizes—and therefore the null standard errors—differ.

The first result is stronger evidence against the null according to the test, but neither p-value alone decides whether a 5-percentage-point difference matters. That judgment depends on the consequences of the outcome and the purpose of the investigation.

Worked Example: A Small Difference With Strong Evidence

A hypothetical random sample of 10,000 independent customers finds that 5,150 use a particular service. Test \(H_0:p=0.50\) against \(H_a:p\ne0.50\), and explain why statistical evidence and practical importance need separate consideration.

The binary outcome and random independent sample support the model. Under the null, the expected numbers are \(10{,}000(0.50)=5{,}000\) successes and 5,000 failures, so both expected counts are ample for the normal approximation. The observed proportion is \(\hat{p}=5{,}150/10{,}000=0.515\), an estimated difference of \(0.015\), or 1.5 percentage points.

$$ SE_0=\sqrt{\frac{0.50(1-0.50)}{10{,}000}}=0.005,\qquad z=\frac{0.515-0.50}{0.005}=3.00 $$

For the two-sided alternative, the p-value is \(2P(Z\ge3.00)\), approximately \(0.00270\). The sample provides strong evidence against a population proportion of 0.50 under this test model. The estimated difference is nevertheless only 1.5 percentage points, equivalent to 15 additional users per 1,000 compared with a 50% rate.

Whether that difference is useful or important depends on the service and the decision being made. For a low-cost change affecting many customers, 15 per 1,000 might matter. If changing the service is expensive or the outcome has little consequence, it might not. The small p-value supports a conclusion about evidence against the null; it does not settle the practical decision.

Worked Example: A Larger Difference With Limited Evidence

A hypothetical independent random sample of 40 eligible households finds that 26 have adopted a new program. Suppose the question is whether adoption exceeds 50%, so \(H_0:p=0.50\) and \(H_a:p>0.50\). Calculate the estimated difference and the one-sided p-value.

The outcome is binary, and we assume the households were selected by a suitable random process and their outcomes can be treated as independent. Under the null, the expected numbers are \(40(0.50)=20\) successes and 20 failures, satisfying the expected-count condition for the z approximation. The observed proportion is \(\hat{p}=26/40=0.65\), which is 15 percentage points above the null value.

$$ SE_0=\sqrt{\frac{0.50(1-0.50)}{40}}\approx0.07906,\qquad z=\frac{0.65-0.50}{0.07906}\approx1.897 $$

Because the alternative is upper-tailed, the p-value is \(P(Z\ge1.897)\), approximately \(0.0289\). At the 0.05 significance level, the data provide evidence in the direction that adoption exceeds 50%. The estimated 15-percentage-point difference is larger than the differences in the first two examples, but the sample is smaller. Its p-value therefore reflects both the difference and the sampling variability under the null.

This observed difference is not proof that the population proportion exceeds 50% by exactly 15 percentage points. It is a sample estimate. Nor does the p-value establish that the program caused adoption: this is a sample-based test of a population proportion, and causal claims require an appropriate study design.

A Short Justification: Why Sample Size Changes the Test

The relationship between the estimated difference and the test statistic follows directly from the z formula. Write \(d=\hat{p}-p_0\). Then:

$$ z=\frac{d}{\sqrt{p_0(1-p_0)/n}} =\frac{d\sqrt{n}}{\sqrt{p_0(1-p_0)}} $$

For fixed \(p_0\) and fixed nonzero \(d\), the denominator in the first expression decreases as \(n\) increases. Equivalently, the second expression shows that \(|z|\) increases in proportion to \(\sqrt{n}\). For a two-sided test, a larger \(|z|\) corresponds to a smaller tail probability under the standard normal model. This explains why equal differences can produce different p-values. It does not mean that increasing the sample size makes the underlying difference larger; it makes the test more capable of distinguishing that difference from sampling variation under \(H_0\).

How to Report Both Quantities

A clear conclusion reports the estimated difference in context and describes the p-value as evidence under the null model. For example: “The observed adoption proportion was 65%, 15 percentage points above the 50% null value. The one-sided p-value was about 0.0289, indicating evidence against the null in the direction specified by the alternative.” This sentence reports the estimate and evidence separately. A practical judgment would then need information about consequences, costs, and what size of difference would matter.

There is no universal threshold at which a percentage-point difference becomes practically important. The same numerical difference may matter greatly for a common, serious outcome and little for a rare or inconsequential one. Practical importance must be judged in the substantive context, not inferred from whether the p-value falls below a conventional cutoff.

Common Mistakes

  • Calling a small p-value a large effect. A p-value is calculated from the test statistic and null distribution; it is not a measurement of percentage points. Report \(\hat{p}-p_0\) to describe the observed difference.
  • Calling a large p-value proof of no effect. It means the data did not provide strong evidence against the specified null by this test. The observed difference may still be substantial, especially when the sample is small.
  • Equating statistical evidence with practical importance. A result can provide strong evidence against \(H_0\) while having little practical consequence. Context and consequences matter.
  • Interpreting the p-value as the probability that \(H_0\) is true. The p-value is calculated assuming the null model; it is not the probability that the null hypothesis is true.
  • Reporting a percentage change when the result is in percentage points. A change from 50% to 55% is 5 percentage points, not a 5% relative increase.
Key takeaway: State the estimated difference \(\hat{p}-p_0\) in the units of the problem, then report the p-value as evidence against the null under the test assumptions. Neither quantity alone answers whether the difference is practically important.

Check Your Understanding

Use the distinction between evidence and effect size to answer these questions.

  1. In a one-proportion test, what does \(\hat{p}-p_0\) measure, and what does the p-value measure?
  2. Two tests have the same observed difference from \(p_0\), but one has a much larger sample. Why might its p-value be smaller?
  3. A test reports a very small p-value and an estimated difference of 0.4 percentage points. What can and cannot be concluded from those facts alone?
  4. Why is a change from 40% to 46% a 6-percentage-point increase but a 15% relative increase?
  5. What should be checked before interpreting a one-proportion z-test p-value?

Previous

Next

Back to Tutorials