Calculating P-Values for Two-Proportion Tests

From a Two-Proportion Statistic to a P-Value

A two-proportion test asks whether the observed difference between two sample proportions would be unusual if the population proportions were equal. The z statistic measures how many estimated standard errors the observed difference lies from zero. The p-value then translates that statistic into a probability under the null model.

The p-value depends not only on the size of the statistic but also on the direction specified by the alternative hypothesis. A statistic far to the right is evidence for a greater-than alternative; a statistic far to the left is evidence for a less-than alternative. When the alternative is that the proportions differ in either direction, extremeness in both tails counts.

Model, Statistic, and Assumptions

Let $x_1$ and $x_2$ be the numbers of individuals with the outcome of interest in two groups, and let $n_1$ and $n_2$ be the group sample sizes. The sample proportions are $\hat{p}_1=x_1/n_1$ and $\hat{p}_2=x_2/n_2$. The difference being tested is $p_1-p_2$, with group 1 minus group 2 as the order throughout.

The test uses a model in which outcomes are binary, observations are independent within each group, and the two groups are independent of one another. Under the null hypothesis $H_0:p_1=p_2$, both groups share a common outcome probability. The pooled estimate of that common probability is the total number of observed outcomes divided by the total sample size. The normal approximation for the test also requires that it be reasonable to approximate the null distribution of the difference by a normal distribution; the model and approximation conditions must be appropriate for the data.

Definition: For testing $H_0:p_1=p_2$, the pooled estimate and pooled standard error are $$ \hat{p}_{\text{pool}}=\frac{x_1+x_2}{n_1+n_2}, \qquad SE_{\text{pool}}= \sqrt{\hat{p}_{\text{pool}}(1-\hat{p}_{\text{pool}}) \left(\frac{1}{n_1}+\frac{1}{n_2}\right)}. $$ The two-proportion z statistic is $$ z=\frac{\hat{p}_1-\hat{p}_2}{SE_{\text{pool}}}. $$ Under $H_0$ and the model and approximation assumptions, this statistic is approximately standard normal, $Z\sim N(0,1)$.

The pooled standard error is used because the null hypothesis says the two population proportions are equal. The statistic compares the observed difference with the amount of sampling variation expected under that null. The p-value is calculated from the standard normal distribution, not by treating the estimated proportions as known population values.

Choosing the Correct Tail

Write the alternative hypothesis before calculating the p-value. It determines which values of the statistic count as evidence at least as extreme as the observed value. Let $\Phi(z)$ denote the cumulative probability $P(Z\le z)$ for a standard normal random variable.

Alternative hypothesis	Evidence in the statistic	p-value
$H_a:p_1>p_2$	Large positive values	$P(Z\ge z_{\text{obs}})=1-\Phi(z_{\text{obs}})$
$H_a:p_1<p_2$	Large negative values	$P(Z\le z_{\text{obs}})=\Phi(z_{\text{obs}})$
$H_a:p_1\ne p_2$	Large values in either direction	$P(\|Z\|\ge \|z_{\text{obs}}\|)=2[1-\Phi(\|z_{\text{obs}}\|)]$

These expressions are tail areas under the null distribution. A right-tailed test asks how often the null model would produce a statistic at least as large as the observed one. A left-tailed test asks how often it would produce one at least as small. A two-sided test counts either kind of departure, so for a symmetric standard normal distribution it doubles the area beyond the observed statistic's absolute value.

Key interpretation: A p-value is the probability, assuming the null hypothesis and the model are correct, of obtaining a test statistic at least as extreme as the observed statistic in the direction or directions specified by $H_a$. It is not the probability that $H_0$ is true.

Worked Examples

Worked Example: A Greater-Than Alternative

In a hypothetical evaluation, 36 of 120 people in Group 1 and 24 of 120 people in Group 2 experience a specified outcome. Test $H_0:p_1=p_2$ against $H_a:p_1>p_2$, and calculate the p-value. Assume the independent binary-outcome model and normal approximation are appropriate.

The sample proportions are $\hat{p}_1=36/120=0.3000$ and $\hat{p}_2=24/120=0.2000$, so the observed difference is $0.1000$. Under the null, the pooled estimate is

\hat{p}_{\text{pool}}=\frac{36+24}{120+120} =\frac{60}{240}=0.2500.

The pooled standard error is

SE_{\text{pool}} =\sqrt{0.25(1-0.25)\left(\frac{1}{120}+\frac{1}{120}\right)} =\sqrt{0.25(0.75)(0.016667)} =\sqrt{0.003125} =0.05590.

Thus,

z=\frac{0.3000-0.2000}{0.05590}=1.789.

The alternative is greater-than, so the p-value is the area to the right:

P(Z\ge 1.789)=1-\Phi(1.789)\approx 1-0.9632=0.0368.

Under the null model, the probability of a z statistic at least 1.789 is about 0.0368 in the direction that Group 1's proportion is higher. At significance level $\alpha=0.05$, this p-value is below $\alpha$, so the test rejects $H_0$ in favor of evidence that $p_1>p_2$. This conclusion concerns the population proportions under the stated model; the calculation alone does not establish a causal effect.

Worked Example: A Less-Than Alternative

In a hypothetical comparison, 18 of 100 people in Group 1 and 30 of 100 people in Group 2 experience an outcome. The research question is whether Group 1's population proportion is lower: $H_a:p_1<p_2$. Assume the model and normal approximation are appropriate.

The sample proportions are $\hat{p}_1=18/100=0.1800$ and $\hat{p}_2=30/100=0.3000$. Pooling the outcomes gives

\hat{p}_{\text{pool}}=\frac{18+30}{100+100} =\frac{48}{200}=0.2400.

The pooled standard error and test statistic are

SE_{\text{pool}} =\sqrt{0.24(0.76)\left(\frac{1}{100}+\frac{1}{100}\right)} =\sqrt{0.003648}=0.06040, $$ $$ z=\frac{0.1800-0.3000}{0.06040} =\frac{-0.1200}{0.06040} =-1.987.

Because the alternative is less-than, use the lower tail:

P(Z\le -1.987)=\Phi(-1.987)\approx 0.0235.

The p-value is about 0.0235: if the population proportions were equal and the model held, a z statistic at or below $-1.987$ would occur with probability about 0.0235. At $\alpha=0.05$, the result is statistically significant in the specified lower direction. The sign of $z$ is consistent with the direction of the observed difference, but the alternative—not the sign alone—determines which tail is used.

Worked Example: A Two-Sided Alternative

In a hypothetical study, 42 of 150 people in Group 1 and 27 of 150 people in Group 2 experience an outcome. Test whether the population proportions differ, $H_a:p_1\ne p_2$. Assume the model and normal approximation are appropriate.

The sample proportions are $\hat{p}_1=42/150=0.2800$ and $\hat{p}_2=27/150=0.1800$, for an observed difference of $0.1000$. The pooled estimate is

\hat{p}_{\text{pool}}=\frac{42+27}{150+150} =\frac{69}{300}=0.2300.

Then,

SE_{\text{pool}} =\sqrt{0.23(0.77)\left(\frac{1}{150}+\frac{1}{150}\right)} =\sqrt{0.1771(0.013333)} =\sqrt{0.0023613} =0.04859, $$ $$ z=\frac{0.2800-0.1800}{0.04859}=2.058.

The two-sided alternative counts results at least this far from zero in either direction:

p\text{-value}=P(|Z|\ge 2.058) =2[1-\Phi(2.058)] \approx 2(0.0198)=0.0396.

The p-value is approximately 0.0396. Under the null model, about 3.96% of test statistics would be at least 2.058 standard normal units from zero in either direction. At $\alpha=0.05$, the p-value supports rejecting equality in favor of a difference. This two-sided result does not, by itself, establish a particular direction; the observed sample difference is positive, but the alternative allowed either direction.

Why the Tail Choice Matters

A p-value is not a property of the z statistic alone. It is calculated from the statistic together with the alternative hypothesis. For example, a positive statistic is evidence in the direction $p_1>p_2$, but if the stated alternative is $p_1<p_2$, the p-value is the area to the left of that statistic. It may be large because the observed result points away from the alternative.

The choice between a one-sided and two-sided alternative should be made from the research question before examining the results. Choosing a one-sided tail after seeing which group has the larger sample proportion changes the question and can make the evidence look stronger than the pre-specified test warrants. When the scientific question is whether proportions differ in either direction, use a two-sided test.

The p-value also is not the probability of seeing the exact observed data, nor does a small value measure the size or practical importance of a difference. It describes how incompatible the observed statistic is with the null model, using the specified tail or tails. The sample difference itself gives the estimated direction and magnitude; those are distinct from the p-value.

A Short Justification of the Two-Sided Formula

Under the null hypothesis, the z statistic is approximately standard normal, whose distribution is symmetric around zero. If the observed statistic is $z_{\text{obs}}$, a two-sided test counts values at least as far from zero on either side. When $z_{\text{obs}}$ is positive, those values are $Z\ge z_{\text{obs}}$ and $Z\le -z_{\text{obs}}$. By symmetry, these two tail areas are equal:

P(Z\ge z_{\text{obs}})+P(Z\le -z_{\text{obs}}) =2P(Z\ge |z_{\text{obs}}|) =2[1-\Phi(|z_{\text{obs}}|)].

If $z_{\text{obs}}$ is negative, the same argument applies after taking its absolute value: the two tails begin at $z_{\text{obs}}$ and $-z_{\text{obs}}$. Thus the formula works for either sign. It also shows why using just one tail for a two-sided alternative would omit half of the evidence counted by the test.

Common Mistakes

Using the wrong tail. A positive z statistic does not automatically mean “use the right tail.” Read $H_a$: a less-than alternative uses the left tail, even if the observed statistic happens to be positive.
Forgetting to double the tail area. For a two-sided test, include equally extreme results in both directions. Use $2[1-\Phi(|z_{\text{obs}}|)]$, not just the single upper-tail area.
Calling the p-value the probability that the null is true. The p-value is calculated assuming the null model; it is a probability of statistics at least as extreme as the observed one under that assumption.
Treating a nonsignificant result as proof of equality. If the p-value exceeds $\alpha$, the test does not provide sufficient evidence against $H_0$ at that level. It does not prove that $p_1=p_2$.

Key takeaway: Calculate $z$ using the pooled standard error for a test of equal proportions, then let $H_a$ determine the tail probability. Report the p-value with its direction and interpret it conditionally on the null model.

Check Your Understanding

For each question, identify the relevant tail or explain the interpretation.

If $H_a:p_1>p_2$ and $z_{\text{obs}}=1.40$, which standard normal probability gives the p-value?
If $H_a:p_1<p_2$ and $z_{\text{obs}}=-2.10$, which tail is used?
For a two-sided test with $z_{\text{obs}}=-1.75$, write the p-value formula using $\Phi$.
In words, what does a p-value of 0.03 mean for a two-sided test of equal population proportions?
Why is it incorrect to say that a p-value of 0.03 means there is a 3% probability that $H_0$ is true?

Alternative hypothesis	Evidence in the statistic	p-value
\(H_a:p_1>p_2\)	Large positive values	\(P(Z\ge z_{\text{obs}})=1-\Phi(z_{\text{obs}})\)
\(H_a:p_1<p_2\)	Large negative values	\(P(Z\le z_{\text{obs}})=\Phi(z_{\text{obs}})\)
\(H_a:p_1\ne p_2\)	Large values in either direction	\(P(\|Z\|\ge \|z_{\text{obs}}\|)=2[1-\Phi(\|z_{\text{obs}}\|)]\)