Why Estimate a Difference?
Many questions compare the proportion of individuals with an outcome in one group with the corresponding proportion in another. For example, a clinic might want to estimate how much the rate of a particular outcome differs between two treatment groups. The sample proportions will usually differ, but that difference alone does not tell us how precisely the population difference has been estimated. A confidence interval combines the observed difference with an estimate of its sampling uncertainty.
The order of subtraction matters. Throughout this tutorial, group 2 is subtracted from group 1, in the order “group 1 minus group 2”: a positive difference means the population proportion in group 1 is larger. The unit for a difference between proportions is a proportion, often expressed as percentage points.
A confidence interval estimates a range of plausible values for \(p_1-p_2\), accounting for sampling variability. To construct the large-sample interval described here, we first need an estimate of the standard deviation of the sampling distribution of \(\hat{p}_1-\hat{p}_2\). This estimate is called the unpooled standard error.
Deriving the Unpooled Standard Error
Suppose the two groups are independent, and within each group the outcome is recorded as a success or failure under the independent Bernoulli model. The group 1 success count has a binomial model with sample size \(n_1\) and success probability \(p_1\); group 2 has sample size \(n_2\) and success probability \(p_2\). For a sample proportion from group \(i\), its variance is \(p_i(1-p_i)/n_i\).
Because the groups are independent, the covariance between their sample proportions is zero. The variance of a difference is the sum of the two variances when that covariance is zero. Therefore, the sampling variance of the difference in sample proportions is:
The standard deviation is the square root of this variance. Since the population proportions are unknown, estimate each one separately using its sample proportion. This substitution gives the unpooled standard error:
It is called unpooled because the formula uses a separate estimate for each group’s proportion. The interval estimates \(p_1-p_2\) when the two proportions may differ, so each group’s estimated variability is retained separately.
From Standard Error to Confidence Interval
For a large sample, the sampling distribution of \(\hat{p}_1-\hat{p}_2\) is approximately normal when the Bernoulli model and independence assumptions are reasonable and each group has enough successes and failures. A common large-sample condition is that each group has at least 10 observed successes and at least 10 observed failures. Under this approximation, a standard-normal critical value \(z^*\) gives the margin of error \(z^*SE\).
The interval rests on the following conditions: observations within each group follow the Bernoulli model; the two groups are independent; the data come from an appropriate random sampling or assignment process for the intended inference; and the sample sizes are large enough for the normal approximation. For sampling without replacement from a finite population, the observations should also be sufficiently independent, commonly justified when each sample is no more than 10% of its population. The success-and-failure counts provide a practical check on the normal approximation. These conditions concern how the data were obtained as well as the observed counts.
State which group is group 1 and define the parameter as \(p_1-p_2\).
Find \(\hat{p}_1=x_1/n_1\) and \(\hat{p}_2=x_2/n_2\), then calculate their difference.
Substitute each sample proportion and sample size into the unpooled formula.
Multiply the standard error by the critical value, add and subtract the margin of error, and describe the resulting range in context.
Worked Confidence Intervals
Worked Example: Comparing Two Hypothetical Outcome Rates
In a hypothetical study, 48 of 150 individuals in group 1 and 32 of 160 individuals in group 2 have a specified outcome. Construct a 95% large-sample confidence interval for \(p_1-p_2\), where the difference is taken as group 1 minus group 2.
Check the counts and model. The groups are assumed to be independent, with independent Bernoulli outcomes within each group, and to arise from an appropriate sampling or assignment process. Group 1 has 48 successes and \(150-48=102\) failures; group 2 has 32 successes and \(160-32=128\) failures. Each count is at least 10, supporting the large-sample approximation.
Calculate the point estimate. The sample proportions are \(\hat{p}_1=48/150=0.3200\) and \(\hat{p}_2=32/160=0.2000\). Thus the estimated difference is \(0.3200-0.2000=0.1200\).
Calculate the standard error and margin of error.
Using the unrounded standard error in the margin-of-error calculation, \(1.96\sqrt{0.3200(1-0.3200)/150+0.2000(1-0.2000)/160}\approx0.09703\).
Construct and interpret the interval. The endpoints are \(0.1200-0.09703\approx0.0230\) and \(0.1200+0.09703\approx0.2170\). The 95% confidence interval is approximately \((0.0230,0.2170)\). In this hypothetical setting, the data support a population outcome proportion in group 1 that is about 2.30 to 21.7 percentage points higher than the proportion in group 2.
Worked Example: An Interval That Includes Zero
Suppose a hypothetical comparison records the outcome for 84 of 140 individuals in group 1 and 70 of 140 in group 2. Find a 95% confidence interval for the difference \(p_1-p_2\).
Check the conditions. Assume the groups are independent, observations within each group follow the Bernoulli model, and the data-collection process supports inference to the intended populations. Group 1 has 84 successes and 56 failures; group 2 has 70 successes and 70 failures. All counts exceed 10.
The sample proportions are \(\hat{p}_1=84/140=0.6000\) and \(\hat{p}_2=70/140=0.5000\), so the point estimate is \(0.1000\).
The margin of error is \(1.96(0.05916)\approx0.1160\). Therefore, the endpoints are \(0.1000-0.1160=-0.0160\) and \(0.1000+0.1160=0.2160\), giving a 95% interval of approximately \((-0.0160,0.2160)\).
The interval includes zero, so values consistent with the data range from a small proportion in which group 1 is lower to one in which group 1 is higher. This does not prove that the population proportions are equal; it indicates that this interval does not rule out a zero difference. The interval’s interpretation is about plausible values for \(p_1-p_2\), not a probability that the fixed parameter lies in this particular interval.
Worked Example: Keeping the Group Order Clear
In another hypothetical data set, 90 of 200 individuals in group 1 and 60 of 180 in group 2 have the outcome. Construct a 95% interval for \(p_1-p_2\) and state the direction of the estimated difference.
Assume independent groups, independent Bernoulli observations, and an appropriate sampling or assignment process. Group 1 has 90 successes and 110 failures; group 2 has 60 successes and 120 failures. The counts satisfy the large-sample condition.
The proportions are \(\hat{p}_1=90/200=0.4500\) and \(\hat{p}_2=60/180=0.3333\), so the estimated difference is \(0.4500-0.3333\approx0.1167\).
The 95% margin of error is \(1.96(0.04972)\approx0.09745\). The endpoints are \(0.1167-0.09745\approx0.0192\) and \(0.1167+0.09745\approx0.2141\). Thus the interval is approximately \((0.0192,0.2141)\), corresponding to a group 1 proportion about 1.92 to 21.4 percentage points higher than group 2.
The direction follows from the stated order: the parameter is \(p_1-p_2\), not \(p_2-p_1\). Reversing the subtraction would negate both endpoints and would answer a different, though equivalent, question.
Why the Construction Matters
The standard error measures the estimated sampling variability of the difference in sample proportions. A larger standard error produces a wider interval for the same confidence level; larger sample sizes generally reduce the standard error, all else equal. The interval therefore reports both the estimated direction and size of the difference and the uncertainty arising from sampling.
The unpooled formula uses \(\hat{p}_1\) in the group 1 variance term and \(\hat{p}_2\) in the group 2 term because these are estimates of separate population proportions. Do not replace them with a single combined proportion when constructing this confidence interval. The formula is an approximation: when the data are sparse or the normal approximation is poor, this interval may be unreliable and can even produce endpoints outside the possible range for a proportion difference, \([-1,1]\).
A 95% confidence procedure produces intervals that cover the fixed population difference in about 95% of repeated samples when its assumptions hold. It does not mean that, after observing these data, there is a 95% probability that the particular interval contains the parameter. In reporting results, state the group order, the interval, and the difference in proportion or percentage-point units so that the direction and practical meaning are clear.
Check Your Understanding
Use the definitions and interval construction in this tutorial to answer the following.
- In a comparison with \(\hat{p}_1=0.40\) and \(\hat{p}_2=0.25\), what is the point estimate for \(p_1-p_2\), and what does its sign indicate?
- Why does the unpooled standard error contain one variance term for each group?
- A group has 7 successes and 43 failures. Does it meet the stated large-sample success-and-failure condition?
- A 95% interval for \(p_1-p_2\) is \((-0.04,0.12)\). What does inclusion of zero mean, and what does it not prove?
- Explain why “there is a 95% probability that \(p_1-p_2\) lies in this observed interval” is not the correct interpretation of a 95% confidence interval.