Why Compare Two Proportions?
Many actuarial and statistical questions compare the frequency of a binary outcome across two populations. For example, a hypothetical insurer might compare the proportion of customers who renew under two different service procedures. The outcome for each customer is either a renewal or a nonrenewal, so the comparison concerns proportions rather than averages of numerical measurements.
The direction of the comparison matters. In this tutorial, population 1 is the first group named, and the parameter is the proportion in population 1 minus the proportion in population 2. A negative estimate therefore means the observed proportion was lower in group 1. The sign does not by itself indicate whether the difference is important.
The model assumes that, within each sample, observations are independent Bernoulli trials with a common probability: \(X_1\sim\operatorname{Bin}(n_1,p_1)\) and \(X_2\sim\operatorname{Bin}(n_2,p_2)\). It also assumes the two samples are independent of one another. A result must be classified consistently as either the outcome of interest or its complement; the two groups must use the same definition of success.
These assumptions are not just formalities. If the same people are measured under two conditions, the resulting proportions are paired rather than independent, and the variance formula below generally does not apply. Likewise, clustering or dependence among observations can change the variance. The calculations here concern independent binomial samples.
Estimating the Difference and Its Sampling Variance
The sample proportion in each group estimates that group’s population probability. Subtracting the estimates gives a natural estimate of the population difference:
This estimator is unbiased under the binomial model. Since \(E[\hat p_j]=p_j\) for either group \(j\), linearity of expectation gives \(E[\hat\Delta]=p_1-p_2=\Delta\). Its variability depends on both samples, not only on the numerical distance between their observed proportions.
Why the Variance Formula Holds
For group \(j\), write each binary observation as \(I_{jk}\), equal to 1 when the specified outcome occurs and 0 otherwise. Then \(X_j=\sum_{k=1}^{n_j} I_{jk}\), and \(\hat p_j\) is the average of these indicators. Each indicator has a Bernoulli distribution with probability \(p_j\), so its variance is \(p_j(1-p_j)\). Independence within a sample means the variance of the sum is the sum of the individual variances. Dividing by \(n_j^2\) gives the variance of the sample proportion:
For any two random variables, the variance of their difference includes a covariance term. Here, independence of the two samples makes that covariance zero. Therefore:
The exact variance contains the unknown population proportions. Replacing each by its sample estimate gives the plug-in estimated variance. This substitution produces an estimate of the variance; it does not make the unknown variance known. Under conditions where the sample proportions are well approximated by normal distributions, the difference estimator is also approximately normal. The formula here supplies its variance estimate, not a confidence interval.
Worked Calculations
Worked Example: Comparing Renewal Rates
A hypothetical insurer independently samples 120 customers offered service procedure 1 and 150 customers offered procedure 2. Renewal is the specified success outcome. There are 78 renewals in the first sample and 87 in the second. Assume independent Bernoulli observations within each group and independent binomial samples, so \(X_1\sim\operatorname{Bin}(120,p_1)\) and \(X_2\sim\operatorname{Bin}(150,p_2)\). No interest or mortality assumption is involved. Estimate \(p_1-p_2\), its sampling variance, and its standard error.
Method. Calculate the two sample proportions, subtract in the stated order, and substitute each proportion into its own variance contribution.
First proportion. \(\hat p_1=78/120=0.6500\), so \(1-\hat p_1=0.3500\).
Second proportion. \(\hat p_2=87/150=0.5800\), so \(1-\hat p_2=0.4200\).
Difference estimate. \(\hat\Delta=0.6500-0.5800=0.0700\). The observed proportion in group 1 exceeds the observed proportion in group 2 by 0.0700, or 7.00 percentage points.
Estimated variance. The first contribution is \(0.6500(0.3500)/120=0.2275/120=0.0018958333\). The second is \(0.5800(0.4200)/150=0.2436/150=0.0016240000\). Thus \(\widehat{\operatorname{Var}}(\hat\Delta)=0.0018958333+0.0016240000=0.0035198333\).
Estimated standard error. \(\sqrt{0.0035198333}=0.0593\) to four decimal places. As a check, the variance is positive and both contributions decrease with larger sample sizes. The common mistake of using the same denominator for both groups would ignore the unequal sample sizes; each contribution uses its own group’s \(n_j\).
Worked Example: Equal Observed Proportions Do Not Mean Zero Variance
In a hypothetical independent study, 20 of 50 sampled members in group 1 and 80 of 200 sampled members in group 2 report the specified outcome. Assume \(X_1\sim\operatorname{Bin}(50,p_1)\), \(X_2\sim\operatorname{Bin}(200,p_2)\), independent samples, and independent Bernoulli outcomes within each group. No interest or mortality assumption applies. Find the estimated difference and estimated variance.
Method. Use the separate sample proportions and the separate binomial variance contributions.
Sample proportions and difference. \(\hat p_1=20/50=0.4000\) and \(\hat p_2=80/200=0.4000\). Therefore \(\hat\Delta=0.4000-0.4000=0.0000\).
Estimated variance. For group 1, \(0.4000(0.6000)/50=0.2400/50=0.004800\). For group 2, \(0.4000(0.6000)/200=0.2400/200=0.001200\). Add the contributions: \(\widehat{\operatorname{Var}}(\hat\Delta)=0.004800+0.001200=0.006000\).
The estimated standard error is \(\sqrt{0.006000}=0.07746\) to five decimal places. The calculation is internally consistent: although the observed difference is zero, each sample proportion varies from sample to sample, so the difference still has positive sampling variance. The trap is to infer zero variance from equal observed proportions; variance measures sampling variability, not the observed distance between the two estimates.
Worked Example: Preserve the Order When the Estimate Is Negative
A hypothetical survey independently samples 80 users of plan 1 and 90 users of plan 2. A specified feature is used by 34 users in the first sample and 63 in the second. Assume \(X_1\sim\operatorname{Bin}(80,p_1)\), \(X_2\sim\operatorname{Bin}(90,p_2)\), with independent binomial samples and independent Bernoulli observations within each. There is no interest or mortality assumption. Estimate \(p_1-p_2\), its estimated variance, and its standard error.
Method. Keep the requested order, group 1 minus group 2, throughout the estimate. Variance contributions are nonnegative and are added regardless of the sign of the difference.
Proportions. \(\hat p_1=34/80=0.4250\), with complement \(1-\hat p_1=0.5750\). Also, \(\hat p_2=63/90=0.7000\), with complement \(1-\hat p_2=0.3000\).
Difference estimate. \(\hat\Delta=0.4250-0.7000=-0.2750\). The negative sign indicates that the observed proportion in group 1 is lower by 0.2750.
Estimated variance. The group 1 contribution is \(0.4250(0.5750)/80=0.244375/80=0.0030546875\). The group 2 contribution is \(0.7000(0.3000)/90=0.210000/90=0.0023333333\). The total is \(0.0030546875+0.0023333333=0.0053880208\).
Estimated standard error. \(\sqrt{0.0053880208}=0.0734\) to four decimal places. The variance is positive despite the negative difference, as it must be. Reversing the subtraction would change the estimate to \(+0.2750\), but would not change the variance.
Interpretation and Common Traps
The estimated difference describes the observed samples; the variance estimate describes how much the difference estimator would vary across repeated samples under the assumed model. These are distinct quantities. Equal sample proportions yield a zero estimated difference, but not generally a zero variance. Similarly, a large estimated difference does not alone determine its precision: the sample sizes and the proportions both affect the variance.
A useful way to organize the calculation is to make a separate variance contribution for each group. For group \(j\), the contribution is \(\hat p_j(1-\hat p_j)/n_j\). The product \(\hat p_j(1-\hat p_j)\) is largest near a proportion of one-half and smaller near zero or one; division by \(n_j\) reflects the reduction in sampling variability as the sample size grows.
- Changing the order silently. The target is defined as \(p_1-p_2\). Write the group order before subtracting, because exchanging groups reverses the sign.
- Using a difference to calculate the variance. The estimated variance is not \(\hat\Delta(1-\hat\Delta)\). Each group has its own Bernoulli proportion and sample size.
- Dropping a group’s contribution. The independent difference contains variability from both samples, so both terms must be included.
- Treating dependent data as independent. When observations are paired or otherwise dependent, the covariance need not be zero, and the independent-sample formula is not justified.
Check Your Understanding
For each problem, use the stated order for the difference and the independent-binomial plug-in variance formula. Round variances to six decimal places where needed.
- A hypothetical survey records 63 successes among 100 independently sampled members of group 1 and 75 among 125 members of group 2. Find the estimated difference and estimated variance. (Answer: difference \(0.0300\); estimated variance \(0.004251\).)
- Independent binomial samples have sizes 80 and 160, with 28 and 72 successes, respectively. Find the group 1 minus group 2 estimate and its estimated variance. (Answer: difference \(-0.1000\); estimated variance \(0.004391\).)
- Two independent samples have sizes 50 and 200, with 20 and 80 successes. Find the estimated difference, variance, and standard error. (Answer: difference \(0.0000\); estimated variance \(0.006000\); standard error \(0.07746\).)
- A hypothetical study observes 45 successes among 72 in group 1 and 54 among 96 in group 2. Find the estimated difference and estimated variance. (Answer: difference \(0.0625\); estimated variance \(0.005819\).)
- Independent samples of sizes 60 and 140 have 21 and 70 successes. Compute the group 1 minus group 2 difference and its estimated variance. (Answer: difference \(-0.1500\); estimated variance \(0.005577\).)