The Confidence Interval and Test Connection

Why Compare a Test With an Interval?

A two-sided hypothesis test and a confidence interval answer related questions. A test asks whether the data are sufficiently inconsistent with a particular hypothesized difference. An interval gives a range of differences that remain compatible with the data under a specified method and confidence level. When the procedures are matched, their decisions about a particular null value agree: at significance level \(\alpha\), a two-sided test rejects that value exactly when the corresponding \(100(1-\alpha)\%\) confidence interval excludes it.

The word matched matters. Two procedures can use different estimates of uncertainty and therefore make different decisions for the same data. In particular, the usual Wald confidence interval for a difference in proportions uses an unpooled standard error, while the usual two-proportion score test uses a pooled standard error under the null hypothesis. This tutorial first establishes the algebraic connection for a Wald test paired with a Wald interval, then explains what changes when the pooled score test is compared with that interval.

Model and Matching Procedures

Suppose Group 1 and Group 2 each provide a binary outcome, coded as a success or failure. Let \(n_1\) and \(n_2\) be their sample sizes, and let \(x_1\) and \(x_2\) be their numbers of successes. The sample proportions are \(\hat{p}_1=x_1/n_1\) and \(\hat{p}_2=x_2/n_2\). The parameter of interest is the difference in population proportions, \(\delta=p_1-p_2\), estimated by \(\hat{p}_1-\hat{p}_2\).

Use the independent Bernoulli model: observations are independent within each group, each observation in Group 1 has success probability \(p_1\), each in Group 2 has success probability \(p_2\), and the two groups are independent. Under the usual conditions for a normal approximation, the estimated standard error for a difference in proportions is

SE_W=\sqrt{\frac{\hat{p}_1(1-\hat{p}_1)}{n_1}+ \frac{\hat{p}_2(1-\hat{p}_2)}{n_2}}.

The subscript \(W\) indicates the unpooled Wald standard error: each group’s observed proportion estimates its own outcome variability. The procedures below are normal-approximation procedures, so their reliability depends on the sampling design and whether the approximation is suitable for the data. The formulas do not remove the need to consider those issues.

Definition: For testing \(H_0:\delta=0\), the matching two-sided Wald test uses \(z_W=(\hat{p}_1-\hat{p}_2)/SE_W\), provided \(SE_W>0\). The corresponding two-sided \(100(1-\alpha)\%\) Wald confidence interval for \(\delta\) is \((\hat{p}_1-\hat{p}_2)\pm z_{1-\alpha/2}SE_W\), where \(z_{1-\alpha/2}\) is the standard normal quantile with cumulative probability \(1-\alpha/2\).

Here \(SE_W\) is an estimate of the standard deviation of the estimated difference. The confidence interval uses the same estimate and the same normal approximation as the matching Wald test. For a null value other than zero, the comparison is between that null value and the interval; the same algebra applies after subtracting the null value from the estimate.

Why the Decisions Agree

Write \(d=\hat{p}_1-\hat{p}_2\) and \(c=z_{1-\alpha/2}\). When \(SE_W>0\), the Wald interval is \(d\pm cSE_W\). Zero is outside this interval if the whole interval lies above zero or the whole interval lies below zero. In symbols,

0\notin[d-cSE_W,\ d+cSE_W] \quad\Longleftrightarrow\quad d-cSE_W>0\ \text{or}\ d+cSE_W<0.

Rearranging the first inequality gives \(d>cSE_W\). Rearranging the second gives \(d<-cSE_W\). Together these mean that the absolute value of \(d\) is greater than \(cSE_W\). Since \(SE_W\) is positive, dividing by it preserves the inequalities:

0\notin[d-cSE_W,\ d+cSE_W] \quad\Longleftrightarrow\quad |d|>cSE_W \quad\Longleftrightarrow\quad \left|\frac{d}{SE_W}\right|>z_{1-\alpha/2}.

The right-hand condition is exactly the two-sided Wald test’s critical-value rule. Equivalently, if \(\Phi\) is the standard normal cumulative distribution function, the approximate two-sided p-value is \(2[1-\Phi(|z_W|)]\). It is less than \(\alpha\) exactly when \(|z_W|>z_{1-\alpha/2}\). Thus the matching test rejects \(H_0:\delta=0\) if and only if the matching interval excludes zero.

If the test statistic is exactly at the critical value, the p-value equals \(\alpha\), and zero is an endpoint of the interval. Under the strict rejection rule \(p\text{-value}<\alpha\), the test does not reject and the interval includes that endpoint. This boundary convention preserves the equivalence.

Worked Examples Near the Decision Boundary

Worked Example: A 95% Interval and Wald Test Give the Same Decision

Suppose hypothetical independent samples each contain 100 people. Group 1 has 60 successes and Group 2 has 45 successes. Test \(H_0:p_1-p_2=0\) against a two-sided alternative at \(\alpha=0.05\), and compare the matching 95% Wald interval with the test.

The sample proportions are \(\hat{p}_1=60/100=0.60\) and \(\hat{p}_2=45/100=0.45\), so \(d=0.60-0.45=0.15\). The estimated unpooled standard error is

SE_W=\sqrt{\frac{0.60(0.40)}{100}+ \frac{0.45(0.55)}{100}} =\sqrt{0.002400+0.002475} =\sqrt{0.004875} \approx 0.06982.

For a two-sided 5% test, \(z_{0.975}\approx1.960\). The matching interval is

0.15\pm1.960(0.06982) =0.15\pm0.1368 \approx[0.0132,\ 0.2868].

The lower endpoint is above zero, so the interval excludes zero. The Wald statistic is

z_W=\frac{0.15}{0.06982}\approx2.148.

Since \(2.148>1.960\), the matching test rejects the null hypothesis. Its approximate two-sided p-value is \(2[1-\Phi(2.148)]\approx0.0317\), also below 0.05. In context, the data provide evidence of a difference in success proportions between the groups under this model and approximation. The estimated difference is 0.15, or 15 percentage points; the interval estimates the difference as about 1.32 to 28.68 percentage points.

Worked Example: A Wald Decision Just Across the Boundary

Consider hypothetical independent samples with \(n_1=20\), \(x_1=16\), \(n_2=100\), and \(x_2=60\). Compare a 95% Wald interval and its matching Wald test with the pooled two-proportion score test.

The sample proportions are \(16/20=0.80\) and \(60/100=0.60\), so the estimated difference is \(d=0.20\). The Wald standard error and test statistic are

SE_W=\sqrt{\frac{0.80(0.20)}{20}+ \frac{0.60(0.40)}{100}} =\sqrt{0.008000+0.002400} =\sqrt{0.010400} \approx0.10198, \qquad z_W=\frac{0.20}{0.10198}\approx1.961.

The 95% Wald interval is

0.20\pm1.960(0.10198) \approx0.20\pm0.19988 \approx[0.00012,\ 0.39988].

Using the unrounded values, the interval’s lower endpoint is just above zero and the Wald statistic is just above 1.960. The matching Wald test therefore rejects at the 5% level, with an approximate p-value of 0.0499. This is a boundary-sensitive result: rounding the lower endpoint to 0.000 can conceal which side of zero it lies on.

The pooled score test does not use this Wald standard error. Under its null hypothesis of equal proportions, the pooled estimate is \(\hat{p}=(16+60)/(20+100)=76/120\approx0.6333\). Its null-based standard error and statistic are

SE_{\text{pooled}} =\sqrt{0.6333(1-0.6333)\left(\frac{1}{20}+\frac{1}{100}\right)} \approx0.11804, \qquad z_{\text{pooled}}=\frac{0.20}{0.11804}\approx1.694.

The score-test statistic does not exceed 1.960; its approximate two-sided p-value is 0.0902, so it does not reject at 5%. There is no contradiction in these results. The algebraic equivalence applies to a test and interval that use the same standard error; this pooled score test and unpooled Wald interval are different procedures.

Worked Example: When the Wald Standard Error Is Zero

Suppose hypothetical samples each have 20 observations. All 20 Group 1 outcomes are successes, while all 20 Group 2 outcomes are failures. Then \(\hat{p}_1=1\), \(\hat{p}_2=0\), and \(d=1\). Calculate the Wald interval and consider whether its matching Wald test is defined.

The unpooled standard error is

SE_W=\sqrt{\frac{1(1-1)}{20}+ \frac{0(1-0)}{20}}=0.

The Wald interval formula gives \(1\pm z_{1-\alpha/2}(0)=[1,1]\), which excludes zero. But the proposed Wald test statistic would be \(z_W=1/0\), which is undefined. The decision-equivalence derivation cannot be applied: it requires \(SE_W>0\), and division by zero is not permitted. This is a degenerate case for the ordinary Wald test, not a case in which an infinite Wald statistic has been calculated.

For comparison, the pooled estimate under the equality null is \((20+0)/(20+20)=0.50\), giving \(SE_{\text{pooled}}=\sqrt{0.50(0.50)(1/20+1/20)}\approx0.1581\). The pooled score statistic is \(1/0.1581\approx6.325\), so that particular test is defined. This comparison does not repair or define the Wald statistic: it uses a different standard error. A zero Wald standard error requires separate handling, and the point interval should not be treated as proof that the usual Wald approximation is reliable.

What the Connection Does—and Does Not—Say

The derivation is a statement about consistency between two matched procedures. It does not say that every confidence interval for a difference in proportions must reproduce every two-proportion test decision. To compare results, identify both the interval method and the test method, including their standard errors. The Wald interval estimates each group’s variability separately. The pooled score test estimates a common proportion under the equality null, then uses that pooled estimate to calculate the null standard error.

A disagreement between those procedures is therefore possible, especially near a decision boundary. It is not valid to say that one procedure must be wrong merely because their decisions differ. Instead, report which method was used and interpret its result accordingly. If the purpose is to check the test–interval connection, pair the Wald test with the Wald interval as defined here. Do not silently substitute a pooled standard error into one formula but not the other.

Also keep the inferential interpretation precise. A 95% confidence procedure produces intervals that cover the fixed population difference in 95% of repeated samples under its assumptions; it does not assign a 95% probability to the parameter being in this particular interval. A p-value measures how extreme the data, or more extreme data, would be under the null model; it is not the probability that the null hypothesis is true. Neither a rejection nor an interval excluding zero establishes that group membership caused the difference.

Common Mistakes

Comparing procedures with different standard errors. A pooled score test and an unpooled Wald interval need not give the same decision. The correction is to match methods before asserting equivalence.
Ignoring the boundary. If the statistic equals the critical value, the interval includes zero at an endpoint, and a strict \(p\text{-value}<\alpha\) rule does not reject. Retain enough digits to check the comparison.
Dividing by a zero estimated standard error. When \(SE_W=0\), the ordinary Wald statistic is undefined. State that limitation; do not call the result an infinite statistic or use the positive-standard-error derivation.
Overstating the conclusion. A test decision concerns evidence against a specified null under a model. It does not establish causation or give the probability that the null is true.

Key takeaway: For \(SE_W>0\), the two-sided Wald test of \(H_0:p_1-p_2=0\) at level \(\alpha\) rejects exactly when the matching \(100(1-\alpha)\%\) Wald interval excludes zero. This algebraic equivalence requires the same standard error in both procedures.

Check Your Understanding

Try these questions using the definitions and decision rule above.

In one sentence, explain why a two-sided Wald test and its matching confidence interval make the same decision about zero.
A matching 95% Wald interval for \(p_1-p_2\) is \([-0.04,\ 0.18]\). What does the matching test decide at \(\alpha=0.05\)?
Why can a pooled score test and an unpooled Wald interval disagree even when both use a 5% significance level or 95% confidence level?
What happens to the algebraic equivalence argument if \(SE_W=0\)?
If a test statistic equals \(z_{1-\alpha/2}\) exactly, does the strict \(p\text{-value}<\alpha\) rule reject, and does the matching interval include zero?