The Link Between Two-Sample Tests and Confidence Intervals

Why a Test and an Interval Can Agree

A two-sided test asks whether the data provide enough evidence against one specified difference between two population means. A confidence interval instead gives a range of plausible values for that difference. Although the questions sound different, the standard Welch test and its matching confidence interval use the same estimate, standard error, and degrees of freedom. Their decisions therefore agree exactly when they are matched correctly.

The word matching matters. A test at significance level \(\alpha\) corresponds to a two-sided confidence interval with confidence level \(100(1-\alpha)\%\). For example, a test at \(\alpha=0.05\) corresponds to a 95% interval. The hypotheses must refer to the same population difference, and both procedures must use the Welch method.

Definition: Let \(\mu_1-\mu_2\) be the difference between two population means, and let \(\Delta_0\) be the difference specified by the null hypothesis. A matching pair consists of the two-sided test \(H_0:\mu_1-\mu_2=\Delta_0\) against \(H_a:\mu_1-\mu_2\ne\Delta_0\), at level \(\alpha\), and the two-sided Welch confidence interval for \(\mu_1-\mu_2\) at confidence level \(100(1-\alpha)\%\).

The Shared Welch Quantities

For two independent samples, let \(\bar{x}_1,\bar{x}_2\) be the sample means, \(s_1,s_2\) the sample standard deviations, and \(n_1,n_2\) the sample sizes. The estimated difference is \(\bar{x}_1-\bar{x}_2\). Welch inference estimates its standard error by combining the two sample variances, without assuming that the population variances are equal.

SE=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}

The Welch degrees of freedom are approximated by the Satterthwaite formula. Writing \(A=s_1^2/n_1\) and \(B=s_2^2/n_2\) makes its structure easier to see.

df=\frac{(A+B)^2}{\dfrac{A^2}{n_1-1}+\dfrac{B^2}{n_2-1}}, \qquad A=\frac{s_1^2}{n_1},\quad B=\frac{s_2^2}{n_2}

Let \(t_{1-\alpha/2,df}\) denote the \(1-\alpha/2\) quantile of the \(t\) distribution with these degrees of freedom. The confidence interval and test statistic are

(\bar{x}_1-\bar{x}_2)\ \pm\ t_{1-\alpha/2,df}SE

t_{\text{obs}}= \frac{(\bar{x}_1-\bar{x}_2)-\Delta_0}{SE}

The interval is centered at the estimated difference. The test statistic measures how many estimated standard errors separate that estimate from the null difference. Both also use the same Welch degrees of freedom.

Conditions for Using the Correspondence

The algebraic link is exact for the stated Welch procedures, but its usefulness depends on whether the procedures are appropriate for the data. The two samples should come from independent groups, with observations independent within each group. For small samples, the distributions in each population should be approximately normal, with no severe skewness or extreme outliers. With sufficiently large samples, the sampling distributions of the means can often be treated as approximately normal provided there are no severe departures that undermine that approximation. The Welch method does not require equal population variances.

These conditions concern the reliability of the inference, not the algebra connecting the test and interval. If a condition is badly violated, the two calculations can still be made to agree while both provide an unreliable analysis. A researcher should therefore assess the study design and the data, rather than treating matching output as evidence that assumptions hold.

Why the Decisions Match

The key is that a two-sided test compares the absolute test statistic with a positive critical value, while the interval places that same critical value on either side of the estimate. The steps below show the equivalence for a general null difference \(\Delta_0\), not just a null difference of zero.

Write the test rejection rule.
At level \(\alpha\), reject \(H_0\) when \(\lvert t_{\text{obs}}\rvert>t_{1-\alpha/2,df}\). Equivalently, the observed statistic must be less than the negative critical value or greater than the positive critical value.

Substitute the test statistic.
Since \(SE>0\), the inequality \(\left|((\bar{x}_1-\bar{x}_2)-\Delta_0)/SE\right|>t_{1-\alpha/2,df}\) is equivalent to \(\left|(\bar{x}_1-\bar{x}_2)-\Delta_0\right|>t_{1-\alpha/2,df}SE\).

Rearrange the inequality.
The last inequality holds precisely when \(\Delta_0<(\bar{x}_1-\bar{x}_2)-t_{1-\alpha/2,df}SE\), or when \(\Delta_0>(\bar{x}_1-\bar{x}_2)+t_{1-\alpha/2,df}SE\).

Recognize the interval endpoints.
Those two bounds are the lower and upper endpoints of the matching confidence interval. Thus the test rejects precisely when \(\Delta_0\) lies outside the interval.

This is an algebraic equivalence, not a claim that a test and an interval answer identical questions. The test gives a decision about one specified value; the interval displays the values that would not be rejected by the corresponding two-sided tests at level \(\alpha\). The interval is also more informative about the range and direction of differences compatible with the data.

At the exact boundary, \(\lvert t_{\text{obs}}\rvert=t_{1-\alpha/2,df}\), the null difference is an endpoint of the interval and the two-sided p-value equals \(\alpha\). Using the strict rejection rule \(p<\alpha\), the test does not reject at that boundary, and the interval does not exclude the null value. This convention makes the word “exactly” precise; other decision conventions must handle equality consistently.

Worked Example: A 95% Interval and a Test at 5%

Suppose two independent groups each contain 11 hypothetical participants. Their measured recovery times, in days, have sample means \(\bar{x}_1=18\) and \(\bar{x}_2=13\), and sample variances \(s_1^2=s_2^2=22\). Assume the participants were sampled independently and that plots of the two groups show no severe skewness or extreme outliers. For this example, suppose the population distributions are sufficiently close to normal for Welch inference with these small samples. Test \(H_0:\mu_1-\mu_2=0\) against a two-sided alternative at \(\alpha=0.05\), and compare it with the matching 95% interval.

The observed difference is \(18-13=5\) days. The standard error is

SE=\sqrt{\frac{22}{11}+\frac{22}{11}} =\sqrt{2+2} =2\text{ days}

Here \(A=B=2\), so the Welch degrees of freedom are

df=\frac{(2+2)^2}{\dfrac{2^2}{10}+\dfrac{2^2}{10}} =\frac{16}{0.4+0.4} =20

The test statistic is

t_{\text{obs}}=\frac{(18-13)-0}{2} =\frac{5}{2} =2.5

For a two-sided test at \(\alpha=0.05\) with \(df=20\), the critical value is approximately \(2.086\). Since \(\lvert2.5\rvert>2.086\), the test rejects the null difference of zero. The two-sided p-value is approximately \(0.02123\), which is less than \(0.05\).

The matching interval uses the same \(SE\) and \(df\):

5\ \pm\ 2.086(2) =5\ \pm\ 4.172 =(0.828,\ 9.172)\text{ days}

The interval excludes zero, in agreement with the test. In context, the hypothetical data provide evidence at the 5% level that the population mean recovery times differ. The interval estimates the difference \(\mu_1-\mu_2\); its positive values indicate that the first group’s mean recovery time is higher, with the interval ranging from about 0.828 to 9.172 days.

Worked Example: When the Interval Includes the Null Value

Now consider hypothetical independent samples of 11 participants per group with sample means of 16 and 13 days and sample variances of 22 days squared in each group. Suppose the sampling and distribution conditions described above are reasonable. Test a zero mean difference at the 5% level and form the matching 95% interval.

The standard error and degrees of freedom remain \(SE=2\) days and \(df=20\), because the sample sizes and variances are unchanged. The observed difference is \(16-13=3\) days, giving

t_{\text{obs}}=\frac{3-0}{2}=1.5

Because \(\lvert1.5\rvert<2.086\), the test does not reject at the 5% level; its two-sided p-value is greater than \(0.05\). The interval is

3\ \pm\ 2.086(2) =3\ \pm\ 4.172 =(-1.172,\ 7.172)\text{ days}

Zero lies inside this interval, matching the test decision. The result does not establish that the population means are equal. It indicates that a zero difference remains among the values compatible with these data at the 95% confidence level.

Worked Example: Changing the Significance Level

Return to the first hypothetical data set, for which the estimated difference is 5 days, \(SE=2\) days, \(df=20\), and \(t_{\text{obs}}=2.5\). Now use \(\alpha=0.01\), which requires the matching 99% interval. The assumptions have not changed; only the decision threshold has changed.

The two-sided critical value for \(df=20\) at this level is approximately \(2.845\). Since \(\lvert2.5\rvert<2.845\), the test does not reject the null difference of zero at the 1% level. The p-value remains approximately \(0.02123\), now greater than \(0.01\).

The 99% interval is

5\ \pm\ 2.845(2) =5\ \pm\ 5.690 =(-0.690,\ 10.690)\text{ days}

This interval includes zero, just as the test does not reject zero. The first example’s 95% interval excluded zero, but the 99% interval includes it: increasing the confidence level widens the interval and makes rejection at the corresponding, smaller significance level harder.

Common Mistakes

Comparing unmatched levels. A 95% interval corresponds to a two-sided test at \(\alpha=0.05\), not a test at \(\alpha=0.01\). Match the confidence level to \(100(1-\alpha)\%\).
Using different standard errors or degrees of freedom. If one calculation uses Welch and the other uses a pooled procedure, the algebraic identity does not apply to that pair. Use the same method, \(SE\), and \(df\) for both.
Claiming that failure to reject proves equality. An interval that includes the null value says that value is not excluded by the matching procedure; it does not prove the population difference is zero.
Confusing exclusion with the direction of evidence. Excluding zero supports a nonzero difference in the direction indicated by the interval. It does not establish that one group caused the other group’s outcome.
Applying the two-sided rule to a different test. The equivalence established here is for a two-sided test and its matching two-sided interval. A test with a different alternative does not use this same two-sided cutoff.

Key takeaway: For the Welch test of \(H_0:\mu_1-\mu_2=\Delta_0\) against a two-sided alternative at level \(\alpha\), reject using the strict rule \(\lvert t_{\text{obs}}\rvert>t_{1-\alpha/2,df}\) if and only if \(\Delta_0\) is outside the matching \(100(1-\alpha)\%\) Welch confidence interval.

Check Your Understanding

Use the matching test-and-interval relationship to answer each question.

What confidence level corresponds to a two-sided test with \(\alpha=0.10\)?
A 95% Welch interval for \(\mu_1-\mu_2\) is \((-2.4,\ 6.1)\) hours. What is the matching test’s decision about a null difference of zero at the 5% level?
Why must the Welch test and confidence interval use the same standard error and degrees of freedom for the equivalence to hold?
If a null difference equals an endpoint of the matching interval, what is the corresponding absolute test statistic, and what is the p-value at that boundary?
A researcher reports a 99% interval alongside a two-sided test at \(\alpha=0.05\). Are the procedures matched? Explain.