Two Questions, Two Statistical Goals
When comparing two groups, a researcher may ask, “How large is the difference?” or “Is the difference inconsistent with a particular value?” These questions can use the same data and the same underlying model, but they have different goals. The first asks for an estimate; the second assesses evidence against a specified null value.
Choosing between a confidence interval and a significance test starts with the research question, not with which calculation seems more familiar. First name the parameter of interest, such as the difference in two population means or the difference in two population proportions. Then decide whether the goal is to describe plausible values for that parameter or evaluate a null claim about it.
For two independent groups, a parameter might be \(\mu_1-\mu_2\), the difference between their population means, or \(p_1-p_2\), the difference between their population proportions. In a paired study, the parameter is typically the population mean of the within-pair differences. The research design and outcome determine which parameter and procedure apply; the goal of estimation versus testing determines which inferential result answers the question.
Read the Question Before Choosing the Result
Language such as “estimate,” “how much,” “by how many,” or “give a range of plausible values” usually signals an interval goal. A useful answer should report the estimate’s direction, its units, and the interval’s uncertainty. A confidence interval does not establish that every value in the range is equally plausible, nor does it assign a probability to the fixed parameter after the data have been collected.
Language such as “test the claim,” “assess evidence against,” or “is there evidence that the parameter differs from this value?” signals a test goal. A test needs a specified null hypothesis, including the parameter value being assessed. Its p-value measures how unusual the data, or data more extreme in the specified direction, would be if that null model were true and the test assumptions held. A small p-value is evidence against the null; it is not the probability that the null hypothesis is true.
The decision to use an interval or a test does not determine the specific procedure. For example, a two-sample t procedure may be used for independent groups with a quantitative outcome, while a paired t procedure uses within-pair differences. The earlier material on choosing Welch and pooled t procedures addresses that choice for independent-group means. For proportions, a proportion procedure is needed. In every case, check that the design and data meet the chosen procedure’s assumptions before relying on its result.
| Question’s main goal | Result to choose | What the result answers |
|---|---|---|
| Estimate the size of a two-group difference | Confidence interval | Which parameter values are compatible with the data and stated confidence level? |
| Assess a specified null value | Significance test | How much evidence do the data provide against that null value? |
| Estimate the difference and assess a null value | Both may be useful | What is the estimated size and precision, and how does the evidence compare with the null? |
Worked Example: Estimating a Difference in Mean Times
In a hypothetical study, two independent random samples of adults complete the same task. The task times, in minutes, are reasonably symmetric in both groups with no marked outliers. Group 1 has \(n_1=25\), \(\bar{x}_1=18.4\) minutes, and \(s_1=4.0\) minutes. Group 2 has \(n_2=25\), \(\bar{x}_2=16.1\) minutes, and \(s_2=3.5\) minutes. The research question is: “By how much do the population mean task times differ?” Choose the inferential result.
The question asks for the size of a difference, so a confidence interval for \(\mu_1-\mu_2\) is appropriate. Because the samples are independent and the group population variances are not assumed equal, use a Welch two-sample t interval. The independent sampling and reasonable within-group distributions support this procedure; the samples are not paired.
The sample difference is \(18.4-16.1=2.3\) minutes. Its estimated standard error is
Welch’s degrees of freedom are approximately \(47\), giving a 95% critical value of about \(2.012\). Thus the interval is
The interval estimates that Group 1’s population mean task time is between about 0.162 and 4.438 minutes greater than Group 2’s. In repeated sampling under the assumptions, intervals constructed this way would contain the true mean difference about 95% of the time. The interval does not mean there is a 95% probability that this particular fixed difference lies in these endpoints.
Worked Example: Testing a Specified Mean Difference
Use the same hypothetical samples and assumptions, but now suppose the research question is: “Is there evidence that the population mean difference is not 1 minute?” A two-sided test has been specified. Here the goal is not simply to estimate the difference; it is to assess the particular null value \(\mu_1-\mu_2=1\) minute.
The hypotheses are \(H_0:\mu_1-\mu_2=1\) minute and \(H_a:\mu_1-\mu_2\ne 1\) minute. The Welch standard error and degrees of freedom are the same as in the interval example. The test statistic measures the distance between the observed difference and the null value in standard-error units:
With approximately 47 degrees of freedom, the two-sided p-value is about \(0.227\). At the 0.05 significance level, this p-value is not small enough to provide evidence against the specified null value. The appropriate conclusion is that these data do not provide convincing evidence that the population mean difference differs from 1 minute; this does not prove that the null value is true.
Notice how the goal changed even though the data did not. The first question called for an interval describing plausible values. This question calls for a test evaluating a particular value. The test does not replace the estimate: reporting the estimated difference and interval can still convey the size and precision of the observed effect.
Worked Example: Choosing an Interval for a Difference in Proportions
In a hypothetical survey, 112 of 200 randomly sampled members of Group 1 and 81 of 180 randomly sampled members of Group 2 report using a particular service. The two samples are independent. The question is: “How much higher or lower is the population use proportion in Group 1 than in Group 2?” Each sample has at least 10 reported users and 10 nonusers, supporting a normal-approximation interval for the difference in proportions.
The sample proportions are \(\hat{p}_1=112/200=0.56\) and \(\hat{p}_2=81/180=0.45\). Since the goal is to estimate the size of the difference, use an unpooled confidence interval for \(p_1-p_2\). Its standard error is
A 95% interval using \(z^*=1.96\) is
The interval estimates that the Group 1 population use proportion is about 0.99 to 21.01 percentage points higher than the Group 2 proportion. This is an interval estimate, not a test of a specified null. If the question instead asked whether the proportions differed from a stated null value, a significance test for two proportions would answer that testing goal.
Why Intervals and Tests Can Agree
For a two-sided test of a parameter value using the same model and standard error as an interval, the two procedures are closely connected. A confidence interval shows which null values would not be rejected by the corresponding test at a particular significance level. This relationship explains why an interval can help interpret a test, but it does not make the goals identical: an interval describes a range, while a test evaluates a specified value.
For example, in the mean-time calculation, the 95% interval was approximately \((0.162,4.438)\) minutes. The specified null value of 1 minute lies inside that interval, consistent with the two-sided test not rejecting at \(\alpha=0.05\). Had the tested null value been outside the interval, the corresponding two-sided test would reject at that level, subject to using the same procedure and assumptions.
This connection follows directly from the test statistic. For a hypothesized value \(\delta_0\), a two-sided t test rejects at level \(\alpha\) when
where \(\hat{\delta}\) estimates the parameter difference and \(t^*\) is the critical value for the corresponding confidence level \(1-\alpha\). Rearranging the non-rejection condition gives
The right-hand side says exactly that the null value lies inside the corresponding confidence interval. The equivalence depends on matching the test and interval: they must use the same parameter, model, assumptions, and two-sided procedure. It should not be assumed for procedures that do not share this construction.
Common Mistake: Choosing by the Data Rather Than the Goal
A common error is to see two groups and immediately decide to run a significance test. The reasoning might be, “There are two groups, so I need to test whether they differ.” But the number of groups identifies the structure of the parameter and possible procedures; it does not tell you whether the question is about estimation or testing. If the research question asks for the size and precision of the difference, a test alone does not answer it. Choose an interval. If the question asks for evidence against a particular null value, choose a test.
Another error is to interpret a non-significant result as proof that the groups are equal. A test that does not reject the null only indicates that the data do not provide sufficiently strong evidence against it at the chosen significance level. To assess the range of differences compatible with the data, examine an interval and its width. Even then, a wide interval can include both small and practically important differences.
Check Your Understanding
For each question, decide whether an interval, a significance test, or both would best serve the stated goal, and give a brief reason.
- A researcher wants a range of plausible values for the difference in average recovery time between two independent treatment groups. Which result should be emphasized?
- A survey asks whether a population difference in service-use proportions is inconsistent with a specified value of zero. Which inferential result directly addresses the question?
- Why does a p-value of 0.23 not mean there is a 23% probability that the null hypothesis is true?
- A two-sided test at the 0.05 level and its matching 95% confidence interval use the same model. What does it mean if the null value lies inside that interval?
- A test does not reject a null hypothesis of no difference. Does this establish that the groups are equal? Explain.