A Worked Welch Test for Two Independent Means

From Group Summaries to a Test

When two groups have quantitative outcomes, their sample means may differ simply because of sampling variability. A Welch test evaluates whether the observed difference is sufficiently unusual under a specified null hypothesis about the two population means. It uses the sample sizes and sample standard deviations from each group separately, so it does not require the population variances to be equal.

The test is for independent groups: each observation belongs to one group, and the observations in one group are independent of those in the other. Before using the calculation, the design must support that independence, and the samples must be suitable for inference about their populations. The distribution of the sample mean difference must also be adequately approximated by the Welch t reference distribution. For small samples, this typically requires population distributions without severe skewness or extreme outliers; larger samples can make the approximation more reliable. Group summaries alone cannot establish these conditions, so information about the design and underlying data matters.

Definition: For two independent samples, let \(n_1,\bar{x}_1,s_1\) describe group 1 and \(n_2,\bar{x}_2,s_2\) describe group 2. The parameter of interest is \(\mu_1-\mu_2\), the difference between the two population means. A two-sided Welch test of no difference uses \(H_0:\mu_1-\mu_2=0\) and \(H_a:\mu_1-\mu_2\ne 0\).

The Welch standard error estimates the sampling variability of \(\bar{x}_1-\bar{x}_2\). The Welch t statistic measures the observed difference in estimated standard-error units under the null hypothesis. Its reference distribution is a t distribution with Satterthwaite degrees of freedom, which are generally not an integer.

Welch test formulas: The estimated standard error, degrees of freedom, and test statistic are

SE=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}, \qquad df=\frac{\left(\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}\right)^2} {\frac{(s_1^2/n_1)^2}{n_1-1}+\frac{(s_2^2/n_2)^2}{n_2-1}}, \qquad t=\frac{(\bar{x}_1-\bar{x}_2)-0}{SE}.

For the two-sided alternative, the p-value is the probability, assuming \(H_0\), of a t statistic at least as far from zero as the observed statistic, using the calculated degrees of freedom:

p\text{-value}=2P\left(T_{df}\ge |t|\right).

Why These Calculations Fit the Question

The standard error follows from independence. If the two sample means are independent, the variance of their difference is the sum of their variances. For a sample mean, the variance is the population variance divided by the sample size. Replacing the unknown population variances with the sample variances gives the estimated variance and its square root, the Welch standard error:

\operatorname{Var}(\bar{X}_1-\bar{X}_2) =\operatorname{Var}(\bar{X}_1)+\operatorname{Var}(\bar{X}_2) =\frac{\sigma_1^2}{n_1}+\frac{\sigma_2^2}{n_2} \quad\longrightarrow\quad SE=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}.

Because the population variances are estimated separately, the test statistic does not generally have one exact t distribution with a fixed integer number of degrees of freedom. The Satterthwaite formula approximates its reference degrees of freedom by matching the variability contributed by the two estimated variance terms. The resulting \(df\) specifies the t curve used to calculate the p-value; it is not an additional sample size.

Specify the parameter and hypotheses.
Name the group order and write the null and alternative for \(\mu_1-\mu_2\).

Calculate the estimated standard error.
Use each group’s sample variance divided by its sample size, then take the square root.

Calculate \(df\) and \(t\).
Use the Satterthwaite formula and divide the observed difference minus the null difference by \(SE\).

Find and interpret the p-value.
For a two-sided test, use both tails of the t distribution. Compare the p-value with the chosen \(\alpha\), then state a conclusion about the population means in context.

Worked Tests

Worked Example: Comparing Two Mean Completion Times

In a hypothetical task study, two independent groups complete the same activity. Completion time is measured in minutes. Group 1 has \(n_1=12\), \(\bar{x}_1=6.8\) minutes, and \(s_1=2.4\) minutes; group 2 has \(n_2=10\), \(\bar{x}_2=8.1\) minutes, and \(s_2=2.16\) minutes. Assume the groups were sampled independently and that the underlying observations are suitable for t inference. Test at \(\alpha=0.05\) whether the population mean completion times differ.

Hypotheses. Define \(\mu_1-\mu_2\) as the population mean time for group 1 minus that for group 2. Then \(H_0:\mu_1-\mu_2=0\) and \(H_a:\mu_1-\mu_2\ne0\). The two-sided alternative matches the question, which asks whether the means differ in either direction.

Check the setup. The stated design gives two independent groups, and completion time is quantitative. The problem stipulates that the observations are suitable for t inference; the summary statistics by themselves would not let us inspect the underlying distributions.

Standard error. Substitute the two sample standard deviations and sample sizes:

SE=\sqrt{\frac{2.4^2}{12}+\frac{2.16^2}{10}} =\sqrt{\frac{5.76}{12}+\frac{4.6656}{10}} =\sqrt{0.48+0.46656} =\sqrt{0.94656} \approx 0.9729\text{ minutes}.

Degrees of freedom and statistic.

df=\frac{(0.48+0.46656)^2}{\frac{0.48^2}{11}+\frac{0.46656^2}{9}} =\frac{0.89598}{0.02095+0.02419} \approx 19.85, \qquad t=\frac{(6.8-8.1)-0}{0.9729} =\frac{-1.3}{0.9729} \approx -1.336.

P-value and conclusion. For \(t=-1.336\) with approximately \(19.85\) degrees of freedom, the two-sided p-value is about \(0.197\). Since \(0.197>0.05\), we do not reject \(H_0\). These data do not provide statistically significant evidence at the 5% level that the population mean completion times differ. The observed sample difference is \(-1.3\) minutes, but it is not precise enough, relative to its estimated sampling variability, to establish a population difference using this test.

Worked Example: Evidence of a Difference in Mean Scores

In a hypothetical assessment, two independent groups take a quantitative score test. Group 1 has \(n_1=25\), \(\bar{x}_1=42.6\) points, and \(s_1=5.0\) points; group 2 has \(n_2=20\), \(\bar{x}_2=39.8\) points, and \(s_2=4.0\) points. Assume independent sampling and adequate conditions for t inference. At \(\alpha=0.05\), test whether the population mean scores differ.

Let \(\mu_1-\mu_2\) be the difference in population mean scores. The hypotheses are \(H_0:\mu_1-\mu_2=0\) versus \(H_a:\mu_1-\mu_2\ne0\). The groups are specified as independent and the outcome is quantitative; we assume the sampling and distribution conditions are adequate for this example.

The standard error is

SE=\sqrt{\frac{5.0^2}{25}+\frac{4.0^2}{20}} =\sqrt{1+0.8} =\sqrt{1.8} \approx 1.342\text{ points}.

The degrees of freedom and statistic are

df=\frac{(1+0.8)^2}{\frac{1^2}{24}+\frac{0.8^2}{19}} =\frac{3.24}{0.04167+0.03368} \approx 43.00, \qquad t=\frac{(42.6-39.8)-0}{1.342} =\frac{2.8}{1.342} \approx 2.087.

With \(t=2.087\) and approximately \(43.00\) degrees of freedom, the two-sided p-value is about \(0.043\). Since \(0.043<0.05\), we reject \(H_0\) at the 5% significance level. The hypothetical data provide evidence that the population mean scores differ. The sample mean for group 1 is higher, but the two-sided test alone does not establish that this difference is caused by group membership or that it will be important in practice.

Worked Example: A Nonsignificant Result With a Different Scale

A hypothetical laboratory compares a quantitative measurement, in milligrams per litre, between two independent groups. Group 1 has \(n_1=8\), \(\bar{x}_1=102\) mg/L, and \(s_1=12\) mg/L. Group 2 has \(n_2=9\), \(\bar{x}_2=96\) mg/L, and \(s_2=10\) mg/L. Assume independent sampling and that the observations meet the requirements for t inference. Test \(H_0:\mu_1-\mu_2=0\) against \(H_a:\mu_1-\mu_2\ne0\) at \(\alpha=0.05\).

The group structure supports an independent two-sample analysis, and the measured outcome is quantitative. The assumptions about the suitability of the sample distributions are stated for this hypothetical case; with samples this small, the summaries alone cannot verify them.

First calculate the standard error:

SE=\sqrt{\frac{12^2}{8}+\frac{10^2}{9}} =\sqrt{18+11.1111} =\sqrt{29.1111} \approx 5.395\text{ mg/L}.

Then calculate the degrees of freedom and test statistic:

df=\frac{(18+11.1111)^2}{\frac{18^2}{7}+\frac{11.1111^2}{8}} =\frac{847.46}{46.2857+15.4321} \approx 13.73, \qquad t=\frac{(102-96)-0}{5.395} =\frac{6}{5.395} \approx 1.112.

The two-sided p-value for \(t=1.112\) with approximately \(13.73\) degrees of freedom is about \(0.285\). Because \(0.285>0.05\), we do not reject the null hypothesis. The result does not provide statistically significant evidence at the 5% level of a difference in population mean measurement. This does not show that the means are identical; it says that this test, with these data and assumptions, does not provide strong evidence against equality.

Interpreting and Reporting the Result

A complete report connects the calculation to the research question. It identifies the groups and outcome, gives the direction of the sample difference, and reports the test statistic, degrees of freedom, p-value, and conclusion. The p-value describes how unusual a test statistic at least as extreme as the observed one would be if the null hypothesis were true and the model assumptions held. It is not the probability that the null hypothesis is true.

For example, the first worked test could be reported as: “The mean completion time was 1.3 minutes lower in group 1 than in group 2. A two-sided Welch test found \(t=-1.336\), \(df\approx19.85\), and \(p\approx0.197\); there was not significant evidence at the 5% level of a difference in population mean completion times.” This gives the observed direction without treating a nonsignificant result as proof of no difference.

The Welch procedure matters when the groups have different variances or sample sizes because its standard error keeps the two variance contributions separate. Its conclusion still depends on the design and the suitability of the t approximation. Calculating a p-value cannot repair dependent observations, biased sampling, or a serious failure of the model assumptions.

Common Mistakes

Using the wrong group order. If the statistic is based on \(\bar{x}_1-\bar{x}_2\), keep that same order in the parameter, hypotheses, and contextual interpretation. Reversing the order changes the sign of \(t\), though not a two-sided p-value.
Using a one-tail probability for a two-sided alternative. For \(H_a:\mu_1-\mu_2\ne0\), double the probability in the tail beyond \(|t|\).
Using equal-variance pooled formulas automatically. Welch’s test uses \(s_1^2/n_1\) and \(s_2^2/n_2\) separately; it does not combine the sample variances into a pooled estimate.
Calling a large p-value proof of equal means. A nonsignificant result means the data do not provide sufficient evidence against the null at the chosen level. It does not prove that the population means are identical.
Reporting arithmetic without checking rounding. Carry adequate precision through \(SE\), \(df\), and \(t\), then round the reported results. In the first example, \(-1.3/0.9729\) rounds to \(-1.336\), not \(-1.335\).

Key takeaway: A two-sided Welch test compares the observed difference in independent sample means with its estimated standard error, using Satterthwaite degrees of freedom to obtain the reference t distribution. Interpret its p-value in context and only when the design and inference assumptions are defensible.

Check Your Understanding

For each question, explain the reasoning as well as the answer.

For two independent groups, what parameter is tested by \(H_0:\mu_1-\mu_2=0\), and what does the alternative \(H_a:\mu_1-\mu_2\ne0\) say?
Why does the Welch standard error use a separate variance contribution from each group?
A two-sided Welch test gives \(t=-2.1\), \(df=18\), and \(p=0.050\). At \(\alpha=0.05\), what is the decision, and what should the conclusion say in context?
Why can’t group summaries alone establish that the observations are suitable for t inference?
A nonsignificant test is reported as “the two population means are equal.” Explain why this conclusion overstates what the test establishes.