Two-Sample Mean Differences

Why Two-Sample Information Needs Care

In a group sequential comparison, observations accrue in two groups and the mean difference is examined at planned looks. To describe the joint behavior of the statistics across those looks, we need more than a standard error at each look: we need a cumulative score whose covariance reflects how information accumulates. For two independent normal means, that construction is explicit, but it depends on using the variance of the difference, not the variance of either group mean alone.

We use a known-variance model so that the score and its information have exact normal distributions. The observations are independent within and between groups, and the samples at later looks contain the earlier observations. These assumptions make it possible to derive the canonical joint distribution directly.

Definition: At look $k$, let $X_{1i}\sim N(\mu_1,\sigma_1^2)$ and $X_{2i}\sim N(\mu_2,\sigma_2^2)$, independently, with known positive variances. There are $n_{1k}$ observations in group 1 and $n_{2k}$ in group 2. The parameter of interest is the mean difference $\theta=\mu_1-\mu_2$. For testing $H_0:\theta=\theta_0$, define $$ D_k=\bar{x}_{1k}-\bar{x}_{2k}-\theta_0,\qquad V_k=\frac{\sigma_1^2}{n_{1k}}+\frac{\sigma_2^2}{n_{2k}},\qquad I_k=\frac{1}{V_k}. $$ Here $V_k$ is the variance of the centered difference and $I_k$ is its information. The cumulative score and standardized statistic are $U_k=I_kD_k$ and $Z_k=U_k/\sqrt{I_k}=D_k/\sqrt{V_k}$.

The null value $\theta_0$ is subtracted so that the score is centered at zero under $H_0$. If the scientific null is no difference, then $\theta_0=0$. The information depends on both group sizes and both variances; it is not generally the total sample size or the sample size in one group.

Deriving the Canonical Joint Distribution

At a single look, $D_k$ is normally distributed with mean $\theta-\theta_0$ and variance $V_k$. Consequently, multiplying by $I_k$ gives the score distribution, while standardizing by $\sqrt{I_k}$ gives the distribution of the $Z$ statistic:

D_k\sim N(\theta-\theta_0,V_k),\qquad U_k\sim N\!\left(I_k(\theta-\theta_0),I_k\right),\qquad Z_k\sim N\!\left((\theta-\theta_0)\sqrt{I_k},1\right).

The multivariate structure follows from nesting. Suppose $j<k$, so that the observations at look $j$ are included among those at look $k$. For either group, the covariance between its early and later sample means is the variance of the later mean: the later mean contains all the early observations, and its covariance with the early mean is $\sigma_g^2/n_{gk}$. The groups are independent, so the covariance of the two centered differences is

\operatorname{Cov}(D_j,D_k) =\frac{\sigma_1^2}{n_{1k}}+\frac{\sigma_2^2}{n_{2k}} =V_k=\frac{1}{I_k}.

Because the scores are scaled versions of these differences, their covariance is $I_jI_kV_k=I_j$. Thus, for looks $j<k$, the standardized statistics have correlation $\sqrt{I_j/I_k}$. This is the canonical covariance pattern: the cumulative score has covariance equal to the information at the earlier look.

\operatorname{Cov}(U_j,U_k)=I_j,\qquad \operatorname{Corr}(Z_j,Z_k)=\sqrt{\frac{I_j}{I_k}}.

A short justification also establishes the independent-increment representation. The score increment from look $j$ to look $k$ is $U_k-U_j$. Its covariance with $U_j$ is $\operatorname{Cov}(U_j,U_k)-\operatorname{Var}(U_j)=I_j-I_j=0$. The scores are jointly normal, so zero covariance implies independence. Its variance is $\operatorname{Var}(U_k-U_j)=I_k-I_j$, and its mean is $(\theta-\theta_0)(I_k-I_j)$. Therefore:

U_k-U_j\sim N\!\left((\theta-\theta_0)(I_k-I_j),\,I_k-I_j\right),

provided $I_k>I_j$. Under the null, this increment has mean zero. Under a fixed alternative, its mean is the effect relative to the null multiplied by the added information. This result concerns increments of the cumulative score; the cumulative $Z$ statistics themselves are correlated, not independent.

Worked Examples at One and Several Looks

Worked Example: A Two-Sample Statistic at One Look

Suppose a hypothetical study compares two independent groups with known standard deviations $\sigma_1=4$ and $\sigma_2=4$. At one look, $n_1=20$, $n_2=20$, and the observed means are $\bar{x}_1=52.0$ and $\bar{x}_2=50.0$. Test the null difference $\theta_0=0$ by calculating the canonical statistic.

The observations are assumed independent and normally distributed with the stated known variances. The variance of the difference is $$ V=\frac{4^2}{20}+\frac{4^2}{20}=0.8+0.8=1.6, $$ so the information is $I=1/1.6=0.6250$. The centered observed difference is $D=52.0-50.0-0=2.0$ units. Therefore, $$ U=ID=0.6250(2.0)=1.250,\qquad Z=\frac{2.0}{\sqrt{1.6}}=1.581. $$

Under $H_0$, $Z\sim N(0,1)$. For a two-sided test, the normal-reference p-value is approximately $2[1-\Phi(1.581)]=0.114$. The observed difference is 2.0 units, and the data do not provide strong evidence against a zero mean difference at the conventional 0.05 level. The p-value is calculated under the null model; it is not the probability that the null hypothesis is true.

Worked Example: Information and Scores at Three Looks

Consider a hypothetical trial with known standard deviations of 4 units in each group, and equal group sizes at each look: 20, 40, and 60 observations per group. Suppose the observed group mean differences at those looks are 0.4, 0.6, and 0.5 units, respectively, and the null difference is zero. Calculate the information, score, and standardized statistic at each look.

At each look, $V_k=16/n_{1k}+16/n_{2k}$. Thus the variances are 1.6, 0.8, and $0.5333$, giving information values $I_k=0.6250,\ 1.250,\ 1.875$. The information increases as observations accrue. With $D_k$ equal to each observed mean difference, $U_k=I_kD_k$ and $Z_k=D_k/\sqrt{V_k}$:

Look	Group sizes	$V_k$	$I_k$	$D_k$	$U_k$	$Z_k$
1	20 and 20	1.600	0.6250	0.4	0.2500	0.3162
2	40 and 40	0.8000	1.250	0.6	0.7500	0.6708
3	60 and 60	0.5333	1.875	0.5	0.9375	0.6847

For example, at the third look, $Z_3=0.5/\sqrt{0.5333}=0.6847$. The information increments are $0.6250$ and $0.6250$, so the score increments have null variances $0.6250$ each. The observed score increments are $U_2-U_1=0.5000$ and $U_3-U_2=0.1875$. The fact that these are independent under the model does not mean the listed $Z$ statistics are independent: their correlations are $\sqrt{I_1/I_2}=0.7071$, $\sqrt{I_1/I_3}=0.5774$, and $\sqrt{I_2/I_3}=0.8165$.

Worked Example: Unequal Variances and Unequal Allocation

In a second hypothetical comparison, suppose $\sigma_1^2=9$, $\sigma_2^2=16$, and the group sizes progress from $(n_1,n_2)=(30,45)$ to $(60,90)$. The observed difference at the later look is 2.0 units, with $\theta_0=0$. Calculate information at both looks and the later standardized statistic.

At the first look, $$ V_1=\frac{9}{30}+\frac{16}{45}=0.3000+0.3556=0.6556,\qquad I_1=\frac{1}{0.6556}=1.525. $$ At the later look, $$ V_2=\frac{9}{60}+\frac{16}{90}=0.1500+0.1778=0.3278,\qquad I_2=\frac{1}{0.3278}=3.051. $$ The information has doubled because both group sizes have doubled while the variances and allocation ratio remain fixed. At the later look, the standard error is $\sqrt{0.3278}=0.5725$, and $$ Z_2=\frac{2.0}{0.5725}=3.493,\qquad U_2=I_2(2.0)=6.102. $$

The calculation uses the two variance contributions separately; replacing them with a pooled variance without justification would change the model. The observed standardized difference is 3.493 standard errors above the null value under the stated known-variance model.

Assumptions, Diagnostics, and Common Errors

The canonical result rests on specific conditions. The observations in each group must follow the stated normal model, have constant known variance within that group, and be independent across participants and groups. Sampling must be nested across looks, and group sizes must be fixed by the design rather than chosen in response to unmodeled outcomes. The contrast and null value must remain the same at each look. Under these conditions, the finite-sample normal distributions and covariance derivation above are exact.

In practice, check whether the recorded sample sizes and variance assumptions match the calculation, whether the two groups are genuinely independent, and whether each interim dataset includes the earlier observations. If the groups are paired, the variance of the difference must include the within-pair covariance; the independent-groups formula is then inappropriate. If variances are estimated rather than known, the exact normal score model used here no longer applies as written. Large-sample score methods may be considered, but their validity and information calculation require separate justification.

A frequent arithmetic and modeling error is to put an inappropriate variance under the square root. For a two-sample mean difference, the standard error is $\sqrt{\sigma_1^2/n_1+\sigma_2^2/n_2}$, not a standard deviation divided by the total sample size. For example, if the observed difference is 5 and its correct variance is 7, then the statistic is $5/\sqrt{7}\approx1.890$. Substituting $64/40+64/60$ instead gives $5/\sqrt{64/40+64/60}\approx3.062$. That erroneous calculation increases the standardized statistic by a factor of about 1.62; it does not more than double it. The correction is to use the variance components justified by the actual model and design, not merely numbers that resemble variances and sample sizes.

The main alternative in routine two-sample analysis is to estimate variances and use a two-sample $t$ procedure, often Welch’s method when equal variances are not assumed. That approach addresses uncertainty from estimating variances, whereas the present tutorial’s known-variance score has an exact normal canonical distribution. A pooled-variance $t$ procedure relies on equal population variances. Neither alternative can be inserted into this known-variance derivation without accounting for its changed distribution and covariance structure. The benefit of the canonical score here is the direct link between information, covariance, and independent increments; its limitation is that the known-variance normal assumptions may be unrealistic.

Key takeaway: For independent normal groups with known variances, the information at a look is the reciprocal of the variance of the mean difference. With nested sampling, cumulative scores have covariance equal to the earlier information, and score increments are independent normal variables with variance equal to the added information.

Check Your Understanding

Answer each question, then compare with the solutions below.

At a look with $n_1=25$, $n_2=50$, and known variances $\sigma_1^2=4$, $\sigma_2^2=9$, what are the variance and information of the mean difference?
If $I_1=2$ and $I_2=8$ at nested looks, what is $\operatorname{Corr}(Z_1,Z_2)$?
Under a fixed difference $\theta-\theta_0=1.5$, what is the mean of the score increment when information increases by 3?
Why does independence of score increments not imply independence of the cumulative $Z$ statistics?

Solutions. (1) $V=4/25+9/50=0.16+0.18=0.34$, so $I=1/0.34=2.941$. (2) The correlation is $\sqrt{2/8}=0.5000$. (3) The increment mean is $(\theta-\theta_0)\Delta I=1.5(3)=4.5$. (4) Each cumulative score includes the earlier score, so they share information: $\operatorname{Cov}(U_1,U_2)=I_1$. Standardizing does not remove that dependence; it gives correlation $\sqrt{I_1/I_2}$, which is $0.5000$ in this case.

What You'll Learn