Information in Two-Sample Mean Comparisons

Why Information Depends on Both Groups

In a two-group comparison, the target is often the difference between population means, \(\Delta=\mu_1-\mu_2\). That difference is estimated by subtracting one sample mean from the other. The precision of this estimate depends on both groups: a noisy or small group contributes more uncertainty, even if the other group is large. For group sequential planning, it is therefore not enough to count participants without considering the variances and sample sizes in each group.

This tutorial considers independent samples with known population variances. The result gives a direct way to calculate the standard error and information for the mean difference at any look where the group-specific sample sizes are known. It also clarifies why a formula for one mean cannot be applied to a two-mean comparison without accounting for both sources of uncertainty.

Information here is the inverse sampling variance of the estimated contrast. Its units are the inverse of the squared outcome units: if the outcome is measured in points, \(V\) is in squared points and \(I\) is in inverse squared points. Information is not a count of participants; the same sample sizes can yield different information when the outcome variances differ.

Deriving the Variance and Information

Assume that observations are independent within and between groups, and that each group consists of identically distributed observations with its stated variance. The sample mean in group \(j\) has variance \(\sigma_j^2/n_j\). Because the group samples are independent, the covariance between their sample means is zero. The variance of a difference is therefore the sum of the two variances:

The sign in the difference does not create a subtraction in the variance: for any random variables \(A\) and \(B\), \(\operatorname{Var}(A-B)=\operatorname{Var}(A)+\operatorname{Var}(B)-2\operatorname{Cov}(A,B)\). Here the covariance term is zero by independence. If the observations are normally distributed, the difference is exactly normal with mean \(\Delta\) and the variance above. With sufficiently large samples, a normal approximation may be appropriate under less restrictive distributional assumptions, but the variance formula still requires independent observations and finite group variances.

Under the normal model with known variances, the reciprocal variance is also the Fisher information for the contrast after accounting for the two group means. One way to see this is to profile the normal likelihood over the nuisance location while holding the contrast \(\Delta\) fixed. The resulting log likelihood, up to a term not depending on \(\Delta\), is

Thus the likelihood curvature for the contrast is \(1/V\), matching the definition of information as inverse variance. This also shows why information increases as the contrast variance decreases. For testing \(H_0:\Delta=0\), the standardized statistic is \(Z=D/SE\); under the normal known-variance model, it has a standard normal distribution under the null.

Unequal Sample Sizes and Variances

The same expression applies when the groups have different sizes or different known variances. Each term is the variance contribution from one group. A smaller \(n_j\) increases that group’s contribution, while a larger \(\sigma_j^2\) also increases it. Neither contribution can be omitted just because the other group is larger or less variable.

When the known variances are equal, \(\sigma_1^2=\sigma_2^2=\sigma^2\), the general formula becomes \(V=\sigma^2(1/n_1+1/n_2)\). If the sample sizes are also equal to \(n\), then \(V=2\sigma^2/n\) and \(I=n/(2\sigma^2)\). If \(N\) denotes the total sample size in this balanced case, so that \(N=2n\), then \(V=4\sigma^2/N\). The distinction between \(n\), the size per group, and \(N\), the combined size, is essential.

Information at Sequential Looks

At an interim look \(k\), let \(n_{1k}\) and \(n_{2k}\) be the cumulative numbers observed in the two groups. With fixed known variances, calculate the contrast variance and information at that look using

This is information for the cumulative mean difference at look \(k\). In a design with equal variances and equal group sizes at each look, it grows in direct proportion to the per-group sample size. In other designs, do not assume that the information fraction equals the fraction of participants enrolled: calculate the variance at each look from both group counts and the variance assumptions.

Assumptions, Diagnostics, and Alternatives

The variance and information calculation is useful only insofar as its assumptions are credible. Check that the analyzed units are independent, the sample sizes refer to the observations contributing to the two means, and each variance is on the outcome’s squared measurement scale. If the same participant contributes to both conditions, or if observations are clustered, the covariance between the estimated means need not be zero; the simple sum of variance terms is then not justified. Likewise, missing outcomes or unequal variances over time can make the assumed variance inputs inaccurate.

• Check group-specific inputs. Confirm which variance belongs to which group and that the group counts are cumulative counts at the look being analyzed.
• Check independence. Paired measurements, repeated observations, or clustering require a variance calculation that accounts for dependence.
• Check the variance scale. If the outcome is measured in millimetres, use variances in millimetres squared, not standard deviations as if they were variances.
• Check whether variances are truly known. If they are estimated from trial data, the resulting information is estimated rather than fixed at the assumed value.

A common alternative when the variances are unknown is a two-sample \(t\) procedure. Under a common-variance model, the pooled sample variance replaces the known variance; when equal variances are not assumed, a Welch procedure uses separate estimated variances and an adjusted reference distribution. Those approaches address uncertainty from estimating variability. They do not make the known-variance formula correct when its variance assumptions are false.

Another mistake is to treat information as additive across looks without checking the design. The cumulative information \(I_k=1/V_k\) is calculated from the cumulative sample sizes. In the balanced equal-variance example, it is proportional to the per-group sample size, so increments in that sample size correspond to increments in information. In general, the reciprocal of a sum of variance contributions is not obtained by simply adding arbitrary sample counts or variance terms. Calculate the cumulative variance first, then invert it.

Finally, this method has limits. It does not account automatically for nuisance-parameter estimation, correlations, clustering, missing data, or variance changes; these require an appropriate model for the actual design. If the known variances are only planning assumptions, the computed information is conditional on those assumptions. Sensitivity calculations with plausible alternative variances can reveal how much the planned precision would change.