Selection With Unequal Interim Sample Sizes

Why Unequal Interim Sample Sizes Matter

At an interim analysis, treatment arms may have different numbers of participants with evaluable outcomes. This can arise from planned allocation, recruitment patterns, or the timing of the analysis. The arm-specific treatment-effect estimates can still target the same clinical contrast, but their precision may differ. A ranking based only on estimated effects and a ranking that accounts for precision can therefore select different arms.

There is no universally correct ranking score independent of purpose. If the objective is to retain the arm with the largest estimated effect on the clinical scale, raw effects address that objective directly. If the objective gives weight to how precisely each effect has been estimated, a precision-aware score may be more relevant. These are different selection objectives, not competing calculations of one identical quantity.

Standard Error With Unequal Sample Sizes

For a concrete formulation, suppose the interim outcome is measured on a continuous scale, and arm \(i\) and the shared control have independent observations. Let \(\bar{x}_i\) and \(\bar{x}_0\) denote their sample means, \(n_i\) and \(n_0\) their analyzed sample sizes, and \(s_i\) and \(s_0\) their sample standard deviations. The estimated effect is the difference in means, \(\hat{\theta}_i=\bar{x}_i-\bar{x}_0\). Under independent sampling, its estimated standard error is:

This formula allows both unequal arm sizes and unequal observed variances. It assumes that observations are independent within and between groups and that the sample variances provide reasonable estimates of the outcome variances. For the usual mean-based interpretation, outcomes should be sufficiently well behaved for the estimated means and standard errors to be useful; severe outliers or strong skew in very small samples can make the approximation unreliable. The treatment and control groups must also be compared on the same outcome scale and at the same interim time point.

The formula follows from the variance of a difference. For independent sample means, \(\operatorname{Var}(\bar{x}_i)=\sigma_i^2/n_i\) and \(\operatorname{Var}(\bar{x}_0)=\sigma_0^2/n_0\). Therefore:

Replacing the unknown variances with \(s_i^2\) and \(s_0^2\), then taking the square root, gives the estimated standard error above. In a common-variance model, \(s_i^2\) and \(s_0^2\) may instead be replaced by an appropriate common variance estimate. The standard-error formula must match the variance model and analysis plan.

When several treatment effects use the same control observations, their estimates are correlated because each includes \(-\bar{x}_0\). For arms \(i\) and \(j\), under independent treatment groups and a common control, the covariance is \(\operatorname{Var}(\bar{x}_0)\). This shared-control dependence matters for joint probability calculations, but each arm’s marginal standard error still includes the control contribution shown above.

Choosing and Interpreting a Selection Score

A raw-effect rule is usually the more direct choice when the selection objective is “retain the arm with the largest estimated benefit on the clinical scale.” It is invariant to sample size: the ranking depends on the estimated effects, not how precisely those estimates were measured. That feature is also a limitation. With unequal sample sizes, a small, noisy arm can rank first because of a chance extreme estimate.

A \(Z_i\) rule accounts for the estimated precision, but its interpretation is different. It ranks signal relative to uncertainty, not treatment benefit in the original units. An arm with a modest estimated effect can outrank one with a larger estimated effect if the former has a sufficiently smaller standard error. Conversely, an arm with a large standard error can have a low score despite a clinically important estimate. A \(Z_i\) score is not itself a clinical effect, and ranking by it does not automatically provide a valid confirmatory test after selection.

The numerator’s direction must be defined before ranking. If lower outcomes are better, either define the effect as control minus treatment so beneficial effects are positive, or consistently rank the original difference in the beneficial direction. Do not compare signed scores whose signs have different clinical meanings.

Assumptions, Diagnostics, and Operational Choices

Before applying either score, verify that the estimates refer to comparable treatment contrasts: the outcome definition, measurement time, analysis population, and effect direction should align across arms. Confirm the analyzed sample sizes and the standard-error calculation. A nominally larger enrolled sample does not necessarily mean a more precise interim estimate if fewer outcomes are available or variability differs substantially.

Useful checks include examining the arm-specific sample sizes, standard deviations, and standard errors alongside the effects. A reversal between the raw and standardized rankings is a signal to inspect these quantities, not proof that one score is wrong. Very small groups can yield unstable standard deviations and extreme standardized scores. If the variance estimate is near zero, the ratio can become exceptionally large; investigate data coding, measurement, and the prespecified variance method rather than accepting the ranking mechanically.

A common-variance calculation can be efficient when a common variance assumption is justified, but it can misstate precision if arm variances differ. The separate-variance formula avoids imposing equal observed variances, although its estimates can also be unstable with small samples. The rule should follow the planned analysis rather than selecting whichever standard-error estimate produces a preferred ranking.

A common mistake is to reason, “Arm A has the larger estimated effect, so it must be the better arm,” without checking whether the decision is intended to prioritize magnitude or precision. The correction is to state the objective first, calculate the relevant score for every arm, and report both the effect and its uncertainty so the implications of the ranking are transparent. Similarly, a higher \(Z_i\) should not be described as a larger treatment benefit: it is a larger estimated effect relative to its standard error.

Raw ranking, precision-standardized ranking, and confidence-bound ranking answer related but distinct questions. A confidence-bound rule ranks a chosen bound and therefore incorporates both the estimate and uncertainty in a way that depends on the confidence level and direction. A prespecified raw-effect rule is simpler and clinically interpretable; a standardized rule explicitly values precision. None eliminates the need to specify tie handling, minimum-retention constraints, or other operational rules relevant to the trial.

Limitations of the Scores

Unequal sample sizes make precision differences especially visible, but no score can repair a poorly defined estimand or incomparable data. The plug-in standard error can be unreliable when sample sizes are very small, variance estimates are unstable, or the assumed independence structure does not hold. Shared-control estimates are dependent, so calculations of probabilities involving several arm rankings must account for that dependence.

Finally, the interim ranking is a selection decision, not a final efficacy conclusion. Because the arm is chosen using interim data, its observed estimate may be optimistic; later confirmatory inference must follow the trial’s planned procedures. The ranking rule should therefore be fixed before the interim results are examined and interpreted only for the objective it was designed to serve.