Boundary Calibration with Small-Sample t Statistics

Why small-sample calibration needs the joint t distribution

In a multi-arm, multi-stage (MAMS) trial, a boundary is crossed if one or more treatment statistics exceed their prespecified thresholds at one or more interim or final looks. The familywise error rate (FWER) is the probability of at least one false rejection among the hypotheses being protected. When arm sizes are small, replacing the t statistics by standard normal statistics can misstate this probability: t tails are heavier, and the difference can matter at the extreme tail probabilities used for sequential boundaries.

The issue is not only that each statistic has a t distribution. The statistics are jointly dependent. At a given look, treatment comparisons may share a control group and a variance estimate. Across looks, cumulative means and cumulative pooled variance estimates are also dependent. Calibration must use the distribution of the whole vector of statistics, not a collection of marginal critical values treated as though the statistics were independent.

The introductory material on nuisance-parameter robust boundaries motivates protection across nuisance parameters. Here the focus is the finite-sample distribution under a specific, transparent model. The resulting calibration is exact only to the extent that the model and analysis match the trial.

Model and exact marginal t statistics

Definition: Suppose there are $K$ treatment arms and one common control arm. Within each arm, observations are independent and normally distributed with a constant arm-specific mean and a common variance $\sigma^2$. Observations are also independent between arms. At look $j$, let $n_{ij}$ be the cumulative sample size in arm $i$, where $i=0$ denotes control, and let $\bar{Y}_{ij}$ be its cumulative sample mean. Define $$ SSE_j=\sum_{i=0}^{K}\sum_{r=1}^{n_{ij}}(Y_{ir}-\bar{Y}_{ij})^2,\qquad df_j=\sum_{i=0}^{K}n_{ij}-(K+1),\qquad s_{p,j}^2=\frac{SSE_j}{df_j}. $$ For treatment $i$, the statistic for testing $H_{0i}:\mu_i-\mu_0\leq 0$ against $H_{ai}:\mu_i-\mu_0>0$ is $$ T_{ij}=\frac{\bar{Y}_{ij}-\bar{Y}_{0j}}{s_{p,j}\sqrt{1/n_{ij}+1/n_{0j}}}. $$

At a fixed look, under the boundary null $\mu_i-\mu_0=0$, the numerator contrast is independent of the pooled residual sum of squares under this normal, common-variance model. Consequently, $T_{ij}$ has a Student t distribution with $df_j$ degrees of freedom. The same result holds for the marginal distribution of a contrast under its null when other arm means differ: within-arm residual variation is independent of the arm sample means in the normal model.

This marginal result does not make the full collection of statistics independent. Two comparisons at the same look share the control mean and pooled variance estimate. At successive looks, the samples are nested, so the cumulative contrast estimates are correlated. More importantly, the cumulative pooled variance estimates are built from the observations at each look and are generally dependent on the vector of cumulative contrasts across looks.

Worked Example: A Single-Look Small-Sample t Boundary

Suppose a hypothetical trial compares one treatment with control at one look, with five independent observations per arm. Assume normal outcomes and a common variance. The observed treatment-minus-control difference is 3.2 points, and the pooled within-arm standard deviation is 4.0 points. Test for a positive treatment effect at one-sided $\alpha=0.05$.

There are $5+5-2=8$ residual degrees of freedom. The estimated standard error is $$ SE=4.0\sqrt{\frac{1}{5}+\frac{1}{5}} =4.0\sqrt{0.4} \approx 2.530\text{ points}. $$ Thus $$ T=\frac{3.2}{2.530}\approx 1.265. $$ The one-sided 5% critical value is approximately $t_{8,0.95}=1.860$. Since $1.265<1.860$, this result does not cross the boundary. The comparison uses a t boundary rather than a normal boundary because the variance is estimated from only eight residual degrees of freedom.

The calculation is exact under the stated model when the null difference is zero. It is not a claim that the model is automatically appropriate: common variance, normal observations and the prespecified analysis remain assumptions to assess.

Sequential FWER as a joint probability

Let $c_{ij}$ be the upper boundary for treatment $i$ at look $j$, and let $I$ be a set of hypotheses that are all true. The probability of at least one false rejection among that set is the probability that any of its statistics crosses at any look. A joint calibration chooses boundaries so that this probability is at most the desired error level. For a common boundary $c$, the defining probability under a specified null configuration is

P\left(\max_{i\in I,\;j} T_{ij}\geq c\right).

The maximum is taken over the entire collection of looks and hypotheses being calibrated. If boundaries vary by look, the corresponding event is $T_{ij}\geq c_{ij}$ for at least one pair $(i,j)$. This is a joint-tail calculation: in general, adding marginal crossing probabilities is not the exact FWER, because crossing events overlap.

For example, at a single look with two treatment arms and equal sample sizes $n$, the two contrast numerators have correlation $1/2$ under their joint null: each contrast contains the same control mean, with the same negative sign. With a shared pooled variance, the pair of t statistics has a multivariate t distribution with this correlation matrix and the common residual degrees of freedom. For repeated looks, however, one must represent the cumulative means and pooled residual sums of squares together. Treating each look’s denominator as an independent chi-square quantity does not give the exact joint distribution.

Why the cumulative pooled variance cannot be separated across looks

A useful identity shows why the denominator needs careful treatment. Consider one arm with two independent increments, each of size $m$. Let their sample means be $\bar{Y}_A$ and $\bar{Y}_B$, and let their within-increment sums of squares be $SSE_A$ and $SSE_B$. The cumulative sample contains $2m$ observations. Its residual sum of squares is

SSE_{\mathrm{cum}} =SSE_A+SSE_B+\frac{m}{2}(\bar{Y}_A-\bar{Y}_B)^2.

The final term is the between-increment contribution. It follows by writing each observation’s deviation from the cumulative mean as its deviation from its own increment mean plus the increment mean’s deviation from the cumulative mean; the cross-products sum to zero within each increment. The term cannot generally be omitted.

Worked Example: The Between-Increment Term Changes the Denominator

In a hypothetical two-stage design, an arm has $m=4$ observations in each increment. Suppose its first- and second-increment sample means are 12 and 14 units, and the within-increment sums of squares are 30 and 26 units squared. Its cumulative residual sum of squares is $$ SSE_{\mathrm{cum}}=30+26+\frac{4}{2}(12-14)^2 =56+8=64\text{ units}^2. $$ Simply adding the within-increment residual sums would give 56, which is too small for this realized data set.

The 8-unit-squared term depends on the difference between the increment means. In a treatment arm, the first-increment mean contributes to the early treatment-control contrast. Therefore the later cumulative pooled variance estimate, which includes such terms across arms, is not in general independent of the vector of earlier and later contrast numerators. In particular, representing cumulative denominators as sums of independent within-increment chi-square components misses both this dependence and the between-increment contributions. Exact sequential calibration must retain them.

How to obtain a joint small-sample boundary

Under the stated model, a direct route is to generate or integrate over the raw observations at all planned looks, form the cumulative means and pooled residual sums of squares exactly as the trial analysis will, and then calculate the probability of crossing the candidate boundaries. A numerical root search can adjust the boundaries until the chosen joint error constraint is met. For low-dimensional designs, numerical integration may be feasible; for more complicated designs, Monte Carlo calibration is often more practical. Monte Carlo error should be reported and small enough relative to the intended error rate.

Specify the data model and analysis.
State the arm means and common variance model, sample sizes at every look, test direction, and the exact pooled-variance calculation.

Generate the complete data path.
Under the null configuration being calibrated, generate independent normal outcomes by arm and increment, retaining observations from earlier increments at later looks.

Recompute each cumulative statistic.
At every look, calculate cumulative arm means, the cumulative pooled residual sum of squares, its degrees of freedom, and all $T_{ij}$.

Estimate and solve the crossing probability.
Record whether any statistic crosses its boundary, estimate the resulting probability, and adjust the boundary vector to meet the target error constraint.

For a closed-testing procedure, the relevant family of hypotheses and intersection tests must also be respected. A calibration for the global null alone does not by itself establish strong FWER control under every configuration of true and false hypotheses. Under a model with subset pivotality, the joint null distribution for a subset can be used for its intersection test; where that property does not hold, calibration must address the relevant nuisance configurations or use a valid robust procedure. These are properties to establish for the actual statistic and design, not assumptions to infer from a successful global-null simulation.

Worked Example: Setting Up a Two-Arm Joint Calibration

Consider one look in a hypothetical three-arm trial: two treatments and a shared control, with five observations in each arm. Under the joint null for both treatment contrasts, the pooled residual degrees of freedom are $$ df=15-3=12. $$ With equal arm sizes, the two standardized contrast numerators have correlation $1/2$. The exact joint t representation is $$ (T_1,T_2)=\frac{(Z_1,Z_2)}{\sqrt{U/12}}, \qquad U\sim\chi^2_{12}, $$ where $(Z_1,Z_2)$ is a mean-zero bivariate normal vector with unit variances and correlation $1/2$, independent of $U$. This representation applies at this single look because the common pooled residual sum of squares is independent of the vector of arm means under the normal model.

For a common upper boundary $c$, the familywise crossing probability is therefore $$ P(\max(T_1,T_2)\geq c) =1-\int_0^\infty \Phi_2\left(c\sqrt{u/12},c\sqrt{u/12};\rho=\frac12\right) f_{\chi^2_{12}}(u)\,du, $$ where $\Phi_2(a,b;\rho)$ is the bivariate standard normal probability of being at most $a$ and $b$, with correlation $\rho$, and $f_{\chi^2_{12}}$ is the chi-square density. To calibrate at FWER 0.05, solve this integral equation with the left-hand crossing probability set to 0.05. The calculation uses $df=12$ and $\rho=1/2$, rather than two independent marginal t probabilities.

A conservative check is Bonferroni: using the single-test one-sided level $0.05/2=0.025$ gives boundary $t_{12,0.975}\approx2.179$, and the union bound guarantees FWER no greater than 0.05. The joint-calibrated common boundary can be found by evaluating the displayed integral; the Bonferroni value is a safe comparison, not a substitute for that joint calculation.

Alternatives, diagnostics, and limitations

There are two practical alternatives to direct joint calibration. A Bonferroni allocation assigns marginal error allowances whose sum is at most the target FWER, using the appropriate t quantile at each look. It is valid without modeling the dependence among crossing events, provided each marginal test is valid under its null, but it can be conservative. A normal approximation is simpler, but with small degrees of freedom it can produce boundaries that are too low for the intended tail probability. It should not be treated as exact merely because the test statistic is standardized.

Calibration should be checked against the analysis actually planned. Useful diagnostics include:

Confirm that the residual degrees of freedom at each look equal the cumulative sample size minus the number of fitted arm means in the stated model.
Check that cumulative pooled sums of squares include between-increment mean terms and are computed from the full cumulative data.
Compare the calibrated crossing probability with an independent simulation using the same raw-data model, and quantify Monte Carlo uncertainty.
Verify control sharing, allocation ratios, look timing and the handling of dropped arms match the assumptions used in calibration.
Check more than the global null when strong control is required and subset pivotality has not been justified.

The exact t approach depends on independent normal observations and a common variance for the pooled estimator. Substantial heteroscedasticity, skewness, data-dependent look timing, adaptive variance re-estimation, missing outcomes or a changed analysis can invalidate the stated joint law. In such settings, a t-based numerical calibration is not automatically robust. The model or statistic may need to change, and error control must be re-established under that changed design.

Common pitfall: It is incorrect to say that later pooled variance estimates are independent chi-square increments and then combine them with a correlated normal vector of stagewise numerators. Cumulative pooled residual sums include between-increment terms that depend on increment means. Use the joint distribution induced by all cumulative means and pooled residual sums of squares, or specify and justify a different variance estimator with its own degrees of freedom and joint law.

Key takeaway: Small-sample sequential calibration is a joint-distribution problem. The marginal t law supplies each look’s reference distribution, but the boundary must account for shared controls, nested observations, and the dependence created by cumulative pooled variance estimation.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions.

In a two-arm trial with five observations per arm at one look, what are the pooled residual degrees of freedom? Solution: $5+5-2=8$.
Why is the cumulative pooled residual sum of squares not generally the sum of the within-increment residual sums? Solution: It also contains between-increment terms such as $\frac{m}{2}(\bar{Y}_A-\bar{Y}_B)^2$.
What event defines FWER for boundaries $c_{ij}$? Solution: At least one true-null statistic crosses at least one look’s boundary: $T_{ij}\geq c_{ij}$ for some true-null $i$ and look $j$.
Why can a Bonferroni t-boundary be valid but conservative? Solution: The union bound limits the probability of any crossing by the sum of marginal probabilities, but it does not account for overlap between crossing events.
Name two assumptions supporting the exact marginal t result. Solution: Independent normal observations and a common variance estimated by the pooled within-arm residual sum of squares.