Calibration Workshop for Adaptive Sample Size Tests

From an Adaptation Rule to a Calibrated Test

An adaptive sample-size rule answers a design question: how much additional information will be collected after the interim data are observed? It does not, by itself, specify a valid final test. To obtain a complete procedure, the rule must be paired with a final statistic, a rejection boundary, and assumptions that justify the null distribution of that statistic.

This workshop constructs one such procedure. The sample size for stage two depends on the first-stage p-value, while the final test combines the two stagewise results using fixed inverse-normal weights. The key calibration question is whether the data-dependent choice of stage-two sample size changes the probability of falsely rejecting the null hypothesis.

Model, Rule, and Assumptions

Suppose the outcome is continuous, the treatment and control arms have a common known standard deviation \(\sigma\), and each stage has independent observations in the two arms. Let \(n_j\) be the number of participants per arm in stage \(j\), and let \(\bar{x}_{Tj}\) and \(\bar{x}_{Cj}\) be the corresponding sample means. We test for a positive treatment effect, \(H_0:\mu_T-\mu_C=0\) against \(H_a:\mu_T-\mu_C>0\). Under \(H_0\), define the stagewise statistics and p-values by

Here \(\Phi\) is the standard normal cumulative distribution function. Under the null model, each \(Z_j\) is standard normal. The statistics from the two stages are independent because stage-two observations are new and independent of stage one. We also assume that \(\sigma\) is known, the normal model applies, the one-sided direction was selected in advance, and the stagewise p-values are computed as specified. These assumptions give continuous, exactly uniform stagewise p-values under \(H_0\).

For a concrete rule, take \(n_1=40\) participants per arm and choose the stage-two sample size as follows: if \(p_1\leq 0.20\), set \(n_2=40\) per arm; otherwise set \(n_2=80\) per arm. Thus, a less promising interim result triggers the larger stage-two sample. The rule is prespecified, and the final combination weights will not change with \(p_1\).

Use fixed weights \(w_1=0.60\) and \(w_2=0.80\). Since \(w_1^2+w_2^2=0.36+0.64=1\), the inverse-normal statistic and final decision rule at one-sided level \(\alpha=0.025\) are

Equivalently, the combined p-value is \(1-\Phi(C)\), and rejection occurs when that value is at most \(0.025\). The weights are fixed in advance; the sample size rule changes how much stage-two data are collected, not how the two stagewise results are combined.

Why the Null Rejection Probability Is Calibrated

The crucial fact is conditional: under \(H_0\), for every possible first-stage result and every stage-two sample size selected by the rule, the independently collected stage-two statistic remains standard normal. In particular, conditioning on the stage-one data determines \(n_2\), but it does not select or alter the independent stage-two observations.

The independence assertion follows directly from the conditional distribution. For any sets \(A\) and \(B\),

The conditional probability in the middle is constant because the conditional distribution of \(Z_2\) is standard normal for every stage-one outcome. Thus the two statistics are independent. Their weighted sum is normal with mean zero and variance \(w_1^2+w_2^2=1\), which gives the stated rejection probability.

The same result can be seen by conditioning directly on \(Z_1=z\). The test rejects when \(Z_2>(1.960-0.60z)/0.80\). Under the null, its conditional rejection probability is therefore

Averaging this quantity over the standard normal distribution of \(Z_1\) gives the overall Type I error. Equivalently, because \(C\sim N(0,1)\),

Diagnostics, Alternatives, and Limitations

A calibration check should follow the actual algorithm from beginning to end: generate stage one under \(H_0\), apply the sample-size rule, generate independent stage-two data at the selected size, compute both stagewise p-values, and apply the fixed combination test. In simulation, the estimated rejection proportion should be compatible with \(0.025\), allowing for Monte Carlo error. For example, with one million simulated trials, the standard error at a true rejection probability of \(0.025\) is approximately \(\sqrt{0.025(0.975)/1{,}000{,}000}=0.000156\). A discrepancy much larger than ordinary simulation error warrants checking the implementation and assumptions.

The exact guarantee depends on the procedure actually matching its specification. Common failure points include using stage-two observations in the adaptation decision, changing the weights after seeing the interim result, reusing stage-one observations in the stage-two statistic, or using p-values that are not conditionally valid for the selected stage-two design. Any of these can invalidate the independence or conditional-distribution argument. If variance is estimated, outcomes are discrete, or nuisance parameters affect the stagewise null distribution, the simple exact normal argument may no longer apply; calibration must then be justified for that setting rather than assumed.

The earlier material on conditional error and combination tests provides two broad ways to protect Type I error after adaptation. This workshop uses a combination test with fixed weights: it is straightforward when independent, valid stagewise p-values are available. Conditional-error recalibration instead preserves a specified conditional rejection probability and can accommodate a suitably recalibrated second-stage test. A naive pooled test is not a safe substitute: its fixed-design reference distribution need not account for how the sample size was selected. These approaches are alternatives, not interchangeable calculations; the final rule must be calibrated as a whole.

The result also has limits. It guarantees Type I error under the stated null model and assumptions; it does not guarantee a particular power, make a sample-size rule scientifically sensible, or correct bias in the data collection. Nor does the rule ensure that a larger sample is always chosen when evidence is weaker unless that relationship is explicitly built into the rule. Keep the adaptation algorithm, stagewise analysis, weights, and decision boundary prespecified and reproducible.