Power Sensitivity to Prespecified Stage Weights

Why Fixed Weights Matter

An inverse-normal combination test brings together evidence from separate stages using weights chosen in advance. Those weights do not change the test’s type I error merely because one stage is larger than another, provided the test is correctly calibrated. They do, however, affect how strongly each stage’s evidence contributes to the combined statistic. Consequently, two valid tests using different prespecified weights can have different power—sometimes substantially so when stage sizes are unequal or treatment effects change across stages.

This tutorial focuses on that sensitivity. The goal is not to choose a weight that is optimal for one particular alternative, but to understand how a fixed choice behaves across plausible information allocations and effect patterns. The calculations below use a one-sided test in a specified direction; for a two-sided test, use the two-sided critical value and define the stage statistics consistently with that test.

Definition: Let $Z_1,\ldots,Z_K$ be oriented stagewise $Z$-statistics, with larger values supporting $H_a$. An inverse-normal combination statistic with prespecified weights $w_1,\ldots,w_K$ is $$ Z_C=\sum_{i=1}^{K}w_i Z_i,\qquad \sum_{i=1}^{K}w_i^2=1. $$ For a one-sided level-$\alpha$ test, reject $H_0$ when $Z_C>z_{1-\alpha}$, where $z_{1-\alpha}$ is the $1-\alpha$ quantile of the standard normal distribution. Weights are commonly nonnegative, although the essential variance condition is the sum-of-squares constraint.

Assumptions and the Power Relationship

For the simple calculations in this tutorial, assume that under $H_0$, each stage statistic is standard normal and the stage statistics are independent. In an adaptive design, a corresponding condition is that each new stage’s null evidence is valid conditional on the information used to make the adaptation. Under these conditions, the fixed-weight combination statistic has a standard normal null distribution. The stage data must support the stated stagewise tests, and the weights and rejection rule must be specified rather than selected after inspecting the stage results.

For power calculations, use a normal approximation under a specified alternative. Let $I_i$ denote the information in stage $i$, measured on the scale for which the stage statistic has approximate mean $\delta_i\sqrt{I_i}$. Here $\delta_i$ is the stage-specific effect in the corresponding standardized units. Assume, for this calculation, that $Z_i$ are independent with variance approximately one and means $m_i=\delta_i\sqrt{I_i}$. Then $Z_C$ has approximate mean $\sum_i w_i m_i$ and variance one. Its approximate power is therefore

\operatorname{Power}\approx P(Z_C>z_{1-\alpha}\mid H_a) =\Phi\!\left(\sum_{i=1}^{K}w_i\delta_i\sqrt{I_i}-z_{1-\alpha}\right).

This expression makes the distinction between calibration and power explicit. Under the null, the weights determine the variance through $\sum_i w_i^2$, not the alternative mean. Under the alternative, the weighted mean depends on how the weights align with the stage-specific signals $\delta_i\sqrt{I_i}$. A valid test can therefore have low power without having a type I error problem.

Why Weight–Signal Alignment Affects Power

Write the vector of stage means as $m=(m_1,\ldots,m_K)$ and the weight vector as $w=(w_1,\ldots,w_K)$. With normalized weights, the combined mean is the inner product $w\cdot m$. By the Cauchy–Schwarz inequality,

w\cdot m\leq \|w\|\,\|m\|=\|m\|.

If all components of $m$ are positive and weights may be chosen to match the specified alternative, equality holds when $w=m/\|m\|$. This is a concise explanation of why a weight choice aligned with the assumed signal can maximize power for that particular alternative. It is not a guarantee of good power for other alternatives: if the actual stage effects have a different pattern, the inner product can be smaller. The resulting loss is a loss of power, not automatically a loss of type I error control.

Worked Example: Unequal Stage Sizes Under a Common Effect

Suppose two independent stages analyze a continuous outcome with known unit variance. There are 40 observations in stage 1 and 160 in stage 2, so $I_1=40$ and $I_2=160$. Consider the hypothetical common standardized effect $\delta=0.20$, with a one-sided test at $\alpha=0.025$. Compare equal weights with weights proportional to the square roots of the stage information.

The stage means under this alternative are $m_1=0.20\sqrt{40}=1.265$ and $m_2=0.20\sqrt{160}=2.530$. The one-sided critical value is approximately $z_{0.975}=1.960$.

With equal weights, $w_1=w_2=1/\sqrt{2}=0.7071$, so the combined mean is

$$ w_1m_1+w_2m_2 =0.7071(1.265)+0.7071(2.530) =2.683. $$

The approximate power is $\Phi(2.683-1.960)=\Phi(0.723)\approx0.765$. For information-proportional weights, $w_1=\sqrt{40/200}=0.4472$ and $w_2=\sqrt{160/200}=0.8944$. These also have squared weights summing to one. Their combined mean and power are

0.4472(1.265)+0.8944(2.530)=2.828, \qquad \Phi(2.828-1.960)=\Phi(0.868)\approx0.807.

Under this common-effect model, information-proportional weights give about 4.2 percentage points more power. Both tests use the same level-$0.025$ critical value and both have the same null calibration under the stated assumptions. The difference arises because the second stage carries more information and contributes a larger stagewise mean.

Effect Patterns Can Reverse the Comparison

Stage size alone does not determine a universally best weight. A larger stage typically provides more information about a common effect, but a stage can have a smaller or larger effect than another. In that case, the vector of stage signals—not just the information allocation—determines the power comparison. A design that emphasizes the larger stage can be less powerful if the effect is concentrated in the smaller stage.

Worked Example: A Larger Effect in the Smaller Stage

Consider two stages with $I_1=I_2=100$, again using independent stage statistics and a one-sided level-$0.025$ test. Suppose the hypothetical effects are $\delta_1=0.25$ and $\delta_2=0.05$. The stage means are $m_1=0.25\sqrt{100}=2.50$ and $m_2=0.05\sqrt{100}=0.50$.

Equal weights give a combined mean of $0.7071(2.50)+0.7071(0.50)=2.121$, and power $\Phi(2.121-1.960)=\Phi(0.161)\approx0.564$. Now consider the prespecified normalized weights $w_1=0.80$ and $w_2=0.60$. Since $0.80^2+0.60^2=1$, their null variance is also one. Their combined mean is

0.80(2.50)+0.60(0.50)=2.30, \qquad \operatorname{Power}\approx\Phi(2.30-1.960) =\Phi(0.340)\approx0.633.

The second weighting has about 6.9 percentage points more power for this effect pattern because it gives greater emphasis to the stage with the larger signal. It does not follow that these weights are preferable in general: if stage 2 instead carried the larger effect, the comparison could change. Weights should not be selected retrospectively to match observed stage effects, since that selection changes the test procedure and its calibration.

Power Curves and Calibration Checks

A single power calculation can hide how sensitive a design is to the assumed effect size. A power curve evaluates the same prespecified test over a range of alternatives. The table below uses the information allocation from the first example, a common effect $\delta$, and $\alpha=0.025$. The equal-weight combined mean is $13.416\delta$; the information-proportional combined mean is $\sqrt{200}\delta=14.142\delta$. Power is calculated as $\Phi(\text{combined mean}-1.960)$.

Common effect $\delta$	Equal-weight power	Information-proportional power
0.10	0.268	0.293
0.15	0.521	0.564
0.20	0.765	0.807
0.25	0.918	0.943

The curves would both increase with $\delta$, while maintaining a gap for this allocation and common-effect pattern. The gap is not a type I error rate: it describes the difference in rejection probability under alternatives. To check calibration, evaluate rejection under $H_0$ using the actual combination statistic and its null distribution, not the power curve.

Worked Example: Spotting a Weight-Normalization Error

Suppose two independent stage statistics are standard normal under $H_0$, and someone uses weights $1$ and $1$ with the usual one-sided critical value $1.960$. The statistic is $Z_1+Z_2$, whose null variance is $1^2+1^2=2$, not one. Thus the actual type I error is

P(Z_1+Z_2>1.960\mid H_0) =1-\Phi\!\left(\frac{1.960}{\sqrt{2}}\right) \approx 1-\Phi(1.386) \approx0.0829.

This is not a valid level-$0.025$ test using the standard normal critical value. Normalizing the weights to $1/\sqrt{2},1/\sqrt{2}$ restores null variance one; alternatively, retaining weights $1,1$ requires comparing the unnormalized sum with $1.960\sqrt{2}$. This example shows why a high observed rejection rate under a null simulation may indicate miscalibration, whereas a low rejection rate under a scientifically relevant alternative indicates low power. The two diagnoses require different checks.

Practical Diagnostics and Limitations

A useful sensitivity analysis holds the testing rule fixed and varies the quantities that plausibly change the stage signals: the stage information allocation, the overall effect size, and the pattern of effects across stages. Report the weights, the assumed $I_i$ and $\delta_i$, the critical value, and the resulting power. If several plausible effect patterns give very different power, that is important design sensitivity rather than evidence that the test has failed.

Check null calibration separately. Verify that the squared weights sum to one when using a standard normal critical value, and verify that the stagewise null evidence has the assumed marginal and conditional distributions. If stage statistics are dependent, their covariance must be reflected in the variance and calibration of the combination statistic. A mismatch between the assumed and actual dependence can cause a type I error problem; changing weights to address power does not correct that mismatch by itself.

The closed-form power expression assumes fixed stage information and the stated normal approximation. If a sample-size re-estimation makes stage 2 information depend on stage 1 results, simply substituting one planned value of $I_2$ may not give unconditional power. Power then depends on the adaptation rule and must account for the possible stage-2 sizes and their relationship to the interim data, typically by integration or simulation. The combination-test null calibration and the power calculation should both reflect the actual design.

Compared with equal weights, information-proportional weights emphasize stages according to their information and align naturally with a common-effect alternative in the simple model above. Weights optimized for a specified alternative can improve power for that alternative, but may be sensitive to misspecification of its stage pattern. Equal weights can be a defensible prespecified choice when stagewise information or plausible effect patterns are uncertain; they are not automatically optimal. No single weighting is best across all alternatives, and changing the weights trades power across patterns rather than creating information.

Key takeaway: With correctly normalized fixed weights, a power difference across weighting schemes is not by itself evidence of type I error failure. Assess power by the weighted stage signals under alternatives; assess calibration by the null distribution and the validity of the stagewise evidence.

Check Your Understanding

Answer each question, then compare with the short solutions below.

Two independent stage statistics have standard normal distributions under $H_0$. What condition on two fixed weights makes their weighted sum have unit variance?
For a one-sided test, explain why a weight choice can reduce power without changing type I error.
With $I_1=25$, $I_2=75$, and a common effect $\delta=0.10$, calculate the stage means and the combined mean using information-proportional weights.
Why may a fixed-$I_2$ normal power formula be inadequate when stage-2 sample size is re-estimated using interim data?

Solutions. (1) Require $w_1^2+w_2^2=1$, assuming independent unit-variance stage statistics. (2) Under the null, normalized weights preserve a standard normal combination statistic, but under an alternative the combined mean $w_1m_1+w_2m_2$ changes with the weights. (3) The stage means are $0.10\sqrt{25}=0.500$ and $0.10\sqrt{75}=0.866$. The information-proportional weights are $w_1=\sqrt{25/100}=0.500$ and $w_2=\sqrt{75/100}=0.866$, giving combined mean $0.500(0.500)+0.866(0.866)=1.000$. (4) The realized stage-2 information depends on interim data, so unconditional power must average over the possible adaptations and their relationship to those data.