Data-Dependent Weights and Validity Conditions

Why Weight Selection Matters

In an inverse-normal combination test, each stage contributes a standardized statistic, and weights determine how much each stage contributes to the final statistic. When those weights are fixed in advance, their role in the null distribution is straightforward. When the weights are chosen using interim outcomes, however, the weights and the first-stage statistic are related. Treating the resulting statistic as standard normal can then give an incorrect rejection probability.

This distinction matters in adaptive sample size re-estimation. A rule may use interim results to change stage-two information and then choose weights to match the information available at each stage. Unless the test’s rejection boundary is also justified for that adaptive rule, the familiar standard normal cutoff need not control Type I error. The earlier material on inverse-normal combination tests establishes the fixed-weight reference result; here the focus is the conditions that make that result fail and the conditional calibration that can replace it.

Definition: Let $Z_1$ and $Z_2$ be the stagewise normal scores, with larger values favoring $H_a$. An inverse-normal combination statistic is $$ T=w_1Z_1+w_2Z_2. $$ For fixed weights satisfying $w_1^2+w_2^2=1$, and independent $Z_1,Z_2\sim N(0,1)$ under $H_0$, $T\sim N(0,1)$. Data-dependent weights are weights selected as functions of interim data, such as $w_1=w_1(Z_1)$ and $w_2=w_2(Z_1)$. The fixed-weight result does not, in general, apply to this statistic.

Conditions Behind the Fixed-Weight Reference

For the standard normal reference to hold, assume a one-sided test and that, under $H_0$, the stagewise statistics are independent standard normal variables. Equivalently, stagewise one-sided p-values are independent Uniform$(0,1)$ variables and are converted to scores by $Z_i=\Phi^{-1}(1-p_i)$. Also assume that $w_1$ and $w_2$ are fixed constants, not functions of either observed stage statistic, and that $w_1^2+w_2^2=1$.

The stagewise null distributions must be appropriate for the design actually used. For example, the second-stage statistic must remain correctly standardized under the null after the sample size has been selected. If a p-value is not valid conditional on the interim information, independence and normality cannot simply be presumed. These assumptions describe the reference result, not a guarantee for every implementation of re-estimation.

The justification for fixed weights follows from the distribution of a linear combination of independent normal variables:

E_{H_0}(T)=w_1E(Z_1)+w_2E(Z_2)=0, \qquad \operatorname{Var}_{H_0}(T)=w_1^2+w_2^2=1.

Because $T$ is a linear combination of independent normal variables, it is normal. Its mean and variance therefore give $T\sim N(0,1)$. Thus a level-$\alpha$ upper-tail test rejects when $T>z_{1-\alpha}$, where $z_{1-\alpha}=\Phi^{-1}(1-\alpha)$.

Worked Example: A Fixed-Weight Combination

Suppose a hypothetical two-stage study uses independent stage statistics $Z_1=1.20$ and $Z_2=1.50$, and prespecifies weights $w_1=0.60$ and $w_2=0.80$. Under the stated null model, the stage statistics are independent standard normal variables, and the weights satisfy $0.60^2+0.80^2=1$.

The observed combination statistic is $$ T=0.60(1.20)+0.80(1.50)=0.72+1.20=1.92. $$

For a one-sided test at $\alpha=0.025$, the critical value is approximately $z_{0.975}=1.960$. Since $1.92<1.960$, the test does not reject $H_0$. The standard normal reference is justified here because the weights were fixed and the stage statistics meet the independence and null-distribution assumptions. The observed upper-tail p-value is $1-\Phi(1.92)\approx0.0274$.

What Changes When Weights Depend on Interim Outcomes

If the weights are functions of $Z_1$, then, conditional on $Z_1=z$, the combination statistic has the form $$ T\mid Z_1=z=w_1(z)z+w_2(z)Z_2. $$

Under the assumptions that $Z_2\mid Z_1=z\sim N(0,1)$ and that the weights are determined by the interim data, this conditional distribution has mean $w_1(z)z$ and variance $w_2(z)^2$. It is not generally standard normal. Unconditionally, the distribution is a mixture over possible interim outcomes, with both the conditional mean and variance varying with $z$. Normalizing the weights at each interim value does not remove that dependence.

A simple example shows how a nominal cutoff can inflate the overall Type I error. The rule below is deliberately stark so the error probability can be calculated exactly. Boundary weights are used to make the arithmetic transparent; the lesson is that outcome-dependent selection itself invalidates automatic use of the fixed-weight reference.

Worked Example: Exact Type I Error Inflation From Outcome-Dependent Weights

Under a hypothetical null model, let $Z_1$ and $Z_2$ be independent standard normal variables. Choose weights $(w_1,w_2)=(1,0)$ if $Z_1\geq0$, and $(w_1,w_2)=(0,1)$ if $Z_1<0$. In either case, the weights have squared sum one. Suppose the test nevertheless uses the usual one-sided $0.025$ cutoff $1.960$.

When $Z_1\geq0$, $T=Z_1$; when $Z_1<0$, $T=Z_2$. Therefore, $$ \begin{aligned} P_{H_0}(T>1.960) &=P(Z_1\geq0,\ Z_1>1.960) +P(Z_1<0,\ Z_2>1.960)\\ &=P(Z_1>1.960)+P(Z_1<0)P(Z_2>1.960)\\ &=0.025+(0.5)(0.025)=0.0375. \end{aligned} $$

The factorization in the second term uses the assumed independence of $Z_1$ and $Z_2$. The actual null rejection probability is $0.0375$, not $0.025$. Thus the usual standard normal cutoff fails to control Type I error for this data-dependent weighting rule.

The distortion can also be seen conditionally. For example, suppose an adaptive rule selects equal weights whenever $Z_1\geq0$. At an interim value $Z_1=1$, the statistic is $T=(1+Z_2)/\sqrt{2}$. Conditional on that interim result, rejection at $1.960$ has probability

\begin{aligned} P_{H_0}(T>1.960\mid Z_1=1) &=P\!\left(\frac{1+Z_2}{\sqrt{2}}>1.960\right)\\ &=1-\Phi(1.960\sqrt{2}-1)\\ &=1-\Phi(1.7719)\approx0.03821. \end{aligned}

This conditional probability is not the nominal $0.025$. It illustrates why a standard normal marginal reference cannot be assumed merely because the weights sum in squares to one. It does not, by itself, calculate the overall Type I error for every rule selecting equal weights; that requires accounting for all possible interim outcomes and the corresponding selected weights.

Conditional Calibration

A general way to preserve error control is to use the original design’s conditional error. For each interim result $z$, define $A(z)$ as the probability, under $H_0$, that the original design would reject using the remaining data, conditional on $Z_1=z$. The original design’s unconditional Type I error is the average of these conditional probabilities over the interim distribution.

Definition: The conditional error function of an original design is $$ A(z)=P_{H_0}(\text{original design rejects}\mid Z_1=z). $$ A redesigned test preserves conditional error if, for every relevant interim value $z$, its conditional probability of rejecting under $H_0$ is no greater than $A(z)$. If the original design has Type I error at most $\alpha$, such a redesign also has unconditional Type I error at most $\alpha$, provided the interim distribution and conditional null model used for the calculation are valid.

For a two-stage, one-sided inverse-normal test with original fixed weights $w_1^0,w_2^0$ and critical value $c_\alpha$, where $w_2^0>0$, its conditional error is $$ A(z)=1-\Phi\!\left(\frac{c_\alpha-w_1^0z}{w_2^0}\right). $$ Suppose the redesigned test selects weights $w_1(z),w_2(z)$ from the interim information, with $w_2(z)>0$, and the new stage-two statistic remains conditionally standard normal under $H_0$. Its conditional rejection boundary can be chosen as $$ c(z)=w_1(z)z+w_2(z)z_{1-A(z)}. $$

Indeed, conditional on $Z_1=z$, the redesigned statistic exceeds this boundary exactly when $Z_2>z_{1-A(z)}$, so its conditional rejection probability is $A(z)$. If $A(z)=0$ or $A(z)=1$, interpret the corresponding quantile as an infinite boundary in the appropriate direction. This construction preserves the original design’s conditional error, rather than applying the unadjusted standard normal cutoff to a statistic with random weights.

Worked Example: Calibrating a Changed Weight Pair

Consider a hypothetical original test with fixed weights $w_1^0=0.60$, $w_2^0=0.80$, and one-sided critical value $c_\alpha=1.960$. At the interim value $Z_1=1$, the original conditional error is $$ A(1)=1-\Phi\!\left(\frac{1.960-0.60(1)}{0.80}\right) =1-\Phi(1.700)\approx0.04457. $$

Suppose the redesigned test selects weights $w_1(1)=0.80$ and $w_2(1)=0.60$. These weights also have squared sum one, but the usual $1.960$ boundary is not justified by that fact. To preserve the original conditional error, use $$ c(1)=0.80(1)+0.60z_{1-0.04457} \approx0.80+0.60(1.700)=1.820. $$

Given $Z_1=1$, the redesigned statistic is $0.80+0.60Z_2$. Its conditional rejection probability at the calibrated boundary is $$ P_{H_0}(0.80+0.60Z_2>1.820\mid Z_1=1) =1-\Phi\!\left(\frac{1.820-0.80}{0.60}\right) \approx1-\Phi(1.700)=0.04457. $$

The conditional rejection probability matches the original design’s conditional error at this interim value. The calibrated boundary is not the standard normal critical value; it is chosen to preserve the error probability left by the original design at the observed interim result.

Why the Calibration Justification Works

The key argument is conditional probability followed by averaging. Let $R$ denote rejection by the redesigned test. If the calibration ensures $P_{H_0}(R\mid Z_1=z)\leq A(z)$ for every relevant $z$, then

P_{H_0}(R) =E_{H_0}\!\left[P_{H_0}(R\mid Z_1)\right] \leq E_{H_0}[A(Z_1)] =P_{H_0}(\text{original design rejects}) \leq\alpha.

The equality at the end of the first line is the law of total probability: average the conditional rejection probability over interim outcomes. The next inequality is the calibration requirement, applied at each interim outcome. For an original test whose Type I error is exactly $\alpha$, the final bound is $\alpha$; if its Type I error is below $\alpha$, the redesigned test inherits that bound. The result depends on using the correct conditional null distribution and on the redesign preserving the specified conditional error for the interim data actually observed.

Assumptions, Diagnostics, and Alternatives

Conditional calibration is only as reliable as its ingredients. The interim data must be represented by the conditioning information used to calculate $A(z)$, and the stage-two p-value or statistic must have the stated null distribution conditional on that information. In particular, if stage-two data are not independent of the interim data under the null, a standard normal $Z_2$ cannot be used without an appropriate conditional model. The weights and boundary must be computed by the prespecified rule; changing either after seeing further outcome information is another adaptation that needs justification.

Useful checks include verifying that the selected weights are finite and satisfy the intended normalization; confirming that the stage-two statistic remains correctly calibrated after sample-size selection; and evaluating the conditional rejection probability at representative interim values, including extreme values and points where the rule changes. Under the assumed model, that probability should equal or remain below $A(z)$. Simulation can help identify implementation errors or model departures, but simulated agreement alone does not establish the mathematical assumptions or guarantee error control in a different null setting.

Approach	What justifies the rejection rule?	Main caution
Fixed inverse-normal weights	Independent standard normal stage statistics and prespecified weights with squared sum one.	Changing the weights based on interim outcomes removes the direct fixed-weight justification.
Data-dependent weights with the usual cutoff	No general justification from weight normalization alone.	The null distribution can be a mixture with interim-dependent conditional mean and variance.
Conditional calibration	The redesigned test’s conditional null rejection probability is bounded by the original conditional error.	Requires a valid conditional null model and correct implementation of the boundary.

A common mistake is to argue, “The weights still satisfy $w_1^2+w_2^2=1$, so the combination statistic is standard normal.” That calculation establishes a unit variance only for fixed weights multiplying independent standard normal variables. When a weight depends on $Z_1$, it is no longer a fixed coefficient independent of the data; the linear-combination argument does not apply. The correction is to use a reference distribution justified for the adaptive rule or to calibrate the redesigned test conditionally, as above.

Conditional calibration is not a universal repair for every design change. It requires a meaningful conditional error function and a valid model for the remaining data. If the stage-two test changes in a way that invalidates its conditional p-value, if relevant interim information is omitted from the conditioning set, or if the null distribution is misspecified, the conditional probability calculation may be wrong. In those situations, the stated error-control conclusion does not follow.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

Under what assumptions does a fixed-weight inverse-normal statistic have a standard normal null distribution?
Suppose weights are selected from $Z_1$, but at every interim result their squared sum is one. Is that sufficient to use the usual standard normal cutoff? Explain.
An original design has conditional error $A(z)=0.03$ at a particular interim result. What conditional rejection probability must a redesigned test satisfy to preserve conditional error at that result?
Why does bounding the redesigned test’s conditional rejection probability by $A(z)$ also bound its unconditional Type I error?

Solutions. (1) The stage statistics must be independent standard normal variables under the null, and the weights must be fixed constants with squared sum one. (2) No. Data-dependent weights make the conditional mean and variance depend on the interim result, so normalization alone does not establish a standard normal reference. (3) It must be no greater than $0.03$. (4) The unconditional rejection probability is the expectation of the conditional rejection probability; averaging a quantity bounded above by $A(Z_1)$ gives a result no greater than $E_{H_0}[A(Z_1)]$, the original design’s Type I error.