Selecting Weights for Inverse Normal Tests

Why Weight Choice Is a Design Decision

An inverse normal test combines stagewise evidence by transforming each stage’s p-value to a normal score and taking a weighted sum. The weights determine how strongly each stage’s score enters that sum. They are therefore part of the test design: a large weight gives a stage more influence on the combined statistic, while a small weight reduces it.

Choosing weights is not simply a matter of assigning the largest coefficient to the stage expected to have the strongest result. The weights must be normalized for the usual null reference distribution, chosen with the intended stage roles in mind, and prespecified if the test is to use the reference distribution associated with fixed weights. The earlier material on weighted inverse normal combination statistics establishes the form of the statistic; here the focus is how to select its weights and what each selection means.

Definition: For $K$ stages, let $p_i$ be the stagewise p-value at stage $i$, and define its upper-tail normal score $U_i=\Phi^{-1}(1-p_i)$, where $\Phi$ is the standard normal cumulative distribution function. The weighted inverse normal statistic is $$ Z_w=\sum_{i=1}^{K} w_i U_i, \qquad \sum_{i=1}^{K}w_i^2=1. $$ The weights $w_i$ are fixed design constants. In the usual design, $w_i\geq 0$, so evidence in the prespecified direction is not deliberately assigned a negative contribution.

The familiar standard normal null distribution requires assumptions. For the exact reference result, suppose that under $H_0$, the stagewise p-values are independent and each is continuously Uniform$(0,1)$. Then each $U_i$ is independently standard normal, and the normalized weighted sum is standard normal. Independence commonly follows from independent stage data and valid stagewise tests, but it should not be assumed merely because the data are labelled as separate stages. With discrete or otherwise non-uniform p-values, or dependent stages, calibration may need to account for their actual joint null distribution.

What the Weights Contribute

A weight is a coefficient, not a percentage of evidence guaranteed to come from a stage. The realized contribution is $w_iU_i$, which depends on both the weight and the observed score. A stage with a large weight can contribute little if its score is near zero; a negative score can reduce the total even when its weight is positive.

Under the independent standard normal null model, the variance contribution from stage $i$ is $w_i^2$, because $$ \operatorname{Var}(w_iU_i)=w_i^2\operatorname{Var}(U_i)=w_i^2. $$ The normalization makes the total variance equal to one. Thus the squared weights describe shares of null variance, not fixed shares of the observed statistic. This distinction is useful when explaining a design: $w_i$ controls the stage’s coefficient, and $w_i^2$ controls its share of null variance.

Worked Example: Equal Weights for Two Stages

Consider a hypothetical two-stage design with independent, continuous null p-values. The design team wants the two stages to have equal coefficients. Equal coefficients combined with the unit-length constraint require $w_1=w_2=w$, where $2w^2=1$. Since the weights are nonnegative, $$ w=\sqrt{\frac{1}{2}}=0.7071. $$ Suppose the observed stagewise p-values are $p_1=0.20$ and $p_2=0.05$. Their upper-tail normal scores are approximately $U_1=0.8416$ and $U_2=1.6449$. Therefore, $$ Z_w=(0.7071)(0.8416)+(0.7071)(1.6449) =0.5951+1.1631=1.758. $$ The coefficients are equal, but the second stage contributes more to this realized sum because its observed normal score is larger. Under the stated null assumptions, the statistic has a standard normal null distribution. This calculation illustrates the statistic and the contributions; a test decision would additionally require the applicable calibrated rejection boundary.

Information-Based Weights

An information-based choice uses the planned statistical information in each stage to determine the relative weights. For stages with positive planned information $I_1,\ldots,I_K$, a common choice is $$ w_i=\sqrt{\frac{I_i}{\sum_{j=1}^{K} I_j}}. $$ The square root is important: the information fraction is the squared weight, not the weight itself. This is a design convention motivated by matching the stages’ null-variance contributions to their planned information fractions. The information quantities must be defined on a comparable scale and specified before the unblinded stage results are available.

Worked Example: Weights From Planned Information

Suppose a hypothetical trial plans information $I_1=100$ for stage 1 and $I_2=300$ for stage 2. The total planned information is $100+300=400$. Applying the information-based rule gives $$ w_1=\sqrt{\frac{100}{400}}=0.5000, \qquad w_2=\sqrt{\frac{300}{400}}=0.8660. $$ The squared weights are $0.2500$ and $0.7500$, respectively, and sum to one. If the stagewise normal scores are hypothetically $U_1=0.20$ and $U_2=1.20$, then $$ Z_w=(0.5000)(0.20)+(0.8660)(1.20) =0.1000+1.0392=1.139. $$ The larger stage-2 coefficient reflects its larger planned information under this rule. It does not mean that stage 2 will necessarily produce more evidence in the realized trial, nor does it establish that these weights maximize power for every treatment effect pattern.

Information-based weighting is attractive when stage information can be planned meaningfully and the scientific design intends evidence contributions to track that plan. It is less compelling when the stages use substantially different estimands, measurement scales, or analysis procedures, or when the information figures are unstable or difficult to compare. In those cases, a numerical information ratio can suggest more precision than the design assumptions justify.

Power-Oriented Weights

A power-oriented design chooses weights to maximize rejection probability under a specified alternative, subject to the normalization and the test’s calibrated rejection rule. In general, power depends on the joint distribution of all the stagewise normal scores under that alternative—not just their means. For a fixed upper-tail boundary $c$, the relevant objective is $$ P\left(\sum_{i=1}^{K}w_iU_i>c\mid H_a\right). $$ The alternative model, the boundary, and any constraints on the weights must be stated before optimizing this quantity. If the alternative is not known precisely, it is often more informative to compare power over several plausible scenarios than to optimize for one assumed effect pattern.

A useful special case illustrates the assumptions behind a simple power-oriented rule. Suppose the stagewise scores are independent and, under a specified alternative, $U_i\sim N(\delta_i,1)$. Then $Z_w\sim N(\sum_i w_i\delta_i,1)$. For a fixed upper-tail boundary $c$, the power is $$ P(Z_w>c\mid H_a) =1-\Phi\left(c-\sum_{i=1}^{K}w_i\delta_i\right). $$ In this particular normal location model, maximizing $\sum_iw_i\delta_i$ maximizes power, because the right-hand side increases as that mean increases. A larger mean alone does not guarantee greater power without a distributional assumption such as this one.

Worked Example: A Power-Oriented Choice Under a Normal Model

Consider a hypothetical two-stage design with independent stage scores. For illustration, assume that under a particular alternative $U_1\sim N(0.40,1)$ and $U_2\sim N(0.80,1)$. These are explicit model assumptions for this comparison; transformed p-values are not generally normally distributed under an alternative. The vector of alternative means is $(0.40,0.80)$.

Among nonnegative weights with $w_1^2+w_2^2=1$, the mean of $Z_w$ is largest when the weight vector points in the direction of $(0.40,0.80)$. The resulting weights are $$ w_1=\frac{0.40}{\sqrt{0.40^2+0.80^2}}=0.4472, \qquad w_2=\frac{0.80}{\sqrt{0.40^2+0.80^2}}=0.8944. $$ The combined mean is $$ (0.4472)(0.40)+(0.8944)(0.80)=0.8944. $$ For comparison, equal weights give a combined mean of $$ \frac{0.40+0.80}{\sqrt{2}}=0.8485. $$ Under the assumed normal model, both combined statistics have variance one. To compare their modeled upper-tail probabilities at the same illustrative boundary $c=1.96$, the power-oriented choice gives $$ 1-\Phi(1.96-0.8944)=1-\Phi(1.0656)\approx 0.1433, $$ whereas equal weights give $$ 1-\Phi(1.96-0.8485)=1-\Phi(1.1115)\approx 0.1332. $$ The difference follows under the stated normal location model, not from a general rule that a larger mean always means more power. If the score distributions under the alternative are not normal with the assumed variances, or if their dependence differs, the exact tail probabilities may not follow this calculation.

Why the Mean-Maximization Result Holds in the Special Case

The preceding power-oriented choice can be justified directly. Write $\boldsymbol{w}=(w_1,\ldots,w_K)$ and $\boldsymbol{\delta}=(\delta_1,\ldots,\delta_K)$. Under the stated model, the combined alternative mean is the inner product $\boldsymbol{w}\cdot\boldsymbol{\delta}$. By the Cauchy–Schwarz inequality, $$ \boldsymbol{w}\cdot\boldsymbol{\delta} \leq \|\boldsymbol{w}\|\,\|\boldsymbol{\delta}\| =\|\boldsymbol{\delta}\|, $$ because the normalization imposes $\|\boldsymbol{w}\|=1$. If $\boldsymbol{\delta}=\boldsymbol{0}$, every normalized weight vector attains equality; otherwise, equality holds when $\boldsymbol{w}$ is proportional to $\boldsymbol{\delta}$, which respects the nonnegative-weight constraint if the components of $\boldsymbol{\delta}$ are nonnegative. Since the combined statistic is normal with variance one in this model, its upper-tail probability at fixed $c$ increases with its mean. Together, these facts justify choosing weights proportional to the assumed alternative means in this special case.

The justification has two essential parts: the geometric result about the mean and the normal location model connecting that mean to power. If the alternative distribution is not a normal location shift with the stated variance, the first part still concerns the mean, but it does not by itself establish which weights maximize the rejection probability. In a less restrictive model, power-oriented selection should evaluate the alternative tail probability itself, analytically when possible or by a prespecified simulation study.

Comparing the Main Choices

Choice	What it prioritizes	Strength	Limitation
Equal weights	Equal coefficients for the stages	Simple and does not require stage information or an effect-size model	May not reflect unequal planned information or the alternative pattern of interest
Information-based weights	Planned information fractions through squared weights	Connects stage influence to a stated design quantity	Depends on comparable, credible planned information and is not automatically power-optimal
Power-oriented weights	Rejection probability under a specified alternative model	Targets a declared scientific scenario or set of scenarios	Can be sensitive to misspecified effects, distributions, or dependence assumptions

These are design alternatives, not interchangeable labels for the same rule. Equal weights may be a reasonable default when no stage is meant to dominate and no credible information or alternative model supports a different allocation. Information-based weights use a planned property of the design. Power-oriented weights use an alternative distribution and an objective. A design team should state which rationale applies rather than selecting numbers first and providing a rationale afterward.

Prespecification, Diagnostics, and Limitations

To preserve the intended interpretation, specify the weights, their normalization, and their rationale before examining unblinded stage results. Planned information may be updated under a prespecified design rule, but that is different from choosing weights after seeing which stage has the more favorable score. If weights are selected using observed evidence, the usual fixed-weight null reference distribution cannot simply be assumed to apply; the data-dependent selection must be included in the calibration.

Useful design checks include confirming that the weights are nonnegative if evidence is intended to contribute in one direction, that $\sum_iw_i^2=1$, and that each planned information value or alternative-model parameter is defined on a compatible scale. Review sensitivity: recompute the design comparison under plausible information ratios and alternative effect patterns. A power-oriented choice that performs well only under one narrow assumed pattern may be a fragile basis for design.

A common mistake is to say, “Stage 2 is expected to have the larger mean score, so giving it the larger weight must increase power.” That reasoning is incomplete. It establishes neither the distribution of the combined statistic under the alternative nor its tail probability. The correction is to state a distributional model and derive power from it, or to evaluate the rejection probability under a more general joint alternative model. The worked normal-model comparison is valid precisely because it explicitly assumes independent normal stage scores with variance one.

Key takeaway: Weights encode a prespecified allocation of coefficients, and their squares encode null-variance shares under independent standard normal stage scores. Select them by a clear design rationale—equal contribution, planned information, or modeled power—and do not infer power from a larger mean without the assumptions that connect the mean to the tail probability.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

For three stages, equal nonnegative weights satisfy $\sum_iw_i^2=1$. What is each weight?
Two stages have planned information $I_1=1$ and $I_2=3$. Under the information-based rule in this tutorial, what are the weights and squared weights?
Under the independent model $U_i\sim N(\delta_i,1)$, explain why weights proportional to nonnegative $\delta_i$ maximize upper-tail power at a fixed boundary.
Why is it not generally valid to conclude that a larger mean of a combined statistic means greater power?
What is the design concern if weights are chosen after reviewing the stagewise scores?

Solutions. (1) Equal weights have $3w^2=1$, so each is $1/\sqrt{3}=0.5774$. (2) The total information is 4, so $w_1=\sqrt{1/4}=0.5000$ and $w_2=\sqrt{3/4}=0.8660$; their squared weights are 0.25 and 0.75. (3) The combined mean is $\boldsymbol{w}\cdot\boldsymbol{\delta}$, maximized by Cauchy–Schwarz when the unit vector $\boldsymbol{w}$ is proportional to $\boldsymbol{\delta}$. Under the specified normal model the combined variance is one, so a larger mean increases the upper-tail probability at a fixed boundary. (4) Means do not determine tail probabilities for arbitrary distributions; distributions with different shapes can have different tail probabilities even when one has the larger mean. (5) The selected weights depend on the observed evidence, so the fixed-weight null distribution may not calibrate the resulting procedure; the selection rule must be prespecified and included in valid calibration.