Changing The Interim Test Statistic

Why a Changed Interim Statistic Requires New Conditioning

An interim decision may be based on a statistic different from the one used in the original design. For example, the original design might use a standardized treatment-effect statistic, while the adapted design uses a statistic formed from a different endpoint or a different combination of interim observations. The numerical value of the new statistic is not automatically on the scale, or in the information set, of the original one.

The relevant question is: given the new interim statistic, how much of the original design's conditional error remains available? Answering it requires the conditional distribution of the original interim information given the new statistic. Substituting the new statistic into the old conditional-error formula is generally unjustified.

Definition: Let $X$ denote the interim information used by the original design, and let $R$ denote its final rejection event. The original conditional error is $A(X)=P_0(R\mid X)$, where $P_0$ denotes probability under the null hypothesis. If an adaptation instead conditions on a new interim statistic $W$, the available original conditional error at $W=w$ is $$ \bar A(w)=E_0\{A(X)\mid W=w\}. $$ This is an average of the original conditional error over the conditional distribution of $X$ given $W=w$, not in general $A(w)$.

This definition applies when the adaptation is made using $W$ and the original interim information $X$ is not otherwise being used to distinguish continuation rules. If the adaptation uses additional interim data, those data belong in the conditioning information as well. As in the earlier material on conditional error across multiple looks, the comparison is made under the null and relative to the information available when the adaptation is chosen.

Gaussian Setup and Conditional Distribution

Consider a one-sided design that rejects at the final analysis when a standardized statistic $Y$ is at least $c$. Under $H_0$, suppose the original interim statistic $X$, the new interim statistic $W$, and $Y$ are jointly normal, each with mean zero and variance one. Write $\operatorname{Corr}(X,Y)=\rho$ and $\operatorname{Corr}(X,W)=r$. For the simple case in which $X$ captures all the interim information relevant to the original final statistic, assume $Y$ and $W$ are conditionally independent given $X$. In this Gaussian model that assumption implies $\operatorname{Corr}(Y,W)=\rho r$.

The original conditional error is obtained from the conditional distribution $Y\mid X=x$. Since its conditional mean is $\rho x$ and its conditional variance is $1-\rho^2$,

A(x)=P_0(Y\ge c\mid X=x) =1-\Phi\!\left(\frac{c-\rho x}{\sqrt{1-\rho^2}}\right).

To evaluate this error when the new statistic $W=w$ is observed, use the conditional distribution $X\mid W=w\sim N(rw,1-r^2)$. Averaging $A(x)$ over this distribution gives $\bar A(w)$. Under the conditional-independence assumption, the same answer follows directly from the conditional distribution of $Y\mid W=w$.

\bar A(w) =E_0\{A(X)\mid W=w\} =P_0(Y\ge c\mid W=w) =1-\Phi\!\left(\frac{c-\rho r w}{\sqrt{1-\rho^2r^2}}\right).

The direct expression is convenient, but its assumptions matter. It is not enough to know the original correlation $\rho$ and the correlation $r$ between the two interim statistics unless the stated conditional-independence structure is appropriate. In a more general jointly normal model, set $q=\operatorname{Corr}(Y,W)$. Then $Y\mid W=w\sim N(qw,1-q^2)$, and $P_0(Y\ge c\mid W=w)=1-\Phi((c-qw)/\sqrt{1-q^2})$. The average of the old error function, however, must still be calculated from the distribution of $X\mid W=w$. The two expressions agree only when the joint model makes them agree.

Worked Comparisons

Worked Example: Re-expressing the Available Error on a New Statistic

Suppose the original design rejects when $Y\ge1.96$. Under $H_0$, let $\operatorname{Corr}(X,Y)=0.60$, $\operatorname{Corr}(X,W)=2/3$, and assume $Y$ and $W$ are conditionally independent given $X$. Find the original conditional error available when the new interim statistic is $W=1.00$.

The original error function is $A(x)=1-\Phi((1.96-0.60x)/0.80)$, because $\sqrt{1-0.60^2}=0.80$. The conditional-independence assumption gives $\operatorname{Corr}(Y,W)=0.60(2/3)=0.40$, so $\operatorname{Var}(Y\mid W)=1-0.40^2=0.84$. Therefore,

\bar A(1.00) =1-\Phi\!\left(\frac{1.96-0.40(1.00)}{\sqrt{0.84}}\right) =1-\Phi(1.702) \approx 0.04437.

Thus, at $W=1.00$, the original design's available conditional error is about 0.04437. Direct substitution of 1.00 into $A(x)$ would instead give $1-\Phi((1.96-0.60)/0.80)=1-\Phi(1.700)\approx0.04457$. These values happen to be close here, but they represent different calculations. At $W=2.00$, the correct conditional error is $1-\Phi((1.96-0.80)/\sqrt{0.84})\approx0.1029$, whereas $A(2.00)=1-\Phi(0.950)\approx0.1711$. The discrepancy can be substantial.

Worked Example: Integrating Over the Original Statistic

Keep the final boundary $c=1.96$, but suppose $\operatorname{Corr}(X,Y)=0.50$, $\operatorname{Corr}(X,W)=0.60$, and $Y$ and $W$ are conditionally independent given $X$. Find the available error at $W=1.50$, and compare it with the incorrect plug-in calculation.

The conditional distribution needed for the averaging calculation is $X\mid W=1.50\sim N(0.60(1.50),1-0.60^2)=N(0.90,0.64)$. The original error function is $A(x)=1-\Phi((1.96-0.50x)/\sqrt{0.75})$. Consequently, the required quantity is the expectation of this function over the stated normal distribution of $X$, not its value at $x=1.50$. Equivalently, the correlation of $Y$ with $W$ is $0.50(0.60)=0.30$, giving

\bar A(1.50) =1-\Phi\!\left(\frac{1.96-0.30(1.50)}{\sqrt{1-0.30^2}}\right) =1-\Phi\!\left(\frac{1.51}{\sqrt{0.91}}\right) \approx 1-\Phi(1.583) \approx 0.0567.

The plug-in value is $A(1.50)=1-\Phi(1.21/\sqrt{0.75})\approx1-\Phi(1.397)\approx0.0812$. It is not the available error conditional on $W=1.50$: the new statistic has a different conditional relationship to the original interim statistic than equality of their numerical values would imply.

Worked Example: Checking Whether a Direct Formula Is Justified

Suppose $X,Y,W$ are standardized jointly normal under $H_0$, with $\operatorname{Corr}(X,Y)=0.60$, $\operatorname{Corr}(X,W)=0.50$, and $\operatorname{Corr}(Y,W)=0.50$. Let the original rejection boundary be $1.96$, and suppose $W=1.00$. Compare the actual conditional rejection probability with the average of the original error function over $X\mid W$.

The conditional covariance of $Y$ and $W$ given $X$ is $0.50-(0.60)(0.50)=0.20$, not zero. Thus the conditional-independence assumption does not hold. The actual conditional rejection probability, using $q=\operatorname{Corr}(Y,W)=0.50$, is

P_0(Y\ge1.96\mid W=1.00) =1-\Phi\!\left(\frac{1.96-0.50}{\sqrt{1-0.50^2}}\right) =1-\Phi(1.6859) \approx0.0459.

By contrast, averaging the original error $A(X)$ over $X\mid W=1.00$ uses the original model's implied correlation $0.60(0.50)=0.30$:

E_0\{A(X)\mid W=1.00\} =1-\Phi\!\left(\frac{1.96-0.30}{\sqrt{1-0.30^2}}\right) =1-\Phi(1.7402) \approx0.0409.

The two answers differ because $W$ carries information about $Y$ beyond what is represented by $X$. A valid comparison must specify which interim information the original error function conditions on and use a joint model that reflects the actual data and design.

Comparison With a Proposed Continuation Rule

Suppose the modified design continues after observing $W=w$ and uses a continuation rejection event $R_{\mathrm{new}}$. Its conditional rejection probability under the null is $B(w)=P_0(R_{\mathrm{new}}\mid W=w)$. To preserve the original conditional-error guarantee at that value of $w$, compare $B(w)$ with $\bar A(w)$, the available original error conditional on the same information. A rule satisfying $B(w)\le\bar A(w)$ at every relevant $w$ does not spend more conditional error than was available there. Equality uses the full available amount. This pointwise comparison is distinct from merely checking that a single overall rejection probability is below a nominal level.

If the modified design retains the same final statistic $Y$ and boundary $c$, its conditional rejection probability is the quantity $P_0(Y\ge c\mid W=w)$. If it changes the future analysis as well, calculate $B(w)$ from that continuation design's own conditional distribution. Do not assume that the original Gaussian formula applies unchanged just because the interim statistic has been renamed.

For vector-valued interim summaries, the same principle applies: use the conditional distribution of the original interim information given the new summary. For jointly normal vectors, this distribution is obtained from the conditional mean and covariance of the multivariate normal model. The scalar expressions above are a transparent special case, not a replacement for checking the full information structure.

Assumptions, Diagnostics, and Limitations

Before using the Gaussian expressions, verify the model and the design assumptions that support them:

Under $H_0$, the relevant standardized statistics have the stated joint normal distribution, or the normal approximation is adequate for the planned analysis.
The variances and correlations describe the actual interim and final statistics, including their shared observations and any changes in information.
The conditioning set contains all interim information used to choose the continuation rule. If the original error depends on additional observed quantities, those cannot be silently omitted.
For the product formula $\operatorname{Corr}(Y,W)=\rho r$, $Y$ and $W$ are conditionally independent given $X$. If this does not hold, use the appropriate joint conditional model.

Useful diagnostics include checking that every variance is positive, that the proposed correlation matrix is positive semidefinite, and that conditional variances such as $1-r^2$ and $1-q^2$ are nonnegative. Also check whether the derived conditional mean moves in the direction implied by the correlations. A negative correlation, for example, should not yield a positive conditional mean for a positive observed $W$ in the corresponding standardized bivariate-normal model.

The Gaussian model is not universal. Discrete statistics, nuisance parameters, data-dependent selection, and changes to the correlation structure can require more detailed conditional distributions. The earlier material on composite nulls and nuisance parameters is relevant when a single null parameter value does not define the error calculation: the appropriate conditional-error requirement must then account for the relevant null parameter values. A normal formula evaluated at one convenient parameter value cannot establish a guarantee for all of them.

Common Mistake and Practical Comparison

Common pitfall: “The new interim statistic is $w$, so the available error is $A(w)$.” This treats a value of $W$ as though it were the same observation as $X$. The correction is to calculate the conditional distribution of $X\mid W=w$ and average $A(X)$ over it, or to use $P_0(Y\ge c\mid W=w)$ when the joint model justifies that equivalent expression.

The main alternatives are therefore not interchangeable shortcuts. Plugging $w$ into $A$ is justified only when the new statistic genuinely represents the same conditioning information on the same scale. Integrating over $X\mid W=w$ is the direct definition of the available error. Using the conditional law of $Y\mid W=w$ can be simpler, but requires a correct joint model linking $Y$, $X$, and $W$. In every case, a valid comparison uses the same conditioning information for the available error and the proposed continuation probability.

Key takeaway: When the interim statistic changes, the original conditional-error function must be transported to the new statistic through a conditional distribution. Derive that distribution from the joint model; do not identify $W$ with $X$ merely because both are interim statistics.

Check Your Understanding

Answer each question using the conditional-error definitions and assumptions above. Short solutions follow the questions.

In one sentence, define the available original conditional error at $W=w$.
Under the Gaussian conditional-independence model, let $\rho=0.40$, $r=0.50$, $w=1.00$, and $c=1.96$. What conditional correlation and conditional variance of $Y\mid W$ are used?
Why is $A(w)$ generally not the same as $\bar A(w)$?
If $\operatorname{Corr}(X,Y)=0.60$, $\operatorname{Corr}(X,W)=0.50$, and $\operatorname{Corr}(Y,W)=0.50$, does the conditional-independence product formula apply? Show the check.
A modified continuation rule has conditional rejection probability $B(w)=0.06$, while the available original conditional error is $\bar A(w)=0.05$. What does this comparison imply at that $w$?

Solutions. (1) It is $E_0\{A(X)\mid W=w\}$, the original conditional error averaged over the null conditional distribution of the original interim information given the new statistic. (2) The correlation used for the conditional distribution of $Y\mid W$ is $\operatorname{Corr}(Y,W)=\rho r=(0.40)(0.50)=0.20$, and $\operatorname{Var}(Y\mid W)=1-0.20^2=0.96$. (3) $A(w)$ evaluates the old function at the number $w$; $\bar A(w)$ averages that function over $X\mid W=w$, which need not be concentrated at $w$. (4) No. The conditional covariance is $0.50-(0.60)(0.50)=0.20$, not zero, so $Y$ and $W$ are not conditionally independent given $X$. (5) At that value of $w$, the proposed rule exceeds the available conditional error by $0.01$, so it does not satisfy the pointwise conditional-error constraint there.