Worked Example: Fisher Combination With Variable Stage Two

Why Variable Stage-Two Size Needs Careful Calibration

A trial may use its first-stage results to decide how many observations to collect in stage two. For example, a prespecified rule may request more observations when the first stage is promising but inconclusive. The second-stage sample size is then random from the perspective of the trial’s start. This does not automatically invalidate a combination test, but it does make the validity of the stagewise p-values essential.

This tutorial considers a two-stage test in which both stages produce p-values in the same prespecified direction, and the second-stage sample size may be chosen using first-stage information. It focuses on Fisher’s combination statistic and on why its usual chi-square calibration can remain valid under that adaptation. The trial is assumed to complete both stages; early stopping requires its own treatment.

Definition: For two stagewise p-values $p_1$ and $p_2$, Fisher’s combination statistic is $$ X_F=-2\{\ln(p_1)+\ln(p_2)\}=-2\ln(p_1p_2). $$ When $p_1$ and $p_2$ are independent Uniform$(0,1)$ random variables under $H_0$, $X_F$ has a chi-square distribution with 4 degrees of freedom. The Fisher-combination p-value is the upper-tail probability $P(\chi^2_4\ge X_F)$.

Assumptions for Adaptation With a Variable Sample Size

The chi-square reference distribution is exact under the null when both stagewise p-values are exact and independent Uniform$(0,1)$ variables. With a sample-size adaptation, this condition is best checked conditionally. Let $\mathcal{F}_1$ denote all information available after stage one, including the stage-one data and the chosen second-stage sample size. The key requirement for stage two is $P(p_2\le u\mid \mathcal{F}_1)=u$ for every $0\le u\le1$, under $H_0$.

This conditional uniformity means that, whatever first-stage result and sample-size choice occurred, the stage-two p-value is still correctly calibrated under the null. It follows when stage-two observations are independent of stage one under $H_0$, the chosen test is valid at the selected sample size, and the sample-size rule does not use stage-two outcomes before defining the test. The first-stage p-value must itself be exact Uniform$(0,1)$ under the null. A fixed direction and a consistent definition of each stagewise test are also needed.

A rule may use the first-stage statistic, p-value, or other prespecified first-stage information to select the second-stage size. The rule need not select a single fixed size. But selection based on unblinded stage-two outcomes, or an unaccounted change in the endpoint or direction of testing, can break the conditional validity argument. In finite samples, discrete tests often give conservative rather than exactly uniform p-values; in that case the exact chi-square result does not automatically apply.

Why the Chi-Square Calibration Still Holds

The result rests on the conditional distribution of the second-stage p-value, not on the second-stage sample size being fixed in advance. For an exact Uniform$(0,1)$ variable $P$, the transformation $-2\ln(P)$ has a chi-square distribution with 2 degrees of freedom. The argument can therefore be applied first to stage one and then conditionally to stage two.

Transform the first-stage p-value.
Under $H_0$, $p_1\sim\mathrm{Uniform}(0,1)$, so $-2\ln(p_1)\sim\chi^2_2$.

Condition on the adaptation.
Given $\mathcal{F}_1$, conditional uniformity gives $p_2\mid\mathcal{F}_1\sim\mathrm{Uniform}(0,1)$. Thus $-2\ln(p_2)\mid\mathcal{F}_1\sim\chi^2_2$, with a distribution that does not depend on $\mathcal{F}_1$.

Combine the transformed values.
Because the conditional distribution in Step 2 does not depend on first-stage information, the stage-two transformed value is independent of the stage-one transformed value. Their sum is $\chi^2_2+\chi^2_2\sim\chi^2_4$, which is Fisher’s statistic.

The last step uses the standard additivity of independent chi-square variables: the degrees of freedom add. It also explains the role of conditional validity. Merely observing that stage-two data were collected independently is not enough if the test used to turn those data into $p_2$ is not valid for the selected design.

Worked Example: Calculating Fisher’s Combined Result

Suppose a hypothetical two-stage trial has exact one-sided stagewise p-values $p_1=0.200$ and $p_2=0.100$. Assume the conditional validity and independence conditions hold, even if the second-stage size was selected from first-stage information. Calculate the combined statistic and p-value.

The product is $p_1p_2=0.200(0.100)=0.0200$. Therefore,

X_F=-2\ln(0.0200)=7.824.

For 4 degrees of freedom, the chi-square upper-tail probability has the form $P(\chi^2_4\ge x)=e^{-x/2}(1+x/2)$. Substituting $x=7.824$,

P(\chi^2_4\ge7.824) =e^{-3.912}(1+3.912) \approx0.09824.

Thus the combined p-value is approximately 0.0982. At significance level $\alpha=0.05$, this result does not reject $H_0$. The combined p-value measures how extreme the observed combined evidence is under the calibrated null procedure; it is not the probability that $H_0$ is true.

Worked Adaptation With a One-Sample Normal Test

A simple model makes the conditional validity requirement concrete. Suppose each stage observes independent measurements with $X\sim N(\mu,\sigma^2)$, with known $\sigma$, and tests the one-sided hypotheses $H_0:\mu=0$ versus $H_a:\mu>0$. For a stage with sample size $n$, the test statistic is $Z=\bar{x}/(\sigma/\sqrt n)$, and the p-value is $1-\Phi(Z)$. Under $H_0$, $Z\sim N(0,1)$, so this p-value is Uniform$(0,1)$. If the second-stage observations are independent of stage one, this remains true conditional on the first-stage information and the selected $n_2$.

Worked Example: An Interim Rule Selects 40 Second-Stage Observations

In a hypothetical study, the outcome is a measurement in units with known standard deviation $\sigma=10$. Stage one has $n_1=25$ observations and a sample mean of $3.36$ units. A prespecified rule selects $n_2=20$ if $Z_1<1$, $n_2=40$ if $1\le Z_1<2$, and $n_2=60$ if $Z_1\ge2$. Stage-two observations are new and independent, and the same one-sided test is used. The stage-two sample mean is $2.026$ units. Calculate the combined result.

For stage one, the standard error is $10/\sqrt{25}=2.000$ units, so $Z_1=3.36/2.000=1.680$. This selects $n_2=40$, as the interim statistic lies between 1 and 2. The stage-one p-value is $p_1=1-\Phi(1.680)\approx0.04648$.

For stage two, the standard error is $10/\sqrt{40}\approx1.5811$ units. Thus

Z_2=\frac{2.026}{10/\sqrt{40}}\approx1.281, \qquad p_2=1-\Phi(1.281)\approx0.1000.

Using the unrounded statistic gives $Z_2\approx1.28136$, which rounds to 1.281. Combining the stagewise p-values gives

X_F=-2\ln(0.04648\times0.1000) \approx-2\ln(0.004648) \approx10.74.

The 4-degree-of-freedom upper-tail probability is $e^{-10.74/2}(1+10.74/2)\approx0.0296$. This is below 0.05, so the combined test rejects $H_0$ at the 5% level. The conclusion depends on using the standard error for the selected $n_2=40$; it does not treat stage two as if its size had been fixed at another value.

Worked Example: A Different Interim Result Selects 20 Observations

Use the same hypothetical model and one-sided null hypothesis. Suppose a prespecified rule selects $n_2=20$ when $Z_1<0$ and $n_2=40$ when $Z_1\ge0$. The observed first-stage statistic is $Z_1=-0.400$, and the second-stage sample mean is $1.677$ units. Calculate Fisher’s statistic and combined p-value.

The first-stage p-value is $p_1=1-\Phi(-0.400)=\Phi(0.400)\approx0.6554$. Since $Z_1<0$, the rule selects $n_2=20$. The stage-two standard error is $10/\sqrt{20}\approx2.236$ units, giving $Z_2=1.677/2.236\approx0.750$ and $p_2=1-\Phi(0.750)\approx0.2266$.

The product and Fisher statistic are

p_1p_2\approx0.6554(0.2266)=0.1485, \qquad X_F=-2\ln(0.1485)\approx3.814.

Using the 4-degree-of-freedom tail formula, the combined p-value is approximately $e^{-3.814/2}(1+3.814/2)\approx0.432$. At $\alpha=0.05$, this result does not reject $H_0$. The different selected sample size changes the stage-two standard error and p-value, but it does not require a different Fisher reference distribution under the stated assumptions.

Diagnostics, Alternatives, and Limitations

Before interpreting the combined p-value, check how each stagewise p-value was produced. Under the null, ask whether the first-stage test is calibrated and whether the second-stage test remains calibrated conditional on the complete adaptation information. In particular, verify that the selected sample size, variance estimator, endpoint, and direction of testing match the rule and test actually used. A p-value calculated with the wrong standard error is not repaired by combining it with a valid p-value from another stage.

Data-dependent size is not the same as invalid p-values. Choosing $n_2$ from stage one is compatible with the result when stage two is independent and its p-value is conditionally valid at that selected size.
Do not reuse stage-one observations as if they were new stage-two data. If data overlap, the independence argument above does not apply without a method that accounts for their dependence.
Keep the test direction fixed. Choosing a one-sided direction after seeing the results changes the p-value’s null distribution unless that selection is accounted for.
Do not apply this two-stage formula to an unhandled early stop. If a stage is omitted or the trial stops early, the test must reflect that stopping rule; simply inserting an arbitrary p-value is not justified.

Fisher’s method combines evidence through the product of p-values, and the 4-degree-of-freedom calibration does not require fixed stage sizes or stage weights. An inverse-normal combination instead transforms stagewise p-values to normal scores and combines them with specified weights. A pooled fixed-design test combines observations through a single statistic, but its usual fixed-design calibration need not hold when sample size depends on interim results. These approaches answer related questions but are not interchangeable: each requires its own valid calibration, and Fisher’s test does not directly provide a pooled effect estimate.

The exact chi-square reference distribution also has limits. It relies on exact continuous uniform p-values under $H_0$. With discrete outcomes, conservative p-values may make the test conservative; ties and other departures from exact uniformity need explicit attention. The argument also does not guarantee good power for every alternative or make an unplanned adaptation acceptable. It establishes type I error control under the stated null assumptions, not universal optimality.

Common mistake: “The second-stage sample size depends on stage-one results, so $p_2$ cannot be uniform and Fisher’s chi-square test is invalid.” This reasoning confuses a random sample size with an invalid conditional test. If $p_2$ is Uniform$(0,1)$ conditional on the first-stage information and selected size, its null distribution does not depend on the adaptation; the Fisher calibration follows. The opposite mistake is also serious: assuming independence alone is sufficient even when the stage-two p-value was calculated using an invalid test or the wrong variance.

Key takeaway: Fisher’s two-stage statistic has a $\chi^2_4$ null distribution when the first-stage p-value is exact Uniform$(0,1)$ and the second-stage p-value is conditionally exact Uniform$(0,1)$ given the information that determines the adaptation. A variable second-stage size is permissible; invalid conditional calibration is not.

Check Your Understanding

Answer each question, then check the solutions below.

For $p_1=0.050$ and $p_2=0.200$, calculate Fisher’s statistic.
What conditional property of $p_2$ is needed when $n_2$ is selected from first-stage results?
Under what conditions does Fisher’s two-stage statistic have a chi-square distribution with 4 degrees of freedom?
Why is a stage-two p-value based on the wrong standard error a problem even if stage-two observations are independent of stage one?
Does the two-stage derivation automatically validate a procedure that stops after stage one? Explain briefly.

Solutions. (1) $p_1p_2=0.0100$, so $X_F=-2\ln(0.0100)=9.210$. (2) Under $H_0$, $P(p_2\le u\mid\mathcal{F}_1)=u$ for all $0\le u\le1$, where $\mathcal{F}_1$ includes the information used to choose $n_2$. (3) Both stagewise p-values must be exact Uniform$(0,1)$ under the null, and $p_2$ must be conditionally uniform given first-stage information; this yields independent $\chi^2_2$ transforms whose sum is $\chi^2_4$. (4) Independence of observations does not make a miscalibrated statistic a valid p-value, so conditional uniformity may fail. (5) No. An early-stop rule changes which stages contribute and must be included in a calibrated procedure.