Why Missing Outcomes Matter to a Combination Test
A combination test uses stagewise p-values as inputs. Its calibration depends not only on the formula that combines them, but also on whether each input is valid for its stage and whether the joint behavior of the inputs matches the calibration used. Missing outcomes can threaten the first requirement before any combination calculation begins: the p-value calculated from observed participants may no longer have its claimed null distribution.
For example, a response analysis among participants with observed outcomes can be misleading if outcome observation is related to response. The resulting p-value may be too small under the null, and a mathematically correct combination of such inputs does not repair that problem. Conversely, a stage may have no usable outcome data at all. That is not automatically equivalent to observing a valid p-value of a particular value. The handling rule must be chosen and justified in advance.
For standard inverse-normal or Fisher calibration, one commonly assumes independent, continuous uniform stagewise p-values under the null. Other designs may use conditional validity or a different justified joint calibration. In every case, the relevant stagewise tests and their missing-data analyses must support the assumptions being used. The earlier material on stagewise validity and combination-test calibration provides the combination framework; here the focus is what missing outcomes do to its inputs.
How Missingness Can Change a Stagewise Test
A useful description of missingness distinguishes three mechanisms. Outcomes are missing completely at random (MCAR) if missingness is unrelated to observed or unobserved outcomes and relevant covariates. They are missing at random (MAR) if, conditional on specified observed information, missingness does not depend on the unobserved outcome. They are missing not at random (MNAR) if missingness can still depend on the unobserved outcome after conditioning on the observed information.
These labels concern assumptions about the data-generating and observation processes; they cannot usually be established just by looking at the observed outcomes. Under MCAR, a complete-case analysis can sometimes retain validity, though it may lose precision. Under MAR, a correctly specified likelihood or multiple-imputation analysis can support valid inference if it accounts for uncertainty and the model assumptions hold. Neither statement licenses an unexamined analysis: the estimand, model, and randomization or sampling structure still matter. Under MNAR, observed data alone generally do not identify the missing-outcome distribution without additional assumptions. Sensitivity analyses are therefore important.
Worked Example: Complete Cases Do Not Reveal the Full Response Rate
Suppose, hypothetically, 100 participants in one randomized trial arm have a binary response outcome. Outcomes are observed for 60 participants, of whom 42 responded; the other 40 outcomes are missing. The observed-case response proportion is $$ \hat{p}_{\mathrm{obs}}=\frac{42}{60}=0.700. $$
This is the response proportion among participants whose outcomes were observed. If the target is the response proportion among all 100 participants, the observed data alone do not determine it. If all 40 missing participants were nonresponders, the full-arm proportion would be \(42/100=0.420\). If all 40 were responders, it would be \(82/100=0.820\). Thus the full-arm response proportion is bounded by 0.420 and 0.820 without further assumptions.
The complete-case value 0.700 is not automatically an unbiased estimate of the full-arm response proportion. It would be plausible under suitable missingness assumptions, but if response affects whether an outcome is observed, the observed cases may be a selected subset. A p-value calculated as if the 60 observed outcomes were an unselected sample could then fail to be super-uniform under the null. The arithmetic of the test cannot establish the missingness assumption.
Calculate and Check the Stagewise Input First
Before combination, identify the estimand and the exact analysis population, then establish how missing outcomes enter the stagewise test. The standard error and reference distribution must correspond to that analysis. The p-value should be computed in the prespecified direction and with the appropriate treatment of missingness. If a two-sided p-value is converted to a signed normal score for an inverse-normal test, the sign must come from the estimated effect, not from the p-value alone.
Worked Example: A Valid Stagewise Test as an Inverse-Normal Input
Consider hypothetical stage data analyzed by a prespecified test with a valid normal approximation and an estimated treatment effect of 0.700 units. The null value is 0.500 units and the model-based standard error is 0.08226 units. Assume the outcome analysis properly accounts for missing outcomes under its stated assumptions, so the resulting stagewise test is valid. The test statistic is $$ z_1=\frac{0.700-0.500}{0.08226}=2.431. $$
For a two-sided test, the p-value is $$ p_1=2P(Z\geq 2.431)=0.01506,\qquad Z\sim N(0,1). $$ The estimate is in the beneficial direction, so the corresponding signed normal score for a one-sided combination test is \(z_1=2.431\), equivalently the one-sided stagewise p-value is approximately \(0.007529\). The two-sided p-value by itself does not encode direction.
Suppose a second independent stage has a valid one-sided p-value of 0.1000 in the same beneficial direction. Its normal score is \(z_2=\Phi^{-1}(0.9000)=1.2816\). With prespecified equal inverse-normal weights \(w_1=w_2=1/\sqrt{2}\), the combined statistic is $$ Z_{\mathrm{comb}}=\frac{2.431+1.2816}{\sqrt{2}}=2.625. $$
The one-sided combined p-value is \(P(Z\geq 2.625)\approx 0.00433\). This calculation is justified only if the stagewise inputs and their joint calibration meet the test's assumptions. In particular, the valid stagewise p-value assumed here cannot be obtained merely by applying the same formula to a selected subset of outcomes.
A useful diagnostic is to compare outcome availability across treatment groups, stages, and prognostic factors, and to examine reasons and timing of missingness. Large differences raise concerns but do not, by themselves, prove MNAR. Also check whether the missingness process changed after an interim result or operational decision: conditioning only on the final observed cases may then fail to reflect the design.
Defensible Ways to Handle Missing Combination Inputs
The first choice, when feasible, is to obtain the outcome rather than replace it statistically: continue protocol-specified follow-up, retrieve records under appropriate governance, or wait for endpoint maturation. This can reduce missingness, but it does not remove the need to address any outcomes that remain unavailable.
If outcomes remain missing, use a prespecified stagewise method whose assumptions match the estimand and data. Depending on the endpoint and design, this might be a likelihood analysis or multiple imputation under MAR, with uncertainty properly reflected in inference. For plausible departures from MAR, prespecified sensitivity analyses can show how conclusions change across assumptions. A plug-in imputation that fills missing outcomes with a single predicted value and treats them as observed generally understates uncertainty and need not yield a valid p-value.
A different issue arises when a whole stage has no usable input. A prespecified rule can define how the combination test operates with that stage unavailable, but it must retain valid calibration. For example, under independent valid normal scores, retaining the original inverse-normal weights and assigning zero score to a stage known to be absent gives the remaining scores a variance no greater than one. This is conservative against a fixed standard-normal critical value if absence is determined independently of the stage evidence. It is not a universal rule: if stage absence depends on outcomes or interim evidence, the distribution conditional on that absence may differ.
Worked Example: A Prespecified Fisher Rule for an Unavailable Stage
Suppose a three-stage trial prespecifies that, if one stage has no usable outcomes for an operational reason independent of its treatment evidence, that stage contributes no Fisher evidence and the degrees of freedom are based on the number of available stages. Hypothetically, the available stagewise p-values are \(p_1=0.0400\) and \(p_3=0.2000\), while stage 2 is unavailable. Assume the two available p-values are independent and exactly uniform under the null.
The Fisher statistic from the two available inputs is $$ X=-2\{\ln(0.0400)+\ln(0.2000)\} =-2\ln(0.00800)=9.657. $$ Under the stated independence and null assumptions, \(X\) has a chi-square distribution with \(2(2)=4\) degrees of freedom. Its upper-tail p-value is approximately 0.0466. The reference distribution has four, not six, degrees of freedom because only two valid p-values contribute.
This is defensible only under the prespecified availability rule and stated assumptions. If the stage was stopped, omitted, or rendered unavailable because of its emerging outcome evidence, simply recalibrating with four degrees of freedom may not give a valid test. The probability of the availability event and the evidence conditional on that event would need to be addressed by an appropriate design-specific calibration.
For inverse-normal combination, setting an unavailable stage's score to zero is not the same as inserting \(p=1\): \(p=1\) maps to a negative infinite normal score. Renormalizing weights over available stages can also change the test. Such a change may be valid under an appropriately prespecified rule and suitable conditional validity, but it is not justified merely because fewer inputs happened to be observed. In particular, selecting a rule after seeing the stage results can select for favorable evidence.
Comparing Strategies and Recognizing Failure
| Strategy | Potential strength | Key condition or limitation |
|---|---|---|
| Continue follow-up or retrieve outcomes | Reduces missingness and avoids some modeling assumptions | May be impractical; remaining missing outcomes still require a plan |
| Prespecified likelihood or multiple-imputation analysis | Can use observed covariate and outcome information efficiently | Validity depends on the estimand, model, missingness assumptions, and proper uncertainty accounting |
| Sensitivity analysis for departures from MAR | Shows whether conclusions rely on unverifiable assumptions | Does not identify a unique answer without additional assumptions |
| Prespecified unavailable-stage combination rule | Can retain a calibrated test when a stage contributes no usable data | Must account for availability and the null distribution; cannot be chosen opportunistically |
| Drop, replace, or reweight an input after seeing results | May appear computationally convenient | Can alter the null distribution and invalidate type I error control |
Warning signs include unexpectedly different missingness rates by arm, a relationship between observed outcome history and dropout, changes in endpoint ascertainment across stages, or an analysis whose inclusion criteria differ from the prespecified estimand. These checks are diagnostic rather than definitive proofs about the missingness mechanism. Document the number and reasons for missing outcomes by stage and treatment group, and verify that the reported stagewise test matches the analysis plan.
The central limitation is that combination methods combine valid evidence; they do not manufacture validity from incomplete or selectively observed data. When assumptions cannot be defended, report the uncertainty, conduct sensitivity analyses, and avoid interpreting a nominal combined p-value as if all outcomes had been observed.
Common Mistake
Before combining, answer three questions: Are the stagewise p-values valid under their missing-data analyses? Is the rule for an unavailable stage prespecified and calibrated for how availability arose? Does the assumed joint distribution of the available inputs still hold? If any answer is no or unknown, the usual combination p-value is not automatically justified.
Check Your Understanding
Try each question, then compare your reasoning with the brief solutions.
- In a binary-outcome stage, 30 of 50 observed participants respond, while 20 of 80 participants have missing outcomes. What is the observed-case response proportion, and what are the extreme bounds on the full-stage response proportion?
- Why does a valid combination formula not repair a stagewise p-value that is anti-conservative because outcomes were selectively observed?
- For a valid one-sided stagewise p-value of 0.0500 in the beneficial direction, what is its signed normal score to three significant figures?
- Under what conditions can a Fisher test use two available p-values and four degrees of freedom when a third stage is unavailable?
Solutions. (1) The observed-case proportion is \(30/50=0.600\). If all 20 missing participants did not respond, the full-stage proportion is \(30/80=0.375\); if all responded, it is \(50/80=0.625\). (2) Combination calibration presumes valid inputs and a suitable joint null distribution. A too-small stagewise p-value violates that premise, so the combined statistic need not have its reference distribution. (3) The score is \(\Phi^{-1}(1-0.0500)=1.64\). (4) The availability rule must be prespecified and compatible with the null calibration; the two contributing p-values must be independent and exactly uniform under the null, and stage unavailability must not induce unaccounted selection on evidence. Then Fisher's statistic has a chi-square reference distribution with \(2\times2=4\) degrees of freedom.