Simulation Workshop: Checking Combination-Test Size

Why Simulate Combination-Test Size?

A combination test is designed to control the probability of rejecting the global null hypothesis when that null is true. A derivation establishes this control only under its stated conditions, such as valid stagewise p-values, independence where required, and weights fixed independently of the stage results. Simulation provides a practical check: repeatedly generate data under the global null, calculate the stagewise p-values, apply the planned combination test, and count how often it rejects.

This workshop focuses on a two-stage, one-sided test at significance level \(\alpha=0.05\). The direction of evidence is the same at both stages: smaller stagewise p-values indicate evidence against the global null. The test weights and rejection rule are fixed before the simulated stage results are examined.

Definition: The size of a test is its probability of rejection when the null hypothesis is true. Under a global-null simulation, the Monte Carlo estimate of size is \(\hat{r}=R/B\), where \(B\) is the number of simulated trials and \(R\) is the number whose combination test rejects. This estimate has simulation error because it is based on a finite number of trials.

Specify the Tests Before Simulating

Let \(p_1\) and \(p_2\) be the two stagewise p-values. For the inverse normal combination test, choose fixed nonnegative weights \(w_1,w_2\) satisfying \(w_1^2+w_2^2=1\). The combination statistic and rejection rule are:

Z_{\mathrm{IN}}=w_1\Phi^{-1}(1-p_1)+w_2\Phi^{-1}(1-p_2), \qquad \text{reject }H_0\text{ if }Z_{\mathrm{IN}}\geq z_{1-\alpha}.

Here \(\Phi\) is the standard normal cumulative distribution function and \(z_{1-\alpha}=\Phi^{-1}(1-\alpha)\). For Fisher’s combination test, use:

T_{\mathrm{F}}=-2\{\log(p_1)+\log(p_2)\}, \qquad \text{reject }H_0\text{ if }T_{\mathrm{F}}\geq \chi^2_{4,\,1-\alpha}.

The Fisher critical value is the \(1-\alpha\) quantile of a chi-square distribution with four degrees of freedom. If the stagewise p-values are independent Uniform\((0,1)\) random variables under the global null, then \(Z_{\mathrm{IN}}\sim N(0,1)\) for the inverse normal test and \(T_{\mathrm{F}}\sim\chi^2_4\) for Fisher’s test. Thus, both tests have size \(\alpha\) under those conditions. For \(\alpha=0.05\), the critical values are approximately \(1.6449\) and \(9.4877\), respectively.

These reference distributions should not be assumed automatically. In particular, stagewise p-values that are dependent need not produce either reference distribution. A simulation should reproduce the data-generating process and the stagewise testing procedure that the design will actually use. Simulating p-values directly can check how the combination rule responds to specified p-value distributions, but it cannot establish that the real stagewise tests produce valid p-values.

A Reproducible Simulation Plan

For each simulated trial, generate the data under the global null, calculate both stagewise p-values, apply each prespecified combination rule, and record whether each test rejects. Use the same simulated stagewise p-values for the two tests when comparing them; this makes the comparison easier to interpret. Keep the weights, significance level, sidedness, and handling of any unusual outcomes fixed across replicates.

Define the null-generating mechanism.
Specify how data are generated under the global null, including any dependence between stages, discreteness, nuisance-parameter settings, and trial adaptations that affect the stagewise tests.

Compute stagewise p-values.
For each replicate, run the planned stagewise procedures and retain \(p_1\) and \(p_2\). Confirm that their direction and definitions match the combination formulas.

Apply the fixed rejection rules.
Calculate \(Z_{\mathrm{IN}}\) and \(T_{\mathrm{F}}\), then record a rejection for each test if its statistic crosses its prespecified critical value.

Summarize uncertainty.
After \(B\) replicates, report the rejection count \(R\), the estimated size \(\hat{r}=R/B\), and its Monte Carlo standard error.

Conditional on the design being simulated, the rejection indicator in each replicate is Bernoulli with rejection probability \(r\), the size under that simulation. If replicates are independent, the estimate has approximate standard error:

SE(\hat{r})=\sqrt{\frac{\hat{r}(1-\hat{r})}{B}}.

This standard error describes Monte Carlo variability, not uncertainty about all possible clinical-trial populations or all nuisance-parameter values. A rough 95% Monte Carlo interval is \(\hat{r}\pm1.96SE(\hat{r})\); for very small rejection counts, a binomial interval designed for sparse counts is preferable. When the target is 0.05, \(B=100{,}000\) gives an approximate Monte Carlo standard error of \(0.000689\) if the observed rate is near 0.05. Consequently, a small difference from 0.05 may be ordinary simulation noise rather than meaningful miscalibration.

Worked Example: Independent Uniform Stagewise P-Values

Consider a hypothetical global-null simulation with \(B=100{,}000\) replicates. In each replicate, generate \(p_1\) and \(p_2\) independently from Uniform\((0,1)\). Use equal inverse normal weights \(w_1=w_2=1/\sqrt{2}\), and use \(\alpha=0.05\) for both tests.

The equal weights satisfy \(w_1^2+w_2^2=1/2+1/2=1\). The inverse normal rule rejects when \(Z_{\mathrm{IN}}\geq1.6449\), and Fisher’s rule rejects when \(T_{\mathrm{F}}\geq9.4877\). Suppose the simulation produces 5,008 inverse normal rejections and 4,976 Fisher rejections. The estimated sizes are:

\hat{r}_{\mathrm{IN}}=\frac{5{,}008}{100{,}000}=0.05008, \qquad \hat{r}_{\mathrm{F}}=\frac{4{,}976}{100{,}000}=0.04976.

For the inverse normal estimate, the Monte Carlo standard error is:

SE(\hat{r}_{\mathrm{IN}}) =\sqrt{\frac{0.05008(1-0.05008)}{100{,}000}} \approx 0.0006898.

For Fisher’s estimate, it is:

SE(\hat{r}_{\mathrm{F}}) =\sqrt{\frac{0.04976(1-0.04976)}{100{,}000}} \approx 0.0006876.

Both estimates are close to the target 0.05, and their differences from it are small relative to their Monte Carlo standard errors. This is consistent with calibration, as expected for independent uniform stagewise p-values. It does not prove calibration for other p-value distributions or for a different data-generating mechanism.

Checking a Different Stagewise Distribution

Valid p-values need not be exactly uniform. For a null hypothesis, a p-value is valid if \(P(p_i\leq t)\leq t\) for every \(t\) between 0 and 1. A discrete test, for example, may produce a conservative p-value whose distribution places less probability near zero than a uniform distribution. When stagewise p-values are independent and valid, the combination statistics are no more likely to be extreme than under independent uniform p-values: each p-value can be coupled to a uniform value that is no larger, and both combination statistics decrease when either p-value increases. This gives a useful justification for conservatism, but simulation shows its possible size in a particular case.

Worked Example: Independent Discrete Valid P-Values

Suppose, hypothetically, that each stagewise p-value independently takes values \(0.01,0.05,0.20,1\) with probabilities \(0.01,0.04,0.15,0.80\), respectively. This is a valid, conservative p-value distribution: its cumulative probabilities at the nontrivial support points are \(P(p\leq0.01)=0.01\), \(P(p\leq0.05)=0.05\), and \(P(p\leq0.20)=0.20\); between support points the cumulative probability does not exceed the corresponding threshold.

For the inverse normal test with equal weights, the rejection rule is equivalent to:

\Phi^{-1}(1-p_1)+\Phi^{-1}(1-p_2) \geq \sqrt{2}(1.6449)\approx2.3262.

A pair rejects if both p-values are at most \(0.20\) and they are not both \(0.20\). For example, the transformed values for \(0.01\) and \(0.20\) sum to about \(2.3263+0.8416=3.1679\), above the cutoff; for \(0.20\) and \(0.20\), the sum is only \(1.6832\). The probability of rejection is therefore \(0.20^2-0.15^2=0.0175\).

For Fisher’s test, rejection requires:

p_1p_2\leq \exp(-9.4877/2)\approx0.00869.

The qualifying pairs are one \(0.01\) value paired with a value at most \(0.20\), or two \(0.05\) values. Their total probability is \(2(0.01)(0.20)-(0.01)^2+(0.04)^2=0.0055\).

In a hypothetical simulation of \(100{,}000\) replicates, suppose the inverse normal test rejects 1,741 times and Fisher rejects 557 times. The respective estimated sizes are \(0.01741\) and \(0.00557\), reasonably close to the exact probabilities \(0.0175\) and \(0.0055\) given the simulation’s finite size. Both are below 0.05, but they are not equally conservative. This illustrates why reporting only whether a procedure is “valid” can conceal important differences in operating characteristics.

Diagnosing Dependence and Other Failures

A simulation can also expose a failure of the independence assumption. For example, stages may share data or controls, or the second stage may reuse information from the first. The following example intentionally uses complete dependence to make the effect transparent.

Worked Example: The Same Uniform P-Value at Both Stages

For each hypothetical null replicate, draw \(U\sim\text{Uniform}(0,1)\) and set \(p_1=p_2=U\). Each stagewise p-value is marginally uniform and hence valid, but the two p-values are perfectly dependent. With equal inverse normal weights, the statistic becomes \(\sqrt{2}\Phi^{-1}(1-U)\). The test rejects when:

U\leq \Phi\left(-\frac{1.6449}{\sqrt{2}}\right)\approx0.1224.

Thus, the inverse normal test’s size is about \(0.1224\), not \(0.05\). For Fisher’s test, \(T_{\mathrm{F}}=-4\log(U)\), so rejection occurs when:

U\leq\exp(-9.4877/4)\approx0.0933.

Fisher’s size is therefore about \(0.0933\). A simulation with \(100{,}000\) replicates might produce rejection estimates near these values. Such an outcome is not Monte Carlo noise around the intended 0.05: the reference calibration fails because the independent-p-value reference distributions were used for dependent inputs.

When an estimated size is unexpectedly high or low, check the simulation in a structured way:

Check the null generator. Confirm that the simulated treatment effects satisfy the global null and that nuisance parameters and allocation rules are implemented as intended.
Check the stagewise p-values. Under the null, inspect their marginal distributions and their joint dependence. For valid p-values, an empirical cumulative probability at a threshold \(t\) should not systematically exceed \(t\), allowing for simulation uncertainty.
Check the combination inputs. Verify p-value direction, weights, sidedness, and critical values. Do not silently alter a weight after inspecting simulated or observed stage results.
Check the Monte Carlo precision. Increase \(B\) if the uncertainty around the rejection estimate is too large to distinguish acceptable calibration from a concerning deviation.

Choosing What to Simulate and What to Conclude

There are two complementary simulation strategies. A p-value-level simulation supplies stagewise p-values from specified distributions, making it easy to isolate the combination rule, as in the examples above. A trial-level simulation generates patient outcomes and reruns the planned stagewise analyses. The latter is more demanding but can reveal problems in the construction of the p-values themselves, including discreteness, nuisance-parameter effects, missing outcomes, or dependence between stages. A useful calibration assessment may use several null scenarios rather than one convenient setting.

Approach	Strength	Limitation
Analytic calibration under independent uniform p-values	Gives the reference size directly under the stated conditions.	Does not diagnose departures from those conditions.
Simulation from specified p-value distributions	Isolates the combination rule and can explore discreteness or dependence.	Cannot verify how the real stagewise procedures generate their p-values.
Trial-level global-null simulation	Can assess the full planned design and analysis under chosen null scenarios.	Requires a credible data-generating model and enough computation to estimate small error rates precisely.

Simulation is a diagnostic and design-evaluation tool, not a substitute for stating assumptions. It has finite Monte Carlo error, depends on the adequacy of the null-generating model, and may miss rare scenarios if those scenarios were not simulated. If the null is composite, a reassuring result at a single nuisance-parameter value does not establish control over all null values; relevant settings need to be considered separately. Likewise, a result close to 0.05 under one dependence pattern does not establish calibration under another.

Common Mistake: Treating a Close Estimate as a Proof

A tempting conclusion is: “The simulation gave a rejection rate near 0.05, so the test controls type I error.” That reasoning overstates what the result establishes. The estimate describes performance under the simulated null mechanism, subject to Monte Carlo uncertainty. It says nothing by itself about null mechanisms that were not simulated, and it cannot validate assumptions that the simulation imposed rather than checked.

Key takeaway: Report the null scenario, the stagewise p-value construction, dependence assumptions, \(B\), the rejection count, the estimated size, and Monte Carlo uncertainty. Interpret calibration as evidence for the simulated settings—not as proof of universal error control.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

For \(B=50{,}000\) null replicates and \(R=2{,}580\) rejections, calculate the estimated size and approximate Monte Carlo standard error.
Why can independent, conservative stagewise p-values lead to a combination-test rejection rate below \(\alpha\)?
If both stagewise p-values are valid Uniform\((0,1)\) marginally but are dependent, does the independent-p-value Fisher reference distribution necessarily apply?
Name two different issues that a trial-level simulation can assess which a simulation that directly generates p-values cannot assess on its own.

Solutions. (1) \(\hat{r}=2{,}580/50{,}000=0.05160\). Its Monte Carlo standard error is \(\sqrt{0.05160(0.94840)/50{,}000}\approx0.0009893\). (2) A conservative p-value is less likely than a uniform p-value to fall near zero. With independent inputs and nonnegative fixed weights, this reduces the chance that either monotone combination statistic reaches its rejection threshold. (3) No. The Fisher chi-square reference requires independent uniform p-values under the global null; marginal uniformity alone is insufficient. (4) Examples include whether the stagewise tests actually produce valid p-values, and whether data sharing or trial procedures induce dependence or discreteness. A p-value-level simulation can explore specified distributions, but does not establish how those p-values arise from patient data.