Estimating Familywise Error by Simulation

What Is Being Estimated?

A MAMS design can make decisions about several treatment comparisons, possibly at multiple stages and after dropping arms. The familywise error rate (FWER) is not the probability of rejecting any hypothesis whatsoever: it is the probability of rejecting at least one hypothesis that is in fact true. This distinction matters whenever a simulated scenario includes both effective and ineffective treatments.

Suppose the design includes \(J\) null hypotheses, indexed by \(j=1,\ldots,J\), and let \(R_j=1\) if the trial procedure rejects \(H_j\), and \(R_j=0\) otherwise. For a specified parameter configuration \(\theta\), define the set of true null hypotheses as \(T(\theta)=\{j:H_j\text{ is true under }\theta\}\). The familywise error rate at that configuration is:

\operatorname{FWER}(\theta) = P_{\theta}\left(\sum_{j\in T(\theta)} R_j \geq 1\right).

The probability is taken over the trial's random outcomes and any randomization represented in the simulation, with the design and its decision rules held fixed. Each hypothesis belongs to the family according to the protocol's definition of its comparisons. If a comparison is examined repeatedly at interim analyses, it is still one hypothesis for this definition; the simulated procedure must nevertheless apply all its stage-specific rules before deciding whether that hypothesis is rejected.

Definition: A global-null configuration makes every hypothesis in the family true. A partial-null configuration makes some hypotheses true and others false. The simulated FWER is the probability of at least one rejection among the true null hypotheses in the specified configuration.

Under the global null, any rejection is a false rejection. Under a partial null, rejections of false null hypotheses do not count as familywise errors. Estimating FWER under the global null assesses weak error control. Assessing strong error control requires considering configurations in which some, but not all, null hypotheses are true; the target is control for every configuration in the intended parameter space, not just the global null. Simulation at a finite set of configurations estimates error at those configurations; it does not by itself establish a bound over every possible configuration.

Turn the Trial Procedure Into a Replicate-Level Event

Each simulated replicate must represent one complete trial under a specified configuration. Generate the outcomes and any allocation or accrual features included in the simulation, construct the interim data, apply the stopping and arm-dropping rules, and record the final rejection decisions. This is the same trial logic used for the operating-characteristic targets and simulation implementation developed in the earlier material on simulation targets and translating a MAMS protocol into simulation logic. The familywise error calculation is meaningful only if the simulated decisions follow the design being evaluated.

For replicate \(b\), define an indicator \(I_b(\theta)\) equal to one if at least one true null hypothesis is rejected, and zero otherwise. If \(B\) independent replicates are simulated under the same configuration, the estimator is:

\widehat{\operatorname{FWER}}(\theta) = \frac{1}{B}\sum_{b=1}^{B} I_b(\theta).

The numerator is the number of replicates with one or more false rejections. It is not the total number of false rejections across comparisons: a replicate with two false rejections contributes one event, just as a replicate with one false rejection does. The denominator is the number of completed, valid replicates under that configuration.

The basic justification is direct. Under independent, identically generated replicates from the specified simulation model, each indicator has probability \(\operatorname{FWER}(\theta)\) of equaling one. Therefore:

E_{\theta}\!\left[\widehat{\operatorname{FWER}}(\theta)\right] = E_{\theta}\!\left[\frac{1}{B}\sum_{b=1}^{B}I_b(\theta)\right] = \frac{1}{B}\sum_{b=1}^{B}E_{\theta}[I_b(\theta)] = \operatorname{FWER}(\theta).

Thus the simulated proportion is an unbiased estimate of the FWER for the specified model and configuration. It is not an exact value for the design: a different set of random trial outcomes can produce a different proportion. Nor does a low estimate prove error control in unexamined configurations.

Worked Example: Estimating Error Under the Global Null

Consider a hypothetical three-arm design with two active-versus-control hypotheses, \(H_1\) and \(H_2\). Simulate \(B=1{,}000\) complete trials with both treatment effects set to zero, so both hypotheses are true. Suppose 44 replicates reject exactly one hypothesis and 17 reject both. The remaining 939 reject neither.

Every rejection in this scenario is false. A replicate with both rejections still counts as one familywise error event, so the event count is \(44+17=61\). The estimate is:

\widehat{\operatorname{FWER}} = \frac{61}{1{,}000} = 0.061.

The estimated global-null FWER is 6.1% for this hypothetical simulation and implementation. It is not 6.1% per comparison, and it is not a guarantee that the true FWER equals 6.1%. The estimate reflects the whole decision procedure, including its interim rules.

Partial Nulls Change Which Decisions Count

In a partial-null scenario, first identify which hypotheses are true under the treatment effects used to generate the data. Then count a replicate as an error if and only if the procedure rejects at least one of those true hypotheses. A rejection of a false null is a correct rejection for the purpose of FWER, even if it occurs in the same replicate as a false rejection.

Worked Example: Counting Errors With One Effective Treatment

In a hypothetical three-arm trial, let \(H_1\) compare treatment 1 with control and \(H_2\) compare treatment 2 with control. Set the treatment 1 effect to zero and the treatment 2 effect to a prespecified nonzero value. Thus \(H_1\) is true and \(H_2\) is false. Across 1,000 simulated replicates, suppose the procedure rejects \(H_1\) in 12 replicates where it also rejects \(H_2\), in 14 where it does not reject \(H_2\), and rejects \(H_1\) in no other replicate. It rejects \(H_2\) in 420 replicates in total.

Only rejection of \(H_1\) is a false rejection in this configuration. The 12 replicates rejecting both hypotheses count as familywise errors, as do the 14 rejecting \(H_1\) alone. Rejections of \(H_2\) alone do not count. Therefore:

\widehat{\operatorname{FWER}} = \frac{12+14}{1{,}000} = 0.026.

The estimated FWER is 2.6%. Dividing the 420 rejections of \(H_2\) by 1,000 would estimate a rejection probability for a false null, not FWER. Counting every replicate with any rejection would also be wrong because it would include correct rejections of \(H_2\) alone.

Evaluate Configurations, Not Just One Convenient Case

The global-null estimate is useful, but it cannot answer whether error is controlled when some treatments truly work. To assess strong error control by simulation, specify relevant partial-null configurations, keep track of the true-null set for each, and estimate FWER separately in each configuration. For a finite set \(\Theta_{\mathrm{eval}}\) of evaluated configurations, the largest estimate is:

\max_{\theta\in\Theta_{\mathrm{eval}}} \widehat{\operatorname{FWER}}(\theta).

This maximum summarizes the simulated results over the configurations evaluated. It should not be confused with the supremum of the true FWER over all possible parameter configurations. The latter is the target for strong control; a finite simulation exercise samples only selected cases.

Worked Example: Comparing Global and Partial Null Results

Suppose a hypothetical three-comparison design is evaluated with 1,200 replicates per configuration. The global-null scenario produces 48 familywise error events. In partial-null scenario A, where two hypotheses are true and one is false, there are 71 events. In partial-null scenario B, where a different single hypothesis is true, there are 52 events.

Compute each proportion separately:

\widehat{\operatorname{FWER}}_{\mathrm{global}} = \frac{48}{1{,}200} = 0.040, \qquad \widehat{\operatorname{FWER}}_{A} = \frac{71}{1{,}200} \approx 0.0592, \qquad \widehat{\operatorname{FWER}}_{B} = \frac{52}{1{,}200} \approx 0.0433.

Among these three evaluated cases, scenario A has the largest estimated FWER, approximately 5.92%. That result indicates why checking only the global null can be insufficient: the largest estimate here occurs when both true and false null hypotheses are present. It does not establish that scenario A is the worst possible configuration.

Diagnostics, Alternatives, and Limitations

Before interpreting a simulated proportion, verify that each replicate's event indicator corresponds to the correct truth set and the actual design decisions. Useful checks include:

Truth-set check: confirm that the treatment effects used for generation make exactly the intended hypotheses true or false.
Decision check: verify that arm dropping, stopping, and terminal rejection rules are applied in the correct order and use the intended data snapshot.
Counting check: count one event per replicate if any true null is rejected, not one event per rejected true null.
Denominator check: use the number of valid completed replicates under that configuration, and investigate missing or failed replicates rather than silently excluding them.
Scenario check: report global-null and partial-null results distinctly. A single pooled proportion across different configurations generally has no simple interpretation as FWER at any one configuration.

Simulation is especially useful when the design's stages, shared control, and adaptive rules make a direct calculation difficult. When a reliable analytic calculation is available, it can provide a complementary check for cases covered by its assumptions. A comparison of marginal rejection rates for individual hypotheses is not a substitute for FWER: the familywise event depends on the joint pattern of rejections. Likewise, treating each interim look as an unrelated test would fail to represent the actual procedure if the design has a specified rule for combining information across stages.

The estimate is tied to the assumed outcome model, parameter configuration, allocation scheme, and implemented decision rules. Misspecified data generation or a coding error can produce a precise-looking but irrelevant proportion. Finite simulation also has random error, so small differences between estimates should not automatically be treated as meaningful. The present calculation identifies what is estimated and how; it does not turn a finite set of scenarios into a proof of strong control.

A Practical Calculation Sequence

Specify the configuration.
State the treatment effects or other generating parameters and identify the true-null set \(T(\theta)\).

Run complete trial replicates.
Generate each trial under the specified model and apply the full MAMS decision procedure through its terminal decisions.

Record one binary event per replicate.
Set \(I_b=1\) if at least one hypothesis in \(T(\theta)\) is rejected; otherwise set \(I_b=0\).

Calculate and interpret the proportion.
Divide the event count by the valid replicate count, state the configuration, and repeat separately for each configuration of interest.

Common Mistakes

Using any rejection as the event in a partial-null case. That counts correct rejections of false nulls. The event is rejection of at least one true null.
Counting rejections instead of affected replicates. Two false rejections in one trial are one familywise error event, not two.
Reporting the global-null estimate as proof of strong control. It assesses only the all-null configuration; partial-null scenarios must also be evaluated.
Calling a simulated proportion the exact FWER. It is an estimate based on a finite set of random replicates and a specified simulation model.
Ignoring stages or adaptation. Applying only a terminal comparison, rather than the full protocol's decision logic, estimates the error of a different procedure.

Key takeaway: For each parameter configuration, define which hypotheses are true, simulate the entire trial procedure, and estimate FWER as the proportion of replicates rejecting one or more of those true nulls. Global-null and partial-null estimates answer different questions.

Check Your Understanding

Answer each question, then compare with the short solutions below.

In a global-null simulation with 2,000 replicates, 86 replicates reject at least one hypothesis. What is the estimated FWER?
In a partial-null configuration, \(H_1\) is true and \(H_2\) is false. A replicate rejects \(H_2\) but not \(H_1\). Does it count as a familywise error? Why?
Across 500 replicates, \(H_1\) is true in every replicate. It is rejected in 8 replicates, including 3 in which another true null is also rejected. How many familywise error events are there?
Why does an estimated FWER under the global null alone not demonstrate strong error control?

Solutions. (1) The estimate is \(86/2{,}000=0.043\), or 4.3%. (2) No. Only a false null was rejected, so no true null was rejected in that replicate. (3) There are 8 events: the 3 replicates with multiple false rejections are counted once each, and the other 5 rejections of \(H_1\) are also events. (4) The global null has every hypothesis true; it does not evaluate configurations where some nulls are false and others remain true, which are also required when assessing strong error control.