Multiplicity In Bayesian Trial Decisions

Why multiplicity needs trial-level simulation

A trial may evaluate several endpoints, compare several treatments with a control, or do both. If each analysis can independently support a treatment claim, the chance of at least one false-positive conclusion can exceed the chance for any single claim. In clinical trial planning, that probability is an operating characteristic of the whole decision process, not just of one endpoint or comparison.

A Bayesian rule might declare benefit when the posterior probability that a treatment effect is positive exceeds a prespecified threshold. That threshold is not, by itself, a guarantee about the repeated-sampling probability of false-positive conclusions. To assess that probability, simulate complete trials under specified truths, apply every planned analysis and decision rule in each trial, and count whether any claim is false.

Definition: Suppose a trial has \(K\) possible claims, and claim \(j\) concerns a treatment effect \(\delta_j\). Let \(D_j=1\) if the prespecified decision rule makes claim \(j\), and \(D_j=0\) otherwise. A false-positive indicator is \(F_j=D_j I(\delta_j\leq 0)\), for a rule that claims benefit when \(\delta_j>0\). The family-wise false-positive event is \(I(\sum_{j=1}^{K}F_j\geq 1)\). Its probability under a specified trial-generating scenario is the probability of at least one false-positive conclusion across the claims that are truly null or nonbeneficial.

The definition depends on the claim. If a rule claims that an effect exceeds a clinically meaningful margin \(\delta_*\), then the false-positive condition is \(\delta_j\leq\delta_*\), not necessarily \(\delta_j\leq0\). Likewise, a claim of noninferiority or safety needs a false-positive definition aligned with that claim and its decision boundary. Specify the truth and the claim before counting errors.

Simulate the joint decision process

For each simulated trial, generate all relevant outcomes for the same trial participants, using the planned sample size, allocation, outcome models, and dependence structure. Fit the planned Bayesian analysis for each endpoint or comparison, calculate its posterior quantity, and apply the prespecified decision rule. For example, if benefit is defined by \(\delta_j>0\), a rule could be \(D_j=1\) when \(P(\delta_j>0\mid\text{data})>\gamma\), where \(\gamma\) is a fixed posterior-probability threshold.

The simulation must represent dependence that would occur in the trial. Endpoints measured on the same participant may be correlated; comparisons may share a control group; and analyses may share model parameters. Simulating each claim separately and treating the resulting decisions as independent can give the wrong probability of at least one false positive. Preserve the joint structure by generating the trial data together and evaluating all its claims together.

Let \(M\) be the number of simulated trials under one specified scenario. In replicate \(m\), let \(A_m=1\) if at least one false-positive claim is made, and \(A_m=0\) otherwise. The estimated probability is the fraction of replicates with that event:

\widehat{q}=\frac{1}{M}\sum_{m=1}^{M}A_m.

Conditional on the simulation setup, the \(A_m\) are Bernoulli indicators with event probability \(q\). If the simulated trials are independent, the Monte Carlo standard error of \(\widehat{q}\) is estimated by:

\widehat{SE}_{MC}=\sqrt{\frac{\widehat{q}(1-\widehat{q})}{M}}.

This standard error measures simulation noise in the estimated operating characteristic. It is not the standard error of a treatment-effect estimate within a trial, and it does not capture uncertainty about whether the assumed data-generating scenario is realistic.

Worked Example: Two Null Endpoints With Correlated Decisions

Consider a hypothetical trial with two endpoints, both truly having no treatment effect. The planned Bayesian rule is applied to both endpoints in each of \(M=2{,}000\) independently simulated trials. Suppose 54 trials produce a false-positive claim on endpoint 1, 61 on endpoint 2, and 24 produce false-positive claims on both.

A trial is counted once if either endpoint generates a false-positive claim. By inclusion-exclusion, the number of trials with at least one such claim is \(54+61-24=91\). Thus:

\widehat{q}=\frac{91}{2{,}000}=0.0455.

The estimated probability of at least one false-positive conclusion is 4.55% under this hypothetical scenario. Its Monte Carlo standard error is:

\widehat{SE}_{MC} =\sqrt{\frac{0.0455(1-0.0455)}{2{,}000}} =\sqrt{0.00002172} \approx 0.00466.

A rough 95% Monte Carlo interval is \(0.0455\pm1.96(0.00466)\), or approximately 0.0364 to 0.0546. This interval describes simulation precision, not posterior uncertainty about the endpoints. Simply adding the two marginal rates would give \((54+61)/2{,}000=0.0575\), which double-counts the 24 trials in which both claims were false positives.

Partial-null scenarios and the family of claims

A trial can have both true and null effects. In that case, the relevant event is at least one false-positive conclusion among the null claims, not merely at least one positive conclusion of any kind. A true-positive decision on an endpoint with a genuine benefit is not a false positive. This distinction matters when evaluating several treatments or endpoints under a mixture of effects.

The false-positive probability also depends on which claims count as one family. For example, a plan might define a family as all primary endpoints, or all treatment-versus-control comparisons. State that family in advance and include precisely those claims in the event indicator. A result for a global-null scenario, in which every effect is null, does not automatically describe a partial-null scenario.

Worked Example: Two Null Comparisons and One Truly Beneficial Comparison

Suppose a hypothetical trial compares three treatments with a shared control. Treatments A and C have true effects \(\delta_A=\delta_C=0\); treatment B has a true beneficial effect, \(\delta_B>0\). For each comparison, the planned rule declares benefit if \(P(\delta_j>0\mid\text{data})>0.95\). In one simulated replicate, the posterior probabilities are 0.963 for A, 0.988 for B, and 0.941 for C.

The rule declares benefit for A and B, but not C. A is a false-positive conclusion because its true effect is zero. B is a true-positive conclusion in this scenario, so it is not counted as a false positive. The replicate’s family-wise false-positive indicator is therefore \(A_m=1\), regardless of the correct conclusion for B.

Now suppose 1,500 hypothetical replicates yield 36 false-positive decisions for A, 42 for C, and 15 replicates with false positives for both. The number with at least one false positive among the null comparisons is \(36+42-15=63\), so:

\widehat{q}=\frac{63}{1{,}500}=0.0420, \qquad \widehat{SE}_{MC} =\sqrt{\frac{0.0420(0.958)}{1{,}500}} \approx 0.00518.

The estimated probability is 4.20% under this particular partial-null scenario. Counting all trials in which any treatment was declared beneficial would answer a different question, because it would include correct decisions for treatment B.

Justifying the joint-event calculation

For two null claims, let \(F_1\) and \(F_2\) denote their false-positive indicators. The event of at least one false positive is the union of the two events. Adding their probabilities counts trials in the intersection twice, so subtract that overlap once:

P(F_1=1\ \text{or}\ F_2=1) =P(F_1=1)+P(F_2=1)-P(F_1=1,F_2=1).

For more claims, the same principle extends through inclusion-exclusion, but directly recording the trial-level union indicator is usually simpler and less error-prone. The independence shortcut \(1-\prod_j(1-P(F_j=1))\) applies only if the false-positive indicators are independent. Shared participants, endpoints, controls, or fitted parameters commonly make that assumption inappropriate. Joint simulation avoids needing to assume independence.

A posterior threshold and a simulated family-wise error probability answer different questions. The threshold \(\gamma\) describes how strong the posterior evidence must be within a trial before the rule makes a claim. The simulated probability \(q\) describes how often the complete rule makes at least one false-positive claim across repeated trials under a specified truth. Neither quantity can generally be substituted for the other.

Diagnostics, alternatives, and limitations

Before trusting a result, check that each simulated replicate applies the same analysis and decision rule planned for the trial. Confirm that the truth vector correctly identifies null claims, that the family includes the intended endpoints and comparisons, and that the joint data generator reflects shared participants and controls. Compare marginal false-positive rates with the family-wise estimate: the latter should be at least as large as each marginal rate among null claims. Also check that increasing \(M\) reduces Monte Carlo error, while recognizing that more replicates cannot repair a misspecified scenario.

A common mistake is to reason that a posterior threshold of 0.95 means there is a 5% chance of any false positive across the endpoints. It does not. The threshold is a within-trial posterior criterion, while the chance of at least one false-positive conclusion is a repeated-trial probability that depends on the data-generating model, the number and dependence of claims, and the full decision rule. Estimate that probability by simulating and counting the joint event.

Simulation is one approach to multiplicity assessment. Other approaches include adjusting per-claim thresholds or defining a joint posterior decision rule that considers several effects together. These approaches can change the decision rule, whereas simulation evaluates the rule that is supplied to it. If a threshold is selected by repeatedly checking the same simulations until the estimated error rate meets a target, the reported estimate may be optimistic; assess a selected rule on fresh simulations. Simulation estimates are also only as credible as their assumptions, and a low estimate in one scenario does not guarantee a low rate under different nuisance parameters or effect configurations.

Worked Example: Comparing Two Prespecified Posterior Thresholds

In a hypothetical simulation of 1,000 trials with two null endpoints, the same generated datasets are analyzed using two alternative rules: declare benefit when the endpoint’s posterior probability of benefit exceeds 0.95, or use a stricter threshold of 0.975. At 0.95, endpoint 1 has 50 false-positive decisions, endpoint 2 has 39, and 21 trials have both. At 0.975, the corresponding counts are 31, 26, and 14.

For the 0.95 rule, the number of trials with at least one false positive is \(50+39-21=68\), giving \(\widehat{q}=68/1{,}000=0.0680\). For the 0.975 rule, it is \(31+26-14=43\), giving \(\widehat{q}=43/1{,}000=0.0430\). These results show the estimated family-wise false-positive probability for each rule under this scenario; they do not establish that either rule is suitable for every trial or scenario.

Because the same data are used, a stricter threshold cannot add a positive decision for an endpoint that failed the 0.95 rule. The nested counts are therefore consistent with the rules. The comparison is informative only if the thresholds and analysis are applied consistently and the simulation truth and dependence structure match the intended evaluation.

Key takeaway: Define false-positive claims from the trial’s truths and planned decisions, preserve dependence by simulating complete trials, and estimate the probability of at least one false positive by counting the trial-level union event. Report Monte Carlo precision separately from uncertainty about the scenario assumptions.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

For two null endpoints, 30 simulated trials have a false positive on endpoint 1, 25 on endpoint 2, and 10 on both. Out of 1,000 trials, what is the estimated probability of at least one false positive?
A trial has one null comparison and one truly beneficial comparison. Only the null comparison is declared beneficial. Is the trial counted as a family-wise false-positive event?
Why can the formula based on independent endpoint decisions be inappropriate when endpoints share participants?
What does the Monte Carlo standard error quantify, and what does it not quantify?

Solutions. (1) The union count is \(30+25-10=45\), so the estimate is \(45/1{,}000=0.0450\), or 4.50%. (2) Yes. The declared benefit on the null comparison is false, even though the other comparison has a genuine benefit. (3) Shared participants can induce dependence between endpoint decisions, so multiplying probabilities under an independence assumption may give the wrong joint-event probability; simulate the endpoints together. (4) It quantifies simulation noise in the estimated operating characteristic. It does not measure uncertainty about the treatment effect within a trial or the adequacy of the assumed data-generating model.