Addressing Multiplicity in Bayesian Trial Submissions

Why Multiplicity Is a Submission-Level Issue

A trial may examine several endpoints, treatment groups, or subgroups, then support one or more regulatory claims. The multiplicity question is not simply how many analyses appear in a report. It is how many opportunities the decision strategy creates to make a claim, and how the conclusions are meant to be interpreted together. A submission that tests three endpoints but requires success on all three makes a different claim from one that can support approval if any one endpoint succeeds.

Bayesian analysis does not automatically remove this issue. Posterior probabilities describe uncertainty conditional on the model, prior, and observed data; they are not, by themselves, guarantees about the long-run rate of false regulatory conclusions. A submission therefore needs to state the intended claim family, the decision rule applied to that family, and the scope of the conclusions supported by the rule. This extends the earlier material on Bayesian decision thresholds: the threshold is only interpretable once the decision it governs is clear.

A useful plan identifies the claim family before results are known. It states which endpoint defines each claim, whether a claim concerns a particular treatment contrast or population, whether claims are co-primary or ordered, and whether success on one analysis can support a conclusion on its own. This prevents an important ambiguity: the same collection of posterior summaries can support different conclusions under different decision strategies.

Marginal and Joint Posterior Evidence

Suppose the trial concerns effects \(\theta_1,\ldots,\theta_K\), where \(\theta_j>0\) represents benefit on claim \(j\). A marginal posterior probability, such as \(P(\theta_j>0\mid \text{data})\), addresses that claim separately. A joint probability, such as \(P(\theta_1>0,\ldots,\theta_K>0\mid \text{data})\), addresses the event that all claims are beneficial. The two quantities answer different questions.

If a decision requires success on every endpoint, the joint event is often directly relevant. If a strategy permits a claim whenever any endpoint succeeds, the relevant event and the consequences of selecting that endpoint must be described instead. Dependence among endpoints matters: in general, marginal posterior probabilities alone do not determine a joint posterior probability. The joint model, including its dependence assumptions, must be specified.

That calculation depends on the stated posterior independence assumption. With dependent endpoints, multiplying the marginal probabilities would generally be incorrect. The submission should calculate the joint probability from the fitted joint posterior or use an explicitly justified bound; it should not infer joint evidence from the marginal summaries alone.

Strategies for Defining and Evaluating the Family

Several decision structures are common. They are not interchangeable, and none should be selected merely because it gives the most favorable result. The strategy should follow the clinical and regulatory question.

• Conjunctive rule: require all specified endpoints or conditions to meet their criteria. This supports a claim that depends on success across the full set. It is usually more demanding than a rule that permits success on any one endpoint.
• Hierarchical rule: evaluate claims in a prespecified order, with access to a later claim depending on the result for an earlier one. The order and transition rule define which claims can be made.
• Separate treatment-group claims: specify whether each group is being evaluated as a distinct opportunity for a claim, or whether the strategy requires a particular pattern of results across groups.
• Selection-based rule: choose a treatment, endpoint, or subgroup after examining the results. Because the selection is data-dependent, the submission must account for the selection process when explaining the evidence and the scope of any claim.

A strategy may also summarize evidence using the posterior probability that all claims in a set are valid, the posterior expected number of false claims, or another decision quantity. These quantities answer different questions. For example, a low expected number of false claims does not necessarily mean that the probability of at least one false claim is low. Any quantity used to justify a decision should be defined, calculated under the joint model, and tied explicitly to the intended claim.

Connecting Bayesian Decisions to Operating Characteristics

A Bayesian posterior threshold and a frequentist error-rate target are distinct. A rule such as “declare benefit when the posterior probability exceeds 0.95” does not, without further analysis, guarantee a 5% long-run probability of making one or more false claims when all treatments are ineffective. If a submission also aims to satisfy a frequentist operating-characteristic target, it should evaluate the complete decision rule under scenarios that represent the relevant null or unfavorable conditions.

One simple justification uses the union bound. Let \(A_j\) be the event of a false claim on endpoint \(j\). For \(K\) endpoints, \(P(\text{at least one false claim})=P(\cup_{j=1}^{K}A_j)\leq\sum_{j=1}^{K}P(A_j)\). This inequality does not require independence. It provides a conservative way to bound the family-wise probability of one or more false claims.

In Bayesian trials, the individual probabilities in this calculation are typically assessed by evaluating the decision rule under specified data-generating scenarios, rather than by assuming that a posterior cutoff itself supplies the bound. Simulation can incorporate the trial design, endpoint dependence, interim decisions if relevant, and the rule for selecting among claims. The simulation plan should state the scenarios, the number of replicates, the definition of a false claim, and Monte Carlo uncertainty. Estimated rates are evidence about performance under the simulated assumptions; they are not exact guarantees.

Justifying the Strategy and Its Scope

A transparent submission makes the chain from clinical question to conclusion visible. A useful sequence is:

Diagnostics should look for discrepancies between the stated and implemented family, such as endpoints added to the claim set after results are reviewed, unreported treatment comparisons, or subgroup findings presented as confirmatory despite being exploratory. Review the joint posterior assumptions and endpoint correlations, and check that simulation code represents the complete decision rule rather than isolated endpoint analyses. If the estimated operating characteristics are unstable, increase simulation precision or describe the uncertainty instead of presenting the estimate as exact.

The main alternative to explicit multiplicity handling is to present each analysis as a separate Bayesian assessment and argue that no family-level adjustment is needed. That can be defensible when the claims truly are separate decisions and the submission clearly limits their interpretation. It is not defensible to use the separate-analysis framing while making a broad claim based on whichever favorable result appears. Conversely, a conservative family-wise calibration may reduce the chance of a false claim but can also make genuine effects less likely to pass the decision rule. The trade-off should be evaluated against the clinical purpose and the consequences of each kind of decision.

Multiplicity procedures have limitations. A joint posterior probability depends on the joint model and prior; a frequentist operating-characteristic assessment depends on the simulated scenarios; and a hierarchical rule supports only the claims its ordering permits. No numerical adjustment can repair an unclear regulatory question or justify claims beyond the populations and outcomes studied. For subgroup claims in particular, the submission should make clear whether each subgroup is part of the prespecified claim family and whether the evidence supports a subgroup-specific conclusion; detailed assessment of subgroup claims requires its own careful analysis.

Common Mistakes

• “Bayesian analysis needs no multiplicity discussion.” Posterior probabilities do not automatically control the long-run rate of false claims. State the claim family and the decision rule, and evaluate any desired operating-characteristic targets.
• “Every endpoint passed 0.95, so the probability all endpoints are beneficial is at least 0.95.” Marginal probabilities do not establish that joint statement. Calculate the joint posterior probability under the joint model.
• “A small expected number of false claims means a small chance of any false claim.” These are different summaries. Report the quantity that corresponds to the decision concern.
• “The best-looking result can be selected without changing the claim.” Selection based on observed results changes the procedure. Include selection in the model-based evaluation and describe the resulting scope honestly.
• “A rounded per-endpoint bound is close enough.” A rounded number that exceeds the intended bound can break a guarantee. Preserve the exact bound or round conservatively downward.