Decision Robustness to Likelihood Misspecification

Why Examine the Likelihood?

A Bayesian arm decision depends on the observed data, the prior distribution, the likelihood, and the utility assigned to each possible action. A recommendation can therefore be sensitive to the assumed likelihood even when the data, prior, and decision rule are unchanged. This matters in a multi-arm, multi-stage trial: a model that treats observations as independent when they are clustered, or that gives extreme observations too much influence, can change which arm appears preferable at an interim analysis.

Here, likelihood misspecification means that the probability model used for the observed outcomes does not adequately describe their sampling distribution conditional on the model parameters. Decision robustness is not a claim that one likelihood is certainly correct. It is an assessment of whether the arm decision remains useful and stable across a prespecified set of scientifically plausible likelihoods.

The comparison should be controlled. If the prior or utility changes at the same time as the likelihood, a changed recommendation cannot be attributed to likelihood sensitivity alone. Begin with a primary likelihood and specify alternatives that address a concrete concern, such as heavy tails, excess binary-response variation, or clustering. Fit each model to the same data and apply the same decision rule, such as selecting the action with greatest posterior expected utility.

Posterior Decisions Across Likelihoods

Let \(a\) denote an available action, such as selecting an experimental arm or retaining control, and let \(U(a,\theta)\) be its utility at the unknown state \(\theta\). Under likelihood model \(M_k\), the posterior expected utility is

where \(y\) is the observed data and \(\pi_k(\theta\mid y)\) is the posterior under model \(M_k\). The decision under that model is an action maximizing \(\operatorname{EU}_k(a)\). A convenient first summary is whether the maximizing action changes. Also report utility differences: a winner change between two nearly tied actions is less consequential than a large loss in expected utility.

For two hypotheses or states, posterior odds show directly why a likelihood change can matter. If the prior odds are fixed, then

The likelihood ratio can change sharply when a model assigns different relative probabilities to unusual observations. In a multi-arm decision, the same principle applies through the full posterior distribution and the utility integral: changes to the likelihood can alter posterior location, uncertainty, or both.

The comparison assumes that the candidate models describe the same outcome and estimand, that their likelihoods are well-defined for the observed data, and that prior and utility choices are held fixed. If an alternative model changes the estimand—for example, from an individual-level response to a cluster-level response—then the decision comparison needs an explicit mapping to a common utility scale.

Binary Overdispersion and Decision Sensitivity

For binary outcomes, the ordinary binomial likelihood assumes conditionally independent responses with a common event probability within each arm. Clustering, repeated measurements, or unmodeled heterogeneity can make outcomes more variable than that likelihood allows. A beta-binomial model is one possible likelihood alternative when cluster-level variation is scientifically plausible. A design-effect adjustment can also serve as a transparent approximate sensitivity calculation, but it is not a substitute for fitting a coherent overdispersed likelihood when one is required.

For clusters of size \(m\) with within-cluster correlation \(\rho\), the usual design effect is \(D=1+(m-1)\rho\). It approximates the variance inflation relative to independent observations. The effective sample size approximation is \(n_{\mathrm{eff}}=n/D\). It is useful for illustrating sensitivity, provided its approximation and assumptions are stated rather than treating fractional effective counts as actual observed data.

How to Assess and Report Robustness

A useful analysis specifies the model set before interpreting the results. Alternatives should correspond to plausible failure modes, not be selected merely because they produce a preferred arm. Keep the outcome definition, prior, utility, and action set fixed where possible. For each model, record the posterior expected utility of every action, the winning action, and the difference between the best and next-best action. For a small model set, the range of each action’s expected utility and its worst-case regret are informative summaries. Regret under model \(M_k\) is the expected-utility loss from choosing \(a\) rather than the action that would maximize utility under that model.

Diagnostics help decide which alternatives deserve consideration. For continuous outcomes, inspect residual patterns and whether a few extreme values dominate arm contrasts; compare predictive distributions for tail behavior. For binary data, examine variability across clusters or sites and compare observed variation with that predicted under conditional independence. Posterior predictive checks can reveal that a likelihood fails to reproduce features of the observed data, but agreement with such checks does not prove the model is correct.

In a staged trial, the data available at an interim analysis may be limited, and selected arms can have unusually favorable observed results by chance. Repeating the same decision calculation under plausible likelihoods makes this dependence visible, but it does not remove selection effects or guarantee that an interim recommendation will remain appropriate as more data accrue. The analysis also cannot protect against every form of misspecification: the model set may omit the relevant alternative, and a robust decision across a narrow set can still be fragile outside it.

Common Pitfalls and Alternatives

A common mistake is to call a decision robust because the estimated treatment effect is similar across models. The posterior uncertainty may have changed enough to alter the probability of benefit or the expected utility, even if posterior means are close. Conversely, a winner can change when two actions have nearly equal utility, while the practical cost of switching is negligible. Always pair the identity of the selected action with the size of the utility difference.

Another mistake is to change the prior, likelihood, and utility together, then attribute the resulting decision change to the likelihood. Separate sensitivity analyses clarify which assumption matters. Model averaging is an alternative when probabilities over candidate models can be justified: it combines model-specific posterior distributions using model probabilities. It answers a different question from checking whether recommendations agree model by model. Robust likelihood procedures, such as heavy-tailed errors, can reduce sensitivity to specific data features, but they still rely on assumptions and do not replace checking whether the resulting model fits the scientific setting.

A likelihood sensitivity analysis is not a formal guarantee of good operating characteristics. In a MAMS design, trial-level performance across possible data-generating scenarios requires design evaluation, which is distinct from examining the posterior decision for one observed data set. Report the candidate models, the reason each is plausible, and the decision consequences; avoid presenting one preferred model as validated solely because it yields a stable answer.