Why Regulatory Deficiencies Matter
A Bayesian analysis can be mathematically correct and still be difficult to evaluate if the submission does not explain why its assumptions are appropriate, how the design behaves under plausible conditions, or who is authorized to act on its results. A regulatory deficiency is not simply a choice that differs from a reviewer’s preference. It is a material weakness in the evidence, specification, or documentation that prevents a reviewer from understanding or assessing the planned analysis and its consequences.
The focus here is on four recurring weaknesses: vague prior justification, inadequate design simulations, incomplete sensitivity analyses, and unclear decision governance. These areas are connected. For example, a prior may affect the probability of crossing an adaptive decision threshold; the simulation plan should therefore assess that effect, and the governance plan should say who may act if the threshold is crossed. A submission should make these links traceable without requiring the reviewer to infer them from scattered documents.
The concerns described below are review risks, not automatic grounds for rejection. Their importance depends on the trial’s purpose, design, data, and proposed use of the results. A focused, well-supported analysis can be more informative than a large collection of disconnected tables.
Prior Justification and Sensitivity
A prior distribution is part of the model, so a submission should identify its source, form, parameters, and clinical or empirical interpretation. Saying that a prior is “weakly informative,” “conservative,” or “noninformative” is not a sufficient justification by itself. The submission should explain what values the prior makes plausible, whether information is borrowed from external evidence, and why the prior is suitable for the population and endpoint in the current trial.
The justification should also make clear whether the prior was specified before examining the current trial data. If it was informed by external data, the submission should describe the data selection and any adjustments for differences in population, endpoint definition, treatment, or conduct. If the prior is intended to reduce the impact of implausibly large effects, that intention should be expressed in the parameter’s clinical units and checked against plausible effects.
A sensitivity analysis asks whether conclusions relevant to the trial’s intended decision change under reasonable alternative assumptions. It is not enough to display a few alternatives without explaining how they were selected or what changes in the results mean. The analysis should identify the primary model, alternatives, decision-relevant summaries, and any material changes in conclusions.
Worked Example: Showing How Prior Choice Affects a Decision
Suppose a hypothetical trial estimates a treatment difference in a continuous clinical measure as \(y=3.0\) mm, with known standard error \(SE=1.0\) mm. For illustration, assume the sampling model \(y\mid\theta\sim N(\theta,1^2)\), where \(\theta\) is the true treatment difference in millimetres. Compare a primary prior \(\theta\sim N(0,2^2)\) with a more concentrated alternative \(\theta\sim N(0,1^2)\). Assume a hypothetical decision criterion that requires \(P(\theta>0\mid y)>0.99\).
For a normal likelihood with known variance and a normal prior, the posterior variance is the reciprocal of the sum of prior and likelihood precisions, and the posterior mean is the precision-weighted average of the prior mean and the observed estimate. Here the likelihood variance is 1.
With the primary prior, \(\mu_0=0\) mm and \(\tau=2\) mm. Therefore,
The posterior standard deviation is \(\sqrt{0.8}=0.8944\) mm, so \(P(\theta>0\mid y)=\Phi(2.4/0.8944)=\Phi(2.683)\approx0.9964\). Under the alternative prior, \(\mu_0=0\) mm and \(\tau=1\) mm:
The primary analysis exceeds the hypothetical 0.99 criterion; the alternative does not. This does not establish that either prior is appropriate. It shows why a submission needs a substantive prior rationale and should disclose whether a plausible alternative changes a decision-relevant conclusion. The numerical result is conditional on the stated model and hypothetical data, not evidence about an actual treatment.
Simulation Evidence and Its Limits
When a trial has adaptive features, complex endpoints, or posterior-probability decision rules, operating characteristics are often assessed by simulation. A submission should describe the simulated data-generating scenarios, the trial algorithm, the number of repetitions, the performance measures, and the implementation checks. Simulations should cover scenarios that matter to the intended use: for example, no treatment effect, effects near a decision boundary, clinically important effects, plausible nuisance-parameter values, and assumptions that may be wrong.
A common deficiency is to report one favorable scenario, or a single estimated success rate, without enough information to tell whether the design has acceptable behavior in less favorable but plausible cases. Another is to omit the exact rule used in the simulation, making it impossible to confirm that the simulated design matches the protocol. For an adaptive design, the simulation must reproduce the actual timing, data available at each decision, enrollment limits, stopping rules, and any adaptations—not a simplified design that happens to be easier to simulate.
Monte Carlo uncertainty also matters. If \(R\) independent simulated trials are run and an event occurs in \(K\) of them, the estimated probability is \(\hat{q}=K/R\). Its approximate Monte Carlo standard error is \(\sqrt{\hat{q}(1-\hat{q})/R}\). This quantifies simulation noise, not uncertainty from enrolling real patients. A small number of repetitions can make an apparently precise operating-characteristic estimate unstable.
Worked Example: Diagnosing a Thin Simulation Report
A hypothetical submission reports that a design produced 580 false-positive efficacy decisions in 10,000 simulated trials under the null scenario. The estimated false-positive decision rate is
The approximate Monte Carlo standard error is
or about 0.234 percentage points. A rough two-standard-error range for Monte Carlo error alone is \(5.80\%\pm0.47\) percentage points. This range is not a confidence interval for a clinical effect; it indicates the simulation estimate’s numerical variability.
If the design’s prespecified target were a false-positive decision rate no greater than 5%, the reported estimate would be above target, and the Monte Carlo variability would not by itself explain the full difference. The submission would need to examine the simulated design and assumptions rather than describe the estimate as “approximately 5%” without qualification. The report is also incomplete if it gives no scenarios for effects near the decision boundary, no account of missing data or model misspecification, and no confirmation that the simulated algorithm matches the protocol.
For perspective, if the goal were a Monte Carlo standard error near 0.001 when the event rate is around 0.05, a rough repetition count follows from \(SE_{\text{MC}}^2\approx q(1-q)/R\):
This is a planning calculation, not a universal minimum. More repetitions reduce simulation noise but do not repair an unrepresentative set of scenarios or an incorrectly implemented design.
Completeness of Sensitivity Analyses
Sensitivity analysis should address assumptions capable of changing the interpretation or intended decision, not merely offer multiple versions of the same model. Depending on the trial, relevant dimensions may include prior distributions, likelihood or link-function choices, treatment-effect heterogeneity, missing-data assumptions, and the treatment of external information. The submission should explain why each alternative is plausible and how it relates to the primary analysis.
Missing-data assumptions deserve particular attention because different assumptions can imply different evidence about the outcome for participants whose data are not observed. A complete package should specify the primary assumption, justify it in context, and examine reasonable alternatives where the conclusion could depend on it. A sensitivity analysis cannot demonstrate that an untestable assumption is true. It can show how dependent the result is on that assumption.
A practical diagnostic is to ask: if this assumption changed within a clinically plausible range, could a key conclusion or action change? If yes, the assumption should be visible, justified, and evaluated. If no, the submission can explain why it is unlikely to affect the decision. A long list of unmotivated models can obscure rather than strengthen the evidence.
Sensitivity analysis is also distinct from choosing whichever model gives the most favorable result. The primary analysis should remain identifiable, and alternatives should be reported even when they weaken the conclusion. If an alternative reveals a material change, the submission should explain the source of the change and its implications rather than dismiss it as “only a sensitivity analysis.”
Decision Governance and Traceability
A decision rule is not a complete governance plan. The rule states what quantitative condition triggers an action; governance states who receives the relevant information, who is authorized to make or recommend the decision, when the decision is made, what information is available, and how the action is documented. For interim decisions, the plan should also specify how access to unblinded information is controlled and how the integrity of ongoing trial conduct is protected.
Earlier material on prespecifying adaptive rules, interim governance, independent monitoring, and audit trails establishes why the operational details matter. A submission may still be deficient if its documents use different thresholds or describe different decision-makers, if responsibility for final action is ambiguous, or if a change to the rule has no defined control process. The protocol, statistical analysis plan, monitoring charter, and submission narrative should be mutually consistent.
Worked Example: Turning an Ambiguous Rule into a Governed Decision
A hypothetical protocol says, “The study may stop early if the posterior probability of benefit is compelling and safety is acceptable.” This language does not specify the endpoint, the benefit threshold, the safety definition, the data cutoff, or who decides. Two teams could apply it differently to the same data.
Suppose the protocol instead defines \(\delta\) as the treatment-minus-control difference in response probabilities and \(\gamma\) as the treatment-minus-control difference in serious adverse-event probabilities. For illustration, it specifies an interim efficacy criterion \(P(\delta>0\mid D)\geq0.975\), together with a safety gate \(P(\gamma>0.05\mid D)<0.10\), using the prespecified interim data \(D\). The joint action rule is to recommend stopping for efficacy only when both conditions hold.
At a hypothetical interim review, assume the analysis gives \(P(\delta>0\mid D)=0.982\) and \(P(\gamma>0.05\mid D)=0.12\). The efficacy condition is met because \(0.982\geq0.975\), but the safety gate is not met because \(0.12\) is not less than \(0.10\). Thus, the joint rule does not trigger an efficacy stop. The decision follows from the stated inequalities; “benefit is compelling” would not resolve the case.
A complete governance description would additionally identify the data cutoff, the independent body reviewing the unblinded analysis, the person or body with authority to recommend or enact the action, and the documentation and communication process. Those elements do not change the arithmetic, but they determine whether the rule can be applied consistently and audited.
Reviewing and Remedying Deficiencies
A useful review begins by linking each claim to its supporting evidence. The prior rationale should link to its sources and parameter scale; the design claims should link to scenarios and simulation code or reproducible outputs; the sensitivity conclusions should link to specified alternatives; and decision claims should link to a prespecified rule and governance process. The purpose is not to maximize document volume, but to make important assumptions and consequences inspectable.
| Weakness | Diagnostic question | Useful response |
|---|---|---|
| Vague prior justification | What values does the prior imply, and why are they plausible here? | Give the source, scale, rationale, and relevant prior sensitivity results. |
| Inadequate simulations | Do scenarios cover the null, decision boundaries, plausible effects, and failure conditions? | Describe the algorithm, scenarios, repetitions, performance measures, and Monte Carlo uncertainty. |
| Incomplete sensitivity analysis | Could a plausible alternative assumption change a material conclusion? | Identify the primary assumptions, justify alternatives, and explain any consequential changes. |
| Unclear governance | Who acts, on what information, under which rule, and how is the action recorded? | Align the rule and responsibilities across governing documents and specify the decision process. |
The main alternative to a Bayesian analysis is not a single universal method; the appropriate comparison depends on the trial question and design. For regulatory review, the useful comparison is often between the proposed analysis and a prespecified alternative analysis or design strategy. Such comparisons can reveal whether the conclusion depends on borrowing, modeling choices, or an adaptive rule. They should supplement—not replace—a clear justification of the primary approach. A conventional analysis presented only as a reassurance, without explaining its role or assumptions, does not resolve a weakness in the Bayesian submission.
These checks have limits. No finite set of simulations covers every possible data-generating process, and no sensitivity analysis proves that the primary assumptions are true. A well-documented submission instead identifies the assumptions that matter, evaluates plausible departures, quantifies relevant uncertainty, and states how decisions will be governed. The goal is transparent, reviewable evidence—not a claim that uncertainty has disappeared.
Common Pitfalls
- Calling a prior “objective” or “weak” without showing its implications. Labels do not establish suitability; explain the prior on the clinical parameter scale and assess relevant alternatives.
- Treating many simulation repetitions as proof of a sound design. Repetition count controls Monte Carlo noise, not scenario relevance or implementation correctness.
- Reporting sensitivity results selectively. Alternatives that weaken or change an intended conclusion are still informative and should be explained.
- Confusing a numerical threshold with governance. A threshold does not identify who reviews the result, who acts, or how an action is documented.
- Calling an analysis “robust” because one alternative agrees. Robustness is always relative to the assumptions and alternatives actually examined.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A submission calls its prior “weakly informative” but gives no source, parameter-scale interpretation, or sensitivity analysis. What is the deficiency, and what would help remedy it?
- In 5,000 simulated trials, an event occurred 250 times. Calculate the estimated event rate and approximate Monte Carlo standard error.
- Why does increasing the number of simulation repetitions not fix a design simulation that omits plausible scenarios?
- A prespecified rule requires both an efficacy condition and a safety condition. The efficacy condition is met but the safety condition is not. What action follows?
Solutions. (1) The prior’s implications and suitability cannot be assessed. The submission should state its source and scale, explain its relevance to the trial, and examine reasonable alternatives. (2) The estimate is \(250/5{,}000=0.05\), or 5.00%. The approximate Monte Carlo standard error is \(\sqrt{0.05(0.95)/5{,}000}\approx0.003082\), or 0.308 percentage points. (3) More repetitions reduce Monte Carlo variability for the scenarios included, but cannot show how the design behaves in scenarios that were never simulated. (4) The joint rule is not met, so the action it specifies is not triggered; governance determines who reviews and records that result.