From Question to Response
An agency question about a Bayesian trial may appear to ask for a number, a model description, or a justification. Often, however, the central concern is broader: whether the analysis answers the intended clinical question, whether its assumptions are credible, or whether the trial’s decisions can be understood and reproduced. A useful response addresses that underlying concern directly rather than supplying a technically correct but incomplete answer.
A response should make it possible for a reviewer to see what was asked, what evidence was considered, what the evidence supports, and whether the sponsor proposes any change. This requires separating a factual clarification from a new analysis and from a change to the design or analysis plan. The distinction matters: a new sensitivity analysis may help characterize uncertainty without changing the primary analysis, while changing a decision rule can alter the interpretation of the trial.
A practical sequence is to restate the question in operational terms, identify the relevant claim or decision, provide the evidence needed to assess it, and then state the response and any action. For example, “Please clarify the borrowing approach” may concern whether the external data are sufficiently comparable to the current trial, whether borrowing is robust to disagreement, or both. Those concerns require different evidence.
Translate the request into a specific issue, such as model credibility, prior influence, decision-rule performance, or operational implementation.
State which estimand, population, endpoint, analysis, or trial decision the concern affects. Avoid answering a broader question than the agency asked.
Show the relevant model specification, data summary, sensitivity analysis, simulation, or documentation. State assumptions and limitations alongside the result.
Say what the evidence supports, what remains uncertain, and whether the protocol, statistical analysis plan, or operational procedures will change.
Match Evidence to the Concern
Evidence is useful only if it bears on the question. A posterior summary does not by itself establish that a prior is appropriate; a software validation report does not establish that the model is clinically relevant. First identify the claim that needs support, then select the evidence that can evaluate that claim. The response should state the assumptions under which that evidence is informative.
For instance, a question about borrowing from external information may require a description of the source population, endpoint definitions, follow-up, and relevant differences from the current trial. A question about a decision threshold may require the clinical rationale for the threshold and operating characteristics under specified scenarios. A question about implementation may require a clear account of who has access to data, what triggers an action, and how decisions are recorded.
Worked Example: Explaining External Borrowing
Suppose an agency asks why an analysis borrows information about a binary control outcome from an earlier, hypothetical study. The earlier study had 20 responses among 100 participants. The current trial has 24 responses among 120 control participants. The response should address comparability and the effect of borrowing, not treat similar observed rates as proof that the studies are exchangeable.
For a compact illustration, suppose the external data are represented by a Beta prior with one added success and one added failure, giving \(\operatorname{Beta}(21,81)\). Under a binomial likelihood for the current control data, the posterior is obtained by adding current successes and failures to the prior parameters:
The posterior mean under this borrowing model is \(45/(45+177)=45/222=0.2027\), or about 20.27%. As a simple no-borrowing comparison, use a \(\operatorname{Beta}(0.5,0.5)\) prior with the current data alone. The resulting posterior is \(\operatorname{Beta}(24.5,96.5)\), with mean \(24.5/121=0.2025\), or about 20.25%.
These close posterior means are descriptive of this example; they do not establish that borrowing is safe in general. A complete response would describe how the external population and outcome definitions compare with the current trial, state the borrowing model and its assumptions, and show a prespecified sensitivity analysis that reduces or removes borrowing. It would then explain whether the primary conclusion or decision changes across the relevant analyses. If the model is being proposed only as an additional analysis, say so; do not imply that it replaces the primary analysis.
The numerical example also illustrates why a response should distinguish observed agreement from an assumption. Similar rates in two studies are compatible with exchangeability, but do not prove it. Differences in eligibility, calendar time, clinical practice, or outcome ascertainment may still affect the relevance of historical information. The response should make those potential differences assessable rather than dismissing them.
Explain Decision Rules With Their Evidence
When an agency asks why a posterior-probability threshold or other Bayesian decision criterion was selected, a response should connect the rule to its clinical purpose and show how it behaves under relevant scenarios. The threshold is not self-justifying: its meaning depends on the event being evaluated, the population, the model, and the consequences of acting. Earlier material on defending Bayesian decision thresholds and prespecifying adaptive rules is relevant; here the emphasis is on presenting that rationale in a focused response.
If the response includes simulation results, state the simulated scenarios, the decision rule, the number of replicates, and the performance measure. Monte Carlo uncertainty also matters when estimates are based on a finite number of simulated trials. If \(K\) of \(M\) simulated trials meet a criterion, the estimated proportion is \(\hat{p}=K/M\). Under independent simulation replicates, its Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/M}\). This quantifies simulation error, not uncertainty about the clinical treatment effect.
Worked Example: Responding to a Threshold Question
Suppose the agency asks how often a proposed success rule would declare success in a specified scenario. In a hypothetical simulation of 20,000 trials under a scenario chosen to represent a clinically meaningful benefit, 18,400 trials meet the prespecified posterior-probability criterion. The estimated probability of meeting the criterion is:
The Monte Carlo standard error is:
A focused response could report that the estimated success probability in this specified scenario is 92.0%, with Monte Carlo standard error about 0.192 percentage points. It should also identify the assumed sample size, outcome model, prior, and criterion. This result addresses performance in that scenario; it does not show that the rule has the same performance under other treatment effects, event rates, or model departures. The clinical rationale for selecting the rule must be provided separately from the simulation estimate.
Clarifications, Analyses, and Changes
An agency response should explicitly say whether it is clarifying the existing plan, adding an analysis, or changing the study. These are not interchangeable. A clarification explains what a prespecified method means. An additional analysis may assess robustness while leaving the primary analysis intact. A change to an endpoint, prior, decision threshold, adaptation trigger, or data-access procedure may affect trial conduct or interpretation and should be identified as a change, not described as mere clarification.
For any proposed change, describe what will change, why it is needed, when it will take effect, who will approve or implement it, and how the original plan and resulting analyses will remain traceable. If data have already been observed or reviewed, state that timing clearly and explain whether the change was made before or after access to relevant information. Do not present a post hoc analysis as if it had been prespecified. The earlier material on audit trails, protocol amendments, and regulatory governance of interim decisions provides the relevant operational context.
Worked Example: Reporting Operating Characteristics
Suppose an agency asks how often an adaptive rule would recommend proceeding when there is no treatment benefit. In 5,000 hypothetical simulated trials under a specified no-benefit scenario, 213 meet the rule’s proceed criterion. The estimated rate in that scenario is:
The Monte Carlo standard error is:
The response can therefore report an estimated proceed rate of 4.26% in this particular no-benefit scenario, with Monte Carlo standard error about 0.286 percentage points. It must also describe the scenario assumptions and the exact proceed rule. This is a scenario-specific operating characteristic; it should not be called a universal error rate or a guarantee about future trials. If the agency’s concern is the consequences of proceeding, the response should explain why this scenario is relevant and show results under other clinically important assumptions where appropriate.
Check the Response Before Submission
A response is stronger when the reviewer can follow its reasoning without reconstructing the analysis from scattered documents. Use a direct answer first, then the evidence and qualifications. Attach or identify the supporting material, and ensure that numerical statements agree across the response, protocol, statistical analysis plan, and report. The earlier material on writing Bayesian methods in the clinical study report can help maintain consistency between the response and the final account of the analysis.
A useful review asks whether the answer actually resolves the stated concern, whether its assumptions are explicit, and whether the action described is consistent with the trial’s current status. If the available evidence does not settle the issue, say what remains uncertain and what evidence would address it. Avoid claiming that a sensitivity analysis proves an assumption, or that a simulation under a limited set of scenarios guarantees performance in unexamined settings.
| Response element | Question to check |
|---|---|
| Concern | Have we identified what the agency needs to assess? |
| Evidence | Does the analysis directly address that concern, with assumptions stated? |
| Conclusion | Does the conclusion stay within the scope of the evidence? |
| Action | Is each clarification, additional analysis, or change identified accurately? |
| Traceability | Can the reviewer locate the supporting methods, outputs, and decision records? |
A common mistake is to answer a concern about robustness by repeating the primary posterior result. That result may be correct, but it does not show whether the conclusion depends on a particular prior, data source, or modeling assumption. The correction is to name the dependency at issue and provide a targeted sensitivity analysis, then state whether the main interpretation changes. Conversely, adding every conceivable analysis can obscure the answer; include evidence that addresses the question and explain why it is relevant.
Check Your Understanding
Use the response principles above to answer each question. Brief solutions follow the questions.
- An agency asks whether a prior is influential. What concern should you clarify, and what evidence would directly address it?
- In 2,000 independent simulations, 100 trials meet a specified criterion. Calculate the estimated proportion and its Monte Carlo standard error.
- Why do similar outcome rates in an external study and a current trial not, by themselves, establish that borrowing is appropriate?
- A sponsor adds a sensitivity analysis after seeing trial outcomes. How should the response describe it?
Solutions. (1) Clarify whether the concern is the prior’s effect on the posterior estimate, the decision, or both. Compare the prespecified analysis with a justified alternative prior or reduced-borrowing analysis and report whether the relevant conclusion changes. (2) \(\hat{p}=100/2{,}000=0.0500\). The Monte Carlo standard error is \(\sqrt{0.05(0.95)/2{,}000}=\sqrt{0.00002375}\approx0.00487\), or 0.487 percentage points. (3) Similar rates do not establish comparable populations, outcome measurement, follow-up, or other conditions needed to justify borrowing. (4) Identify it as an additional post hoc sensitivity analysis, state when it was chosen relative to access to outcomes, explain its purpose and assumptions, and distinguish it from the prespecified primary analysis.