Why Timing Matters
Agency engagement is most useful when a trial team has a concrete scientific question and enough time to respond to the answer. For a Bayesian design, relevant questions may concern the trial’s decision context, how proposed prior information will be used, the rule that will guide a decision, or what evidence the analysis is intended to support. If the team raises these matters only after the design is fixed or the data are available, a concern may require a costly redesign or leave little opportunity to resolve an evidentiary disagreement.
Early engagement does not mean asking an agency to design or approve every statistical choice in advance. The sponsor remains responsible for the protocol, analysis, conduct, and interpretation. Agency feedback is scientific and regulatory input within a particular interaction; it is not a guarantee that a later submission will be accepted. The appropriate timing and format depend on the jurisdiction, product, development stage, question, and available formal processes.
Choose Questions That Can Still Change the Plan
A useful question is specific enough to receive a meaningful response and important enough that the response could affect the plan. “Is this Bayesian approach acceptable?” is usually too broad on its own. The team should be able to identify what is proposed, what uncertainty remains, and what it would do differently depending on the feedback. For example, it may ask whether the planned design and analysis address a stated regulatory question, whether a proposed source of prior information raises concerns that should be resolved before enrollment, or whether the decision criterion is framed in a way that supports the intended interpretation.
The following checks help establish whether an issue is ready for discussion:
- Decision relevance: Could the answer change the design, analysis, interpretation, or evidence plan?
- Timing: Is there still time to implement a change before the relevant decision or trial milestone?
- Specificity: Can the team explain the proposed approach and the alternatives under consideration?
- Consequence: What would be the scientific, operational, or regulatory cost if the issue remains unresolved?
These checks are planning aids, not a substitute for the applicable agency process. They also discourage discussions dominated by questions that have no bearing on the proposed trial or are not yet sufficiently developed to evaluate.
Worked Example: Deciding Whether to Raise a Design Question Now
A hypothetical sponsor plans to start a trial in 10 months. The protocol is scheduled to be finalized in month 4. The team is uncertain whether its planned Bayesian design is suitable for the intended evidence claim. It estimates that a relevant agency interaction and internal follow-up would take 3 months. Should it wait until the protocol is finalized?
If the team waits until month 4 to raise the question, the estimated discussion and follow-up period would end in month \(4+3=7\). That leaves \(10-7=3\) months before trial start, but the protocol would already have been finalized. If the answer requires a design change, the team may need to reopen the protocol and repeat work.
Suppose, for planning only, the team estimates a 0.30 chance of a consequential design change being needed and assigns 6 months of delay to resolving that change after protocol finalization. The expected delay from waiting is \(0.30\times6=1.8\) months. If engaging earlier requires 1 month of staff time and coordination, the team has a practical reason to discuss the issue before the month-4 freeze.
This calculation does not predict what an agency will say or establish that the team’s estimates are accurate. It makes the trade-off explicit: a discussion has a cost, but waiting can make an important change more expensive. The team should raise the question early enough that feedback can still inform the design.
Prior Information and Decision Criteria Need Lead Time
A proposed prior can affect posterior summaries and decisions, so a concern about its role is easier to address before it becomes embedded in the design. The team should be ready to explain, at a suitable level, what information it proposes to use and how that information enters the analysis. It can also show how the analysis changes under a reasonable alternative. The purpose of early discussion is to surface questions about relevance, influence, or interpretation; an interaction does not replace a careful justification of the prior or the supporting analyses.
Similarly, a decision rule should be discussed in the context of the decision it is meant to support. A posterior probability cutoff is a mathematical rule, not by itself a complete account of what evidence is persuasive. Its interpretation depends on the endpoint, clinical context, design, and consequences of a decision. A team should therefore distinguish “the analysis meets the prespecified rule” from “the total evidence is sufficient for the intended regulatory conclusion.”
Worked Example: Why a Proposed Prior Is Worth Discussing Early
In a hypothetical single-arm study, the endpoint is binary and the parameter \(p\) is the response probability. The team proposes a \(\operatorname{Beta}(2,8)\) prior based on earlier information and expects 18 responses among 30 participants. Under a beta-binomial model, the posterior is \(\operatorname{Beta}(2+18,8+30-18)=\operatorname{Beta}(20,20)\).
The posterior mean is \(20/(20+20)=0.500\). For comparison, using a \(\operatorname{Beta}(1,1)\) reference prior with the same data gives \(\operatorname{Beta}(19,13)\), whose mean is \(19/(19+13)=0.5938\). The difference between these posterior means is \(0.5938-0.500=0.0938\), or 9.38 percentage points.
This illustration shows that the prior choice can materially change a summary even with 30 observations. It does not establish that either prior is appropriate, nor does the mean difference alone determine whether a regulatory decision changes. It gives the team a focused reason to seek feedback before the choice is fixed: does the proposed use of earlier information raise concerns that should be resolved, and what additional analyses or explanation may be important? The sponsor still needs to justify the chosen analysis and its implications.
Worked Example: Clarifying What a Decision Rule Does and Does Not Say
Suppose a hypothetical team proposes a response-probability rule: declare the result positive if \(P(p>0.30\mid\text{data})\geq0.95\). With a \(\operatorname{Beta}(1,1)\) prior and 8 responses among 10 participants, the posterior is \(\operatorname{Beta}(9,3)\). For this integer-parameter beta distribution, the probability can be calculated using a binomial tail:
The proposed rule is met because \(0.9994\geq0.95\). This calculation answers a narrow statistical question under the stated model and prior. It does not, on its own, establish that 0.30 is a clinically appropriate benchmark, that the study’s design is adequate for the intended use, or that the evidence will be sufficient for a regulatory conclusion.
Before relying on the rule, the team could ask whether its clinical interpretation and intended role are clear, and whether the broader evidence plan addresses the agency’s concerns. The example is not a recommendation for this threshold or design. It shows why a numerical criterion and the evidentiary question should be discussed separately.
A Practical Rationale for Early Engagement
A simple expected-cost calculation explains why timing matters. Let \(q\) be the team’s estimated probability that an unresolved issue will require a change, \(C\) the cost of making that change late, and \(M\) the cost of engaging early. Under this simplified planning model, the expected late-change cost is \(qC\). Early engagement is worth considering when:
This is not a validated model of agency interactions. It omits benefits that are difficult to quantify, such as identifying a problem before it affects participant enrollment, and it depends on subjective estimates of \(q\), \(C\), and \(M\). Its value is as a transparent way to compare the cost of engagement with the potential cost of waiting.
The reasoning is straightforward: if a late change is costly, even a modest chance of needing one can justify a timely discussion. Conversely, if the question is unlikely to affect the plan or the team cannot yet describe its proposal, it may be better to refine the issue first. The key is not to engage as early as possible in every case, but early enough for the discussion to inform a decision that remains open.
State which trial, analysis, or evidence decision is unresolved and when it must be made.
Describe the proposed approach, the uncertainty, and the specific feedback sought.
Allow time for the applicable interaction, internal review, and any feasible response to feedback.
Decide how the team will evaluate feedback and what options remain if the issue is not resolved.
Choosing a Route and Recognizing Limits
The main alternatives are to seek a structured agency interaction, send a focused written question through an available process, or proceed without agency feedback. The suitable route depends on the question and the relevant agency’s procedures; meeting types, eligibility, and timelines differ. A focused written exchange may suit a narrow clarification, while a complex design question may benefit from a discussion that allows the team to explain context and hear follow-up questions. Neither route guarantees agreement.
| Approach | Potential value | Limitation to consider |
|---|---|---|
| Structured agency interaction | Can address several linked, consequential questions with shared context. | Requires preparation and lead time; feedback is not a guarantee of a future regulatory decision. |
| Focused written question | Can make a narrow issue and the requested response clear. | May not resolve a complex issue that depends on broader design context. |
| Proceed without early feedback | Avoids interaction time when the issue is minor or the approach is already sufficiently clear. | Leaves the team to manage uncertainty and any later changes without that input. |
Good preparation improves the chance that the interaction is useful. The team should be able to state the decision context, explain the proposed approach, identify the specific uncertainty, and say how the answer could affect the plan. The detailed contents of a formal submission or meeting package depend on the applicable process and should not be confused with the earlier task of deciding when engagement is needed.
Feedback may be conditional, incomplete, or different from what the team expected. A discussion may also reveal that the question is too broad or that additional evidence is needed before the agency can comment. Record the advice and its context, assess it with the relevant clinical, statistical, and regulatory experts, and document the sponsor’s reasoning for the final plan. Do not treat silence, an informal comment, or a response to one narrow question as approval of the entire Bayesian strategy.
Common Pitfalls
- Waiting until the design is effectively fixed. The mistaken reasoning is that feedback is useful at any time. In practice, late feedback may arrive after important changes have become costly or infeasible. Raise consequential uncertainties while alternatives remain open.
- Asking for blanket approval. A broad question such as “Is the Bayesian method acceptable?” may obscure the actual issue. State the proposed method, the decision it supports, and the specific feedback sought.
- Treating a posterior cutoff as self-justifying. Meeting a numerical rule is not automatically equivalent to sufficient evidence. Explain the role of the rule in the clinical and regulatory context.
- Assuming agency feedback transfers responsibility. Feedback is input, not a substitute for the sponsor’s scientific justification, prespecification, or accountability for the study.
- Seeking feedback without a feasible response plan. If no design or analysis change is possible, the team should recognize that limitation and consider whether a discussion can still inform interpretation or future development.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A design issue could delay a trial if discovered late. What two questions help determine whether to raise it early?
- In the prior example, why does the difference between posterior means support early discussion but not prove that either prior is acceptable?
- A team meets a prespecified posterior probability cutoff. What broader conclusion does that calculation not establish by itself?
- When might a focused written question be more suitable than a broad discussion?
- Does agency feedback guarantee that a later regulatory submission will be accepted? Explain.
Solutions. (1) Ask whether the answer could change a consequential decision and whether enough time remains to act on the feedback. (2) The 9.38-percentage-point difference illustrates sensitivity of a summary to the prior, but acceptability depends on the information’s justification and the full evidence context. (3) It does not by itself establish that the clinical benchmark, design, or total evidence is adequate for the intended regulatory conclusion. (4) A written question may suit a narrow, clearly stated issue that does not require extensive design context. (5) No. Feedback is input in a particular interaction, not a guarantee of a future decision or a transfer of sponsor responsibility.