Failure Probability and Design Feasibility

Why Failure Probability Belongs in Design Assurance

A trial design can have attractive operating characteristics when it reaches its intended decision, yet still be difficult to execute or fail to produce a usable result in some settings. A trial might end with no treatment selected, stop before its planned decision point, or require more participants than can be enrolled within the available time. These are different events, and a single overall label such as “failure” does not explain which risk is driving the concern.

Failure probability is therefore meaningful only after the event is defined in operational terms. The event must follow from the protocol’s rules and the design question: for example, whether “no selection” means that no treatment is eligible for selection, or whether an early stop counts as failure only when it prevents the intended decision. Then the event can be recorded for each simulated trial replicate under each prespecified scenario.

A scenario describes the data-generating and operational conditions under which the design is assessed. It may specify treatment outcomes, accrual, follow-up, or other features relevant to the design. A failure probability is conditional on that scenario; it is not a universal property of the design independent of the assumed conditions. The earlier material on specifying data-generating scenarios and simulating one adaptive trial replicate provides the setting: each replicate follows the protocol’s decision rules under one specified scenario.

Define Events Before Counting Them

Three useful categories are no selection, premature termination, and infeasible enrollment. Their precise definitions should be written before simulation, not inferred after looking at the results.

• No selection: the trial reaches its designated selection decision, but the protocol selects no treatment, dose, or other candidate. For example, this may occur when no candidate meets a prespecified eligibility rule. If selecting none is an intended and acceptable decision under some scenarios, it is not automatically a design failure; the event may instead be reported as a specific operating outcome.
• Premature termination: the trial stops before the planned decision point. Stopping under a protocol-permitted futility, safety, or other stopping rule is not necessarily a failure. It counts as a failure only if the design question defines that stopping outcome as undesirable, such as an early stop that leaves the intended decision unresolved.
• Infeasible enrollment: under a specified operational criterion, the required enrollment cannot be achieved. The criterion could be failure to enroll the target sample size by a fixed deadline, or a required sample size exceeding a prespecified cap. The simulation must include the operational assumptions needed to determine whether this event occurs.

These events may overlap. A trial might stop early and consequently make no selection. An enrollment shortfall might also lead to an early stop. Thus the probability of “any failure” is not generally the sum of the separate event probabilities. If \(F_1\) and \(F_2\) are two events, then

Alternatively, the protocol can assign each replicate one mutually exclusive primary failure category, using an explicit hierarchy. Either approach can be valid, but the reporting must say which was used. Reporting both the separate event rates and the rate of any failure is often informative.

Estimating a Scenario-Specific Failure Probability

Suppose \(M_s\) simulated replicates are generated under scenario \(s\). For replicate \(r\), define \(I_r=1\) if the specified failure event occurs and \(I_r=0\) otherwise. The estimated failure probability is the mean of these indicators:

The denominator is the number of replicates to which the event definition applies. For a trial-level event such as no selection, that will generally be all simulated trials in the scenario, including trials that stop early if the definition treats their outcome as no selection. Restricting the denominator to completed trials would answer a different, conditional question and could make the design appear more reliable than it is overall.

This proportion estimates the probability under the scenario’s assumptions. For example, if 37 of 1,000 replicates have an enrollment shortfall, the estimated probability is \(37/1000=0.037\), or 3.7%. The result is not a guarantee that the corresponding proportion of future real trials will fail. It is a simulation-based estimate conditional on the specified scenario, protocol, and operational model.

Justifying the Proportion Estimate

The estimator follows directly from the event definition. For a randomly generated replicate under scenario \(s\), the indicator \(I_r\) is a Bernoulli random variable: it equals 1 with probability \(P(F\mid s)\) and 0 otherwise. Thus its expectation is the event probability. If the \(M_s\) replicates are generated independently under the same scenario and protocol, linearity of expectation gives

So the sample proportion is unbiased for the scenario-specific event probability under these simulation conditions. The calculation relies on replicates being generated according to the intended scenario and the event being applied consistently. If the replicates do not represent the same scenario, or if event classification changes partway through the analysis, this justification no longer applies as stated. The later material on Monte Carlo standard errors treats simulation uncertainty in more detail; here the central task is defining and estimating the event correctly.

Enrollment Feasibility Is an Operational Event

An enrollment feasibility assessment needs a measurable target and a time or resource limit. “Recruitment may be difficult” is not a simulation-ready event. A precise definition might be: the design requires \(n_{\text{target}}\) participants, but fewer than \(n_{\text{target}}\) are enrolled by month \(T\). Another could define failure as requiring more participants than a prespecified site or budget cap. Each definition answers a different operational question.

The simulation must also represent assumptions that determine enrollment: for example, the arrival rate of eligible participants, the number of recruiting sites, and any specified interruptions. These assumptions should be varied across relevant scenarios rather than silently treated as certain. A fixed enrollment cap by itself does not determine the probability of failure: the simulated recruitment process and the deadline or cap must together determine whether the event occurs.

Compare Scenarios Without Hiding Risk

A scenario-by-scenario table is usually the clearest starting point. It shows whether a failure probability is concentrated in a particular set of assumptions and prevents a favorable scenario from obscuring a concerning one. An overall average can be useful only when the scenario weights have a defensible interpretation, such as specified probabilities for the scenarios in a target population. Equal weighting is not automatically justified.

These hypothetical rates show a clear difference in enrollment risk across accrual scenarios, while the other two event rates change less. They do not establish that lower accrual is the only relevant concern or that the reference scenario is “the truth.” The scenario definitions and assumptions determine the scope of the comparison.

Diagnostics, Limitations, and a Common Mistake

Before interpreting a failure rate, check whether the event can be determined from every simulated replicate. Confirm that the event definition specifies what happens when a trial stops early, does not reach the planned decision point, or has incomplete enrollment. Verify that the total replicate count agrees with the outcome counts and that any composite event is calculated as a union or by a stated mutually exclusive classification. Inspect results by scenario: a pooled rate can conceal a high-risk scenario, and a zero observed count does not prove that the event is impossible.

A common mistake is to count only completed trials when estimating an event that can prevent completion. The reasoning might be: “No selection is decided only at the end, so early-stopped trials should be omitted.” The correction is to define the event in advance. If an early stop leaves the intended decision unresolved and the design requirement is a decision for every trial, those replicates belong in the failure numerator and in the denominator for the all-trial failure probability. If the question instead concerns no selection among trials that reached the selection decision, a conditional rate can be reported, but it must be labelled as conditional and reported alongside the all-trial outcome.

Failure rates also do not by themselves explain causes. A high enrollment-shortfall rate may reflect a restrictive deadline, an aggressive target, or an unfavorable accrual scenario. A high no-selection rate may be a deliberate consequence of conservative selection criteria rather than an implementation defect. Simulation describes what happens under the modelled rules and assumptions; it cannot establish that unmodelled operational conditions are harmless. These rates complement, rather than replace, other operating characteristics such as selection accuracy, allocation, sample size, and trial duration.