Why “Trial Success” Must Be Defined
A program probability of success is meaningful only when “success” identifies a specific event. A statement such as “the trial has a 70% chance of succeeding” is incomplete unless success is tied to the clinical question, the outcome being evaluated, the population whose outcomes count, and the rule for making the decision. Changing any of these can change which possible trial outcomes count as successes—and therefore can change the estimated probability of success (PoS).
These distinctions matter in model-informed development because a trial can meet a statistical threshold while failing to establish a clinically important effect, or show a promising result in one analysis population but not in the population relevant to the intended treatment decision. A precise definition makes the decision interpretable, reproducible, and comparable across development scenarios.
From Clinical Question to Success Event
An estimand specifies the treatment effect of interest in a defined population. In the ICH E9(R1) framework, it is characterized by the population, treatment conditions being compared, outcome variable, handling of intercurrent events, and population-level summary. An intercurrent event is an event after treatment initiation that affects either the interpretation or existence of the outcome, such as discontinuation, rescue medication, or switching treatment. Its handling must be stated; for example, a treatment-policy strategy uses outcomes regardless of the intercurrent event.
The endpoint is not the whole estimand. It describes the outcome used to measure the effect, such as change from baseline in a symptom score at Week 12. The estimand also specifies whose outcomes count, how intercurrent events are handled, and how the treatment contrast is summarized. Likewise, the analysis population is not automatically the estimand population: the former identifies the participants whose data enter a particular analysis, while the latter is the population to which the treatment effect refers.
A decision criterion turns the estimate and its uncertainty into a binary success classification. It might require a confidence interval to exclude no effect, a point estimate to exceed a clinically meaningful threshold, or both. A statistical criterion alone does not establish clinical importance unless the rule also addresses the size of effect that matters.
A useful formal statement is to define \(S\) as the event that a trial result meets every required condition. For example, \(S\) might require a positive estimated treatment effect, a lower confidence limit above zero, and an estimated effect at least as large as a prespecified clinical threshold. The definition should make clear whether all conditions are required, how missing data and intercurrent events are handled, and which analysis supplies each quantity.
Worked Example: Applying a Prespecified Success Rule
Consider a hypothetical randomized, parallel-group trial comparing a new treatment with control. The estimand is the mean difference between groups in improvement from baseline in a symptom score at Day 24, among eligible randomized participants, using a treatment-policy strategy for discontinuation and rescue medication. Positive values favor the new treatment. The endpoint is change in symptom score at Day 24. All 200 randomized participants have an observed Day 24 score, so the analysis population is all randomized participants. The decision rule requires both a two-sided 95% confidence interval excluding zero in the favorable direction and an estimated improvement of at least 1.0 score point.
Suppose each group has 100 participants, the mean improvements are 4.8 points for the new treatment and 3.2 points for control, and the sample standard deviation in each group is 4.4 points. The estimated treatment effect is:
Assuming independent participants between groups, the standard error for the difference in means is:
For this illustrative large-sample calculation, use the normal approximation. The 95% confidence interval is:
The interval excludes zero in the favorable direction, and the estimated effect of 1.6 points exceeds the 1.0-point threshold. The result therefore meets both parts of the specified success rule. For reference, the test statistic for a zero-effect null is \(z=1.6/0.6223=2.571\), giving a two-sided \(p\)-value of about 0.010. That p-value is consistent with the confidence interval excluding zero, but the decision also depends on the prespecified clinical threshold.
The calculation assumes the group means and standard deviations appropriately summarize the analysis population and that the large-sample normal approximation is adequate. Randomization supports the between-group comparison; the example’s complete follow-up avoids needing a missing-data method. With substantial missing outcomes, treatment switching, or a skewed endpoint, the analysis and its assumptions would need to be specified and justified rather than presumed.
Estimating Program Success from Defined Events
A program-level success event may require success at several development stages. Let \(S_1,S_2,\ldots,S_k\) denote the required stage-success events. If the program succeeds only when every stage succeeds, the program event is their intersection. For three stages, the probability follows from the chain rule:
This identity does not require independence. It requires that the inputs are the relevant conditional probabilities in the stated sequence. Multiplying marginal probabilities instead is justified only under additional assumptions, such as independence. The stage events must also be defined consistently: “Phase II success” should refer to a clear estimand, endpoint, analysis population, and criterion, not just a general impression of a promising result.
Worked Example: A Population Definition Changes a Stage Input
Suppose a hypothetical development team is estimating program PoS from elicited, conditional stage-transition probabilities. Under a treatment-policy estimand in the intended patient population, the team assigns a 0.680 probability to meeting the Phase II success criterion. Conditional on Phase II success, the assigned probabilities of Phase III success and subsequent technical success are 0.600 and 0.410, respectively. These are hypothetical inputs, not observations from a real program.
The corresponding program estimate is:
Now suppose the team instead defines Phase II success using a per-protocol analysis population. Its elicited Phase II transition probability is 0.740, while the later conditional probabilities are held fixed for this comparison. The estimate becomes:
The difference is attributable to the changed Phase II input and the assumption that the other conditional inputs remain applicable. It is not evidence that the per-protocol population makes the treatment more effective. The populations and estimands answer different questions, and a per-protocol analysis may be affected by post-randomization selection. Before comparing the estimates, the team should justify why each input matches its success definition and whether later-stage probabilities can reasonably be held fixed.
Worked Example: A Stricter Definition Changes Program PoS
Consider two hypothetical definitions for a program requiring success at three sequential stages. Under a broader definition, the conditional success probabilities are 0.800 for the first stage, 0.650 for the second given first-stage success, and 0.410 for the third given success at both earlier stages. Under a stricter definition, the corresponding probabilities are 0.750, 0.600, and 0.410. The stricter rule might, for example, require a more clinically meaningful endpoint result or a more demanding analysis criterion. The probabilities are assumed to have been elicited or estimated under their respective definitions.
For the broader definition:
For the stricter definition:
Using the displayed three-decimal estimates, the stricter definition lowers estimated program PoS by \(0.213-0.184=0.029\), or about 2.9 percentage points. This comparison illustrates why a PoS estimate should always be reported alongside its success definition. The change does not, by itself, show that the drug or development plan changed; the event being counted as success changed. Also, the product is appropriate here because the inputs are conditional stage probabilities, not because the stage outcomes have been shown to be independent.
How to Compare Definitions and Diagnose Problems
When two PoS estimates differ, first determine whether the underlying evidence changed or the success event changed. A useful comparison holds the evidence and modeling approach fixed while changing one definition element at a time. For example, compare the same endpoint and criterion under two analysis populations, then compare the same population and estimand under two endpoint definitions. This identifies which change is driving the difference.
The following checks help reveal an ill-defined or inconsistently applied success event:
- Estimand alignment: Can the treatment effect being estimated be stated in terms of population, treatment conditions, outcome, intercurrent-event strategy, and summary? If not, the clinical question may not be sufficiently specified.
- Endpoint clarity: Is the outcome, assessment time, and direction of benefit explicit? A label such as “response” may conceal different definitions or assessment windows.
- Population consistency: Do the data used to estimate a stage probability correspond to the population named in the success rule? If not, explain the bridge between them.
- Criterion reproducibility: Could two analysts apply the rule to the same trial output and reach the same classification? Ambiguity about thresholds, confidence intervals, or multiplicity can make the event irreproducible.
- Conditional-probability coherence: Are later-stage inputs conditional on the earlier events stated in the product? Replacing them with marginal rates without a justified independence assumption can distort program PoS.
Common alternatives include a single statistical test, a clinical threshold on the point estimate, a confidence-interval rule, or a compound rule requiring several conditions. A single test can be simple but may label a small, statistically detectable effect as success. A point-estimate threshold directly addresses effect size but does not alone account for uncertainty. A confidence-interval rule incorporates uncertainty but still needs a justified clinical interpretation. A compound rule can reflect both statistical evidence and clinical relevance, but it must specify the direction of every inequality and whether all requirements must be met.
A frequent mistake is to treat a familiar label—such as “positive trial”—as a complete definition and then compare its PoS with an estimate based on a different endpoint or analysis population. The correction is to write the success event explicitly before assigning or combining probabilities. Another mistake is to multiply probabilities of stage success as though dependence never matters. Use the chain rule with conditional probabilities, or state and defend the assumptions that permit a simpler product.
Even a carefully specified event has limitations. Its probability depends on the quality and relevance of the probability inputs, and a binary success rule discards information about outcomes just above or below a threshold. Different legitimate estimands may also answer different clinical or regulatory questions, so one definition is not automatically correct for every decision. The purpose is not to force a single universal success rule, but to make each rule transparent enough that its implications can be assessed.
State the estimand, including the population, treatment comparison, outcome, intercurrent-event strategy, and summary.
Specify what is measured and when, and whose data enter the analysis that supports the decision.
Define the statistical and clinical conditions that jointly classify a trial result as successful.
Use conditional transition probabilities that correspond to those definitions, and state assumptions when comparing alternatives.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- In one sentence, distinguish an estimand from an endpoint.
- A decision rule requires a lower 95% confidence limit above zero and a point estimate of at least 2 units. A hypothetical estimate is 2.4 units with interval \((0.3,4.5)\). Does it meet the rule?
- Why is the analysis population not necessarily the same as the population in the estimand?
- Given \(P(S_1)=0.70\), \(P(S_2\mid S_1)=0.50\), and \(P(S_3\mid S_1,S_2)=0.40\), calculate the probability of all three stage successes.
- Two program PoS estimates use different endpoint definitions but the same reported numerical inputs. What must be checked before concluding that the underlying program risk is unchanged?
Solutions. (1) The endpoint is the measured outcome at a specified time, whereas the estimand is the treatment effect of interest and also specifies the population, treatment conditions, intercurrent-event handling, and summary. (2) Yes: \(0.3>0\), and \(2.4\geq2\), so both requirements are met. (3) The estimand describes the population to which the treatment effect refers; an analysis population describes which participants’ data are included, and the two can differ because of data availability or analysis choices. (4) By the chain rule, \(0.70(0.50)(0.40)=0.140\). (5) Check that each input probability is defined for the endpoint and success event in question, and that the conditional probabilities remain appropriate under the changed definition; identical numeric values do not ensure that the inputs represent the same events.