Historical Attrition Rates as Base Rates

Why Historical Attrition Rates Need Context

A historical transition rate can provide a useful starting probability for a development decision. For example, a program entering Phase II may be compared with the fraction of similar Phase II programs that historically advanced to Phase III. But “similar” must be defined: the indication, starting phase, transition being counted, cohort period, and success criteria all affect what the rate describes. A percentage taken from a broad industry summary is not automatically a probability for a particular candidate.

The purpose of a historical base rate is to describe outcomes among a specified reference cohort and help anchor a decision when evidence about the current program is incomplete. It is not a guarantee of the current program’s outcome. The main analytical task is therefore to define the reference cohort, calculate the rate with the correct denominator, and judge whether that cohort is relevant to the program being assessed.

Definition: For indication \(i\), starting phase \(p\), and a defined cohort of \(n_{i,p}\) programs entering that phase, let \(y_{i,p}\) be the number that reach the specified next development milestone. The historical transition rate is \(\hat q_{i,p}=y_{i,p}/n_{i,p}\). It estimates the probability of that transition for programs represented by the cohort, subject to how the cohort and outcome were defined.

The denominator consists of programs that entered the specified starting phase, not all programs discovered, all trials conducted, or all compounds considered. The numerator consists of those denominator programs that met the specified transition outcome. “Advanced to Phase III,” “completed Phase III,” and “received approval” are different outcomes and require different counts. A rate is interpretable only when its numerator and denominator refer to the same clearly defined cohort and transition.

In decision trees, a phase-transition probability is conditional on reaching that phase. If the relevant conditional probabilities are specified for each stage, the probability of traversing the full sequence is their product by the chain rule. The historical rate for one transition is not itself a cumulative probability of eventual success.

Calculating a Phase-Specific Base Rate

Worked Example: A Phase II-to-Phase III Base Rate

Suppose, hypothetically, that a review identifies 40 oncology programs that entered Phase II for a defined class of disease during a specified period. Fourteen of those programs subsequently advanced to Phase III. Assume the review has resolved follow-up status for all 40 programs and applies the same advance criterion to each.

The observed transition rate is

\hat q_{\text{oncology, Phase II}}=\frac{y}{n}=\frac{14}{40}=0.350

Thus, 35.0% of the programs in this hypothetical reference cohort advanced from Phase II to Phase III. The denominator is 40 Phase II entrants; it is not the number of programs that eventually reached Phase III. If the current program is sufficiently comparable to this cohort, 0.350 may serve as a historical base-rate input. It should not be described as the probability that this particular program will advance without further qualification.

For a simple binomial description, the cohort is treated as \(n\) comparable opportunities, each having the same transition probability \(q\), with outcome recorded consistently. Under that model, the standard error of the observed proportion is estimated by

SE(\hat q)=\sqrt{\frac{\hat q(1-\hat q)}{n}} =\sqrt{\frac{0.350(0.650)}{40}} =0.0754

This calculation indicates substantial sampling variability around the observed rate under the binomial model. It does not account for systematic differences between the historical cohort and the current program, nor does it establish that all programs truly share one transition probability. Those questions require examining the cohort and its context, not just calculating a standard error.

Cohort Composition Can Change a Pooled Rate

A pooled historical rate is a weighted average of the rates in its component groups. The weights are the groups’ shares of the pooled denominator. Consequently, a change in the mix of indications can change the overall rate even when each indication-specific rate remains constant. A pooled rate may therefore describe the composition of its source cohort more than it describes a target indication.

Result: If group \(j\) has \(y_j\) transitions among \(n_j\) entrants, then the pooled rate is \(\hat q_{\text{pooled}}=\sum_j y_j/\sum_j n_j=\sum_j (n_j/\sum_k n_k)\hat q_j\). It is a weighted average, with weights determined by the numbers of entrants in the groups.

The following calculation shows why the weights matter. Suppose oncology programs have a transition rate of 0.400 and inflammatory-disease programs have a rate of 0.600. If the first cohort contains proportion \(w\) oncology programs, its pooled rate is \(0.400w+0.600(1-w)\). Increasing the share of inflammatory-disease programs raises the pooled rate even though neither indication-specific rate has changed.

Worked Example: Different Indication Mix, Different Pooled Rate

Consider two hypothetical cohorts with the same indication-specific rates. In both cohorts, 12 of 30 oncology programs advance, giving \(12/30=0.400\), and 24 of 40 inflammatory-disease programs advance, giving \(24/40=0.600\). The cohorts differ in how many programs they include from each indication.

In Cohort A, there are 30 oncology and 10 inflammatory-disease entrants. The pooled rate is

\hat q_A=\frac{12+6}{30+10}=\frac{18}{40}=0.450

Here the six inflammatory-disease transitions are \(10(0.600)=6\). In Cohort B, there are 10 oncology and 30 inflammatory-disease entrants. The pooled rate is

\hat q_B=\frac{4+18}{10+30}=\frac{22}{40}=0.550

The four oncology transitions are \(10(0.400)=4\), and the 18 inflammatory-disease transitions are \(30(0.600)=18\). The pooled rate rises from 0.450 to 0.550 solely because Cohort B has a larger share of programs from the indication with the higher transition rate. If a current program is oncology, the 0.550 pooled rate is not a better indication-specific estimate merely because it is higher or more recent. The oncology-specific rate of 0.400 is the more directly matched comparison, provided the other cohort features are also relevant.

This is a composition effect, not evidence that the transition process improved within either indication. It can occur whenever cohorts differ in indication, phase, modality, development setting, or other characteristics associated with transition. For base-rate selection, report the composition as well as the pooled percentage. If the current program’s indication is known, prefer a matched stratum when it has enough relevant information; if pooling is necessary, make the weighting and its limitations explicit.

Selection, Changing Practices, and Relevance

A historical rate is conditional on how programs entered the reference cohort. A database may include only programs with publicly available outcomes, only programs that reached a certain phase, or only programs from organizations that disclose discontinuations. If the likelihood of inclusion is related to success, the observed rate can differ from the rate among all programs that would otherwise meet the intended cohort definition. A defensible analysis states who was eligible, how programs were found, and how missing or unresolved outcomes were handled.

Time also matters. Development standards, trial design, regulatory expectations, diagnostics, and the types of candidates advanced can change. A historical cohort may include programs evaluated under different endpoints, populations, or success criteria from those used for the current program. Even when the indication and phase match, the meaning of “transition” may not. A rate based on advancement to the next phase can reflect an organization’s investment decision as well as evidence about efficacy and safety; it is not a pure measure of biological success.

Worked Example: Interpreting a More Recent Rate

Suppose, hypothetically, that 20 of 50 programs in an older Phase II cohort for one indication advanced to Phase III. In a later cohort from the same indication and phase, 22 of 40 advanced. Assume both cohorts are fully followed and use the same nominal transition definition.

The older rate is \(20/50=0.400\), and the later rate is \(22/40=0.550\). Across both periods, the pooled rate is

\hat q_{\text{pooled}}=\frac{20+22}{50+40}=\frac{42}{90}=0.467

If the current program is being assessed under conditions most like the later cohort, 0.550 may be more relevant than 0.467. But the difference does not by itself show that the underlying probability increased: the candidates selected for Phase II may have changed, or trial and advancement practices may differ. Conversely, if the outcome definitions are not comparable, even the apparent same-indication comparison is weak. The appropriate conclusion is conditional: the observed rates are 0.400 and 0.550 in their respective hypothetical cohorts; choosing one as a current base rate requires a substantive comparison of cohort eligibility, follow-up, outcome definitions, and practices.

Justifying a Base-Rate Choice

A useful way to justify a historical input is to work from the current decision back to the reference data. The target quantity should be the probability of a clearly stated transition for programs sufficiently comparable to the current one. A historical rate is a candidate estimate of that quantity only to the extent that its source cohort represents the target situation.

Specify the transition.
Name the starting phase and the exact next milestone. Define how progression is recorded and the follow-up period used to determine outcomes.

Define the target cohort.
Identify the indication and other features relevant to comparability, such as modality or development period. Avoid adding distinctions without a reason to expect them to matter.

Check the counts and selection.
Verify which programs entered the denominator, which met the outcome, and whether exclusions, missing outcomes, or eligibility rules could distort the observed rate.

Assess transportability.
Compare the historical cohort’s practices and definitions with the current setting. Use a matched rate when justified; if pooling is needed, state how cohort composition affects the result.

This process does not require assuming that an historical average is exactly the current program’s probability. It makes explicit what must be true for the rate to serve as a useful base rate. When evidence suggests that the reference cohort differs materially, report the mismatch and consider a range of plausible inputs rather than presenting the pooled percentage as uniquely correct.

Diagnostics, Alternatives, and Limitations

Several checks help reveal when an historical rate may be misleading:

Denominator check: Confirm that the denominator is entrants to the stated starting phase, not a mixture of phases or a subset selected after observing later outcomes.
Outcome check: Confirm that success means the same event for every program, and that enough follow-up has elapsed to classify outcomes.
Composition check: Tabulate the cohort by indication and other relevant characteristics; compare subgroup rates before relying on a pooled estimate.
Time-period check: Compare development practices and outcome definitions across periods. A recent cohort is not automatically more relevant if it is small or differently selected.
Precision check: Examine the number of entrants behind the rate. A percentage from a small cohort can vary substantially with a few outcomes; precision does not correct poor comparability.

The main alternative is not one universal replacement, but a different source of evidence suited to the question. The earlier material on eliciting phase-transition probabilities from evidence provides a framework for combining relevant evidence rather than relying on a single historical proportion. Expert judgment may help where comparable observations are scarce, but its assumptions should be made explicit. Program-specific evidence can also justify a probability different from the historical base rate. Each approach has limitations: historical data may be poorly matched, judgment may be uncertain, and current evidence may be sparse or not yet predictive of the transition decision.

An observed historical rate also has sampling uncertainty and possible systematic bias. A large cohort can reduce random uncertainty under a suitable statistical model, but it cannot repair an incorrect endpoint, selective inclusion, or a substantial mismatch in development practice. Conversely, a narrowly matched cohort may be highly relevant but too small to estimate a stable rate. A transparent decision analysis distinguishes these limitations rather than treating one decimal place as evidence of certainty.

Common pitfall: “The overall historical rate is 55%, so the probability for this program is 55%.” This reasoning ignores what programs entered the historical denominator and whether their indication, phase, selection, follow-up, and success definition match the current program. The correction is to define the target transition, inspect the cohort, and describe the rate as a base-rate estimate only when the comparison is defensible.

Check Your Understanding

Try each problem, then compare your reasoning with the solutions.

A hypothetical review finds that 18 of 60 Phase I entrants in a specified indication advanced to Phase II. Calculate and interpret the historical transition rate.
Two indication groups have rates of 0.300 and 0.700. A pooled cohort contains 80% of its entrants from the first group and 20% from the second. Calculate the pooled rate.
A historical rate is based on programs with publicly reported outcomes, and discontinued programs are less likely to be reported. What concern does this create?
A recent cohort has a higher transition rate than an older cohort, but the advancement criterion changed between periods. Explain why the recent rate cannot automatically be treated as a better estimate.
Why is a rate for reaching Phase III not the same as a probability of eventual approval?

Solutions. (1) \(\hat q=18/60=0.300\). Thus 30.0% of the specified hypothetical Phase I entrants advanced to Phase II; this is a cohort rate, not a certainty about any one program. (2) The weighted pooled rate is \(0.80(0.300)+0.20(0.700)=0.240+0.140=0.380\), or 38.0%. (3) Inclusion may depend on outcome, so the observed cohort can overrepresent successful programs and its rate may not represent all eligible entrants. (4) The outcome definitions differ, so the rates do not measure the same transition event; the apparent increase may reflect the changed criterion rather than a changed transition probability. (5) Reaching Phase III is one phase transition. Eventual approval requires additional transitions and a defined approval outcome, so the single-phase rate does not equal the cumulative probability of approval.