Interpreting I-SPY 2 Graduation Predictions

What a Graduation Prediction Says

In I-SPY 2, a regimen’s result was evaluated within biomarker-defined signatures: groups of patients specified by tumor characteristics such as hormone-receptor status, HER2 status, and genomic risk. A signature-specific prediction therefore concerns activity in a defined subgroup, not every patient with breast cancer and not necessarily every other subgroup in the trial.

For this purpose, activity was assessed using pathologic complete response (pCR), the neoadjuvant endpoint introduced in the earlier material on pathologic complete response in I-SPY 2. The platform’s graduation rule was based on the predicted probability that a regimen would meet a success criterion in a future phase III trial. In the commonly described I-SPY 2 framework, a regimen graduated if this predictive probability exceeded 85% in at least one signature, with future success assessed using pCR.

Definition: A graduation prediction is a model-based predictive probability that a regimen will meet a specified success criterion in a future trial, for a particular biomarker signature. It is conditional on the observed trial data, the statistical model, and assumptions about that future trial. It is not the probability that the regimen is effective, nor proof that a future trial will succeed.

The estimated pCR rates and their difference describe the observed trial’s treatment contrast in a signature. The predictive probability answers a different question: given the uncertainty in that contrast and the expected uncertainty in a future trial, how likely is the future success criterion to be met? A large estimated difference can still produce a modest predictive probability if the estimate is imprecise. Conversely, a precise, promising estimate can yield a high probability of future success without guaranteeing it.

How to Read the Signature-Specific Results

A graduation prediction must be read together with the signature, endpoint, comparator, and treatment regimen. In particular, “graduated in a signature” does not mean “proved effective in all patients.” It means that the model’s forecast crossed a prespecified decision threshold for a future trial in that signature. A prediction below the threshold does not establish that the regimen has no activity; it means the specified graduation criterion was not met on the available evidence.

I-SPY 2 used an adaptive platform design and Bayesian modeling to estimate treatment effects across biomarker signatures. That framework can use information across groups, subject to its modeling assumptions. Signatures may overlap or be nested: a broad HER2-negative group, for example, includes more specific hormone-receptor-defined groups. Predictions across such signatures are not automatically independent replications. A favorable result in a broad group does not, by itself, prove equal benefit in each narrower subgroup.

Worked Example: Veliparib and Carboplatin in the Triple-Negative Signature

In the I-SPY 2 evaluation of veliparib plus carboplatin with paclitaxel, the estimated pCR rate in the triple-negative signature was about 51%, compared with about 26% for the control regimen. The reported predictive probability of success in a future phase III trial for that signature was about 92%.

The estimated absolute difference in pCR rates is:

51\%-26\%=25\text{ percentage points}.

The interpretation is that, in this trial and signature, the model estimated pCR to be 25 percentage points more frequent with the treatment regimen than with control. Separately, the 92% figure means that the model assigned a 92% predictive probability to meeting the specified future-trial success criterion, conditional on its assumptions. Since 92% exceeded the 85% graduation threshold, the regimen graduated in that signature.

Neither number establishes that veliparib itself caused the difference. The I-SPY 2 comparison evaluated a regimen containing both veliparib and carboplatin against its control, so the contrast does not isolate each component’s contribution. In the later BrighTNess trial, the pCR rate was 53% with veliparib plus carboplatin and 58% with carboplatin without veliparib. That comparison addresses the incremental contribution of veliparib in a way the I-SPY 2 contrast did not. The different comparisons answer different questions; the graduation forecast was not a substitute for that confirmatory evidence.

Worked Example: Pembrolizumab in HER2-Negative Disease

In the I-SPY 2 pembrolizumab evaluation, the estimated pCR rates in the HER2-negative signature were about 44% with pembrolizumab plus chemotherapy and 17% with control. The reported predictive probability of success in a future phase III trial for the HER2-negative signature was about 99%.

The observed-model contrast in pCR rates was:

44\%-17\%=27\text{ percentage points}.

This result supports a promising pCR signal for the pembrolizumab-containing regimen in the HER2-negative signature under the trial’s model. The 99% forecast is not a 99% probability that an individual patient will respond, a 99% probability that pembrolizumab is beneficial, or a 99% chance that the true treatment effect is positive. It is a predictive probability for a future trial’s specified success criterion.

Predictions can also differ between signatures. For pembrolizumab, the reported predictive probabilities were approximately 88% in the HR-positive/HER2-negative signature and 97% in the HR-negative/HER2-negative signature. Those forecasts can support graduation in each of those signatures under the threshold rule, but they do not show that the underlying effects are identical. Nor should the subgroup forecasts be treated as independent confirmations: the signatures share patients and are modeled within the same platform.

Why Predictive Probability Is Not Just the Estimated Effect

A short calculation shows why predictive probability depends on both the estimated effect and uncertainty. The calculation below is a normal-approximation illustration, not a reconstruction of I-SPY 2’s full model. Let \(\theta\) be the true treatment effect on the log-odds scale, with positive values favoring treatment. Suppose the current data give a posterior distribution \(\theta\mid D\sim N(m,v)\). For a hypothetical future trial, let its estimated effect be \(Y\), with \(Y\mid\theta\sim N(\theta,s^2)\). Suppose future success is declared when a one-sided test at level \(\alpha\) rejects no benefit, so success requires \(Y>z_{1-\alpha}s\).

Integrating over uncertainty in \(\theta\) gives the predictive distribution for the future estimate. Because the uncertainty in \(Y\) has two sources—the uncertain true effect and future sampling error—the variances add:

Y\mid D\sim N(m,v+s^2).

Therefore, the predictive probability is the probability that a random value from this distribution exceeds the success threshold:

P(\text{future success}\mid D) =1-\Phi\left(\frac{z_{1-\alpha}s-m}{\sqrt{v+s^2}}\right).

This expression makes the distinction clear. The forecast depends not only on the posterior mean \(m\), but also on posterior uncertainty \(v\), future-trial standard error \(s\), and the success threshold. The full I-SPY 2 calculations used the platform’s own Bayesian model and trial assumptions; this simple expression illustrates the principle rather than supplying its exact calculation.

Worked Example: A Favorable Estimate Can Still Have a Low Forecast

Suppose, purely hypothetically, that the posterior for a log-odds treatment effect is \(N(0.40,0.20^2)\), and a future trial has standard error \(s=0.25\). Assume a one-sided test with \(\alpha=0.025\), so \(z_{1-\alpha}\approx1.96\). The future estimate must exceed:

$$ 1.96(0.25)=0.49. $$

The predictive standard deviation is:

\sqrt{0.20^2+0.25^2}=\sqrt{0.1025}\approx0.320.

Thus the predictive probability is approximately:

1-\Phi\left(\frac{0.49-0.40}{0.320}\right) =1-\Phi(0.281)\approx0.389.

Although the posterior mean effect is positive, this hypothetical forecast is about 39%, well below an 85% threshold. The example shows why a favorable point estimate is not enough to infer graduation: uncertainty and the future success criterion matter too.

Assumptions, Diagnostics, and Limits

A predictive probability is meaningful only relative to its modeling and design assumptions. For a signature-specific forecast, relevant assumptions include that the pCR outcome is ascertained consistently, the biomarker signature is defined and measured consistently, and the model adequately represents treatment effects and their uncertainty across signatures. The forecast also depends on how the future trial is imagined: its eligibility criteria, comparator, endpoint, sample size, randomization, and success rule.

Diagnostics should therefore look beyond whether a reported probability crossed 85%. Analysts should examine the estimated pCR rates and uncertainty, the number of patients contributing information in each signature, and whether the model’s fit and assumptions are credible. They should also check whether the favorable signature was prespecified for decision-making, whether the result is driven by a small subgroup, and whether overlapping signatures make several apparently favorable predictions reflect much of the same evidence. Changing control outcomes over time or differences between the trial population and the proposed confirmatory population can further weaken the forecast’s relevance.

A graduation prediction is not a conventional p-value, a confidence interval, or a posterior probability that the null hypothesis is false. It does not account automatically for every source of uncertainty that may arise in later practice or a different trial. In particular, it cannot establish long-term clinical benefit from a pCR forecast alone, isolate a component of a combination regimen, or guarantee transportability to patients outside the signature and trial setting. Confirmatory evidence is needed to assess the treatment comparison in the population and design that matter for practice.

Key takeaway: Interpret graduation as a model-based forecast of success in a future trial for a specified signature—not as proof of efficacy. Always pair the threshold decision with the estimated endpoint contrast, its uncertainty, the exact treatment comparison, and the assumptions behind the prediction.

Common Mistakes

Calling the predictive probability a probability the regimen works. It is the probability of meeting a future trial success criterion under the model, not the probability that the treatment is truly effective.
Confusing a pCR difference with the forecast. A 25-percentage-point estimated difference describes the trial contrast; a 92% graduation prediction incorporates uncertainty and a future-trial rule.
Generalizing beyond the signature. Graduation in a biomarker-defined subgroup does not establish the same effect in every patient or in every overlapping subgroup.
Attributing a combination effect to one component. A regimen-versus-control comparison cannot isolate an added drug when the treatment arms differ in multiple components.
Treating graduation as confirmation. Crossing the platform threshold supports moving forward; it does not replace evidence from a confirmatory comparison.

Check Your Understanding

Answer each question, then compare your reasoning with the brief solutions below.

A signature has estimated pCR rates of 40% with treatment and 25% with control. What is the estimated absolute difference, and what does it not tell you by itself?
A regimen has a predictive probability of 82% and the graduation threshold is 85%. What is the correct interpretation?
Why should predictions for a broad signature and a nested subgroup not be counted automatically as independent confirmations?
A combination regimen graduates against control, but a later trial compares the combination with one component removed. Why can the results answer different questions?

Solutions. (1) The difference is \(40\%-25\%=15\) percentage points. It describes the estimated contrast in that signature but does not provide the predictive probability, establish a true effect, or guarantee future success. (2) The forecast is below the stated threshold, so the regimen does not meet that graduation rule on this prediction; this does not prove absence of activity. (3) The groups share patients and may use common model information, so their forecasts are statistically related rather than independent replications. (4) The first comparison estimates the effect of the full combination relative to control, while the second can assess the added contribution of the removed component; the treatment contrasts are different.