Why Sensitivity Analysis Is Needed
An intermediate outcome can make an arm-selection decision possible before the final outcome is sufficiently mature. But the usefulness of that decision depends on assumptions about how intermediate and final outcomes relate. Those assumptions may be uncertain even when the selection rule, significance level, and planned sample size are fixed. Sensitivity analysis makes that uncertainty visible by recalculating selection and final-success probabilities over a justified range of assumptions.
Four inputs deserve separate attention: correlation between intermediate and final outcome statistics; the relationship between treatment effects on the two outcomes; how much outcome information is available at each look; and the accuracy with which the intermediate outcome is measured. They are not interchangeable. A high patient-level correlation does not by itself establish that treatment effects transfer reliably from the intermediate outcome to the final outcome.
For clarity, let \(A\) denote the event that an arm passes the intermediate selection rule, and let \(F\) denote the event that its final test meets the prespecified success criterion. Then \(P(A\cap F)=P(A)P(F\mid A)\). Reporting only \(P(F\mid A)\) can conceal a selection rule that rarely retains the arm; reporting only \(P(A)\) says nothing about its chance of final success.
Separate Correlation From Surrogacy
A useful starting model treats the intermediate and final test statistics for one arm as jointly normal:
Here \(Z_I\) and \(Z_F\) are standardized statistics oriented so that larger values favor treatment, \(\mu_I\) and \(\mu_F\) are their means under the scenario being examined, and \(\rho\) is their correlation. This model assumes an approximately bivariate-normal sampling distribution, unit marginal variances, and a correlation that adequately represents dependence for the specified design and analysis. In practice, the correlation may change with follow-up, missingness, or the analysis population, so it should be varied rather than treated as known without qualification.
A different uncertainty concerns treatment effects themselves. For example, a working predictive relationship might be
where \(\Delta_I\) and \(\Delta_F\) are treatment effects on the intermediate and final outcomes, \(b\) is the treatment-effect surrogacy slope, and \(\tau^2\) represents unexplained variation in final effects at a given intermediate effect. This is a model across effects or settings, not a definition of patient-level correlation. A large \(\rho\) can coexist with substantial \(\tau^2\); treatment effects may still be poorly predicted by intermediate effects.
Deriving the Conditional Final Probability
The joint-normal model gives a direct way to understand why correlation matters after selection. Conditional on an observed intermediate statistic \(Z_I=z\), the final statistic has a normal distribution:
To see this, subtract from \(Z_F\) the linear prediction based on \(Z_I\): \(R=Z_F-\mu_F-\rho(Z_I-\mu_I)\). Under the stated joint-normal model, \(R\) is independent of \(Z_I\), has mean zero, and has variance \(1-\rho^2\). Therefore, fixing \(Z_I=z\) leaves the displayed conditional mean and variance. For a final success boundary \(c_F\), the conditional probability is consequently
where \(\Phi\) is the standard normal cumulative distribution function. Selection is usually based on an event such as \(A=\{Z_I>c_I\}\), not on a single observed value. The probability conditional on selection averages over all intermediate values that pass:
This expression also shows why a result for one observed statistic is not the same as the operating characteristic conditional on selection. For multiple arms or more complex selection rules, the same principle applies, but the relevant joint probabilities must reflect the actual rule.
Worked Example: Correlation and a Strong Intermediate Result
Suppose a hypothetical arm has \(\mu_I=\mu_F=1.5\). It passes selection when \(Z_I>1.5\), and final success requires \(Z_F>1.96\). The observed intermediate statistic is \(z=3.0\). Under the joint-normal model, compare the conditional final-success probability for \(\rho=0.20\) and \(\rho=0.80\).
For \(\rho=0.20\), the conditional mean and standard deviation of \(Z_F\) are
Thus \(P(Z_F>1.96\mid Z_I=3.0)=1-\Phi((1.96-1.80)/0.9799)=1-\Phi(0.1633)\), approximately \(0.435\).
For \(\rho=0.80\), the conditional mean and standard deviation are
The conditional success probability is \(1-\Phi((1.96-2.70)/0.6000)=1-\Phi(-1.233)\), approximately \(0.891\). In this scenario, a strong intermediate statistic is much more informative about final success under the higher correlation assumption. These are probabilities conditional on the observed value \(Z_I=3.0\), not probabilities conditional on merely passing the selection boundary.
Vary Treatment-Effect Surrogacy
When the concern is whether an intermediate treatment effect predicts the final treatment effect, a sensitivity analysis can vary the slope and residual variation in a predictive model. The following calculation uses an explicit approximation: conditional on an observed intermediate effect estimate \(\hat{\Delta}_I=x\), suppose the predictive distribution for the final effect is normal, with mean \(a+bx\) and variance \(\tau^2+b^2SE_I^2\). This variance accounts for unexplained surrogacy variation and uncertainty in \(\hat{\Delta}_I\). Assume, additionally, that the final estimate has independent sampling error with standard error \(SE_F\). Then the predictive distribution of the final estimate has variance \(\tau^2+b^2SE_I^2+SE_F^2\). These are modeling assumptions for the calculation, not universal facts.
Worked Example: Alternative Surrogacy Slopes
Suppose, hypothetically, that \(\hat{\Delta}_I=0.30\) with \(SE_I=0.10\), \(a=0.02\), \(\tau=0.05\), and the final estimate has \(SE_F=0.12\). Final success is defined as a one-sided standardized statistic greater than \(1.96\), equivalent here to a final estimate greater than \(1.96(0.12)=0.2352\). Compare \(b=0.50\) with \(b=1.50\).
For \(b=0.50\), the predictive mean is \(0.02+0.50(0.30)=0.170\). The predictive variance of the final estimate is
giving a standard deviation of \(0.1393\). The predicted probability of success is \(1-\Phi((0.2352-0.170)/0.1393)=1-\Phi(0.468)\), approximately \(0.320\).
For \(b=1.50\), the predictive mean is \(0.02+1.50(0.30)=0.470\), and the predictive variance is \(0.0025+1.50^2(0.10^2)+0.0144=0.0394\), giving a standard deviation of \(0.1985\). The predicted probability of success is \(1-\Phi((0.2352-0.470)/0.1985)=1-\Phi(-1.183)\), approximately \(0.882\). The large difference reflects the assumed slope and predictive model; it is not evidence that either slope is correct.
Measurement Error and Outcome Delay
Measurement error can reduce the intermediate effect signal, increase its standard error, or do both. The appropriate sensitivity model depends on how the outcome is recorded. One simple attenuation model is \(Y_I=\lambda\Delta_I+\epsilon\), where \(Y_I\) is the measured intermediate effect, \(0<\lambda\leq1\) is an attenuation factor, and \(\epsilon\) is sampling error. Under this model, decreasing \(\lambda\) can move an arm’s expected intermediate statistic toward the selection boundary or below it. If the error also increases variability, that change must be included in the standard error; changing the mean alone would not represent the full measurement-error problem.
Worked Example: Attenuation Changes Selection Probability
In a hypothetical study, the true intermediate effect is \(0.40\) units, the measured effect has standard error \(0.20\) units, and an arm is retained when the measured effect exceeds \(0.30\) units. Assume the attenuation model \(Y_I=\lambda(0.40)+\epsilon\), with \(\epsilon\sim N(0,0.20^2)\). Compare \(\lambda=0.90\) and \(\lambda=0.50\).
With \(\lambda=0.90\), the measured effect has mean \(0.36\). The probability of retention is \(P(Y_I>0.30)=1-\Phi((0.30-0.36)/0.20)=\Phi(0.30)\), approximately \(0.618\).
With \(\lambda=0.50\), the mean is \(0.20\), so the retention probability is \(1-\Phi((0.30-0.20)/0.20)=1-\Phi(0.50)\), approximately \(0.309\). The calculation assumes the standard error is unchanged. If measurement error also raised it, retention would need to be recalculated using that larger standard error.
Outcome delay changes how much information is available at a planned calendar-time look. For event-driven time-to-event analyses, a common large-sample approximation under proportional hazards and equal allocation is that a treatment-effect statistic has unit variance and mean approximately \(\theta\sqrt{E}/2\), where \(E\) is the number of observed events and \(\theta=-\log(HR)\) is positive when the hazard ratio \(HR\) favors treatment. This approximation requires the stated model and is not a substitute for modeling complex accrual, censoring, or nonproportional hazards.
Worked Example: Delay Changes Available Event Information
Suppose a hypothetical time-to-event trial has an intermediate selection boundary \(Z_I>1.50\) and a true intermediate effect \(\theta_I=0.40\) log-hazard units. With 80 intermediate events by the planned look, the approximate mean statistic is \(0.40\sqrt{80}/2=1.789\). Therefore, the probability of passing selection is \(1-\Phi(1.50-1.789)=\Phi(0.289)\), approximately \(0.614\).
If slower outcome reporting leaves only 45 events at that same calendar-time look, the mean becomes \(0.40\sqrt{45}/2=1.342\). The selection probability is \(1-\Phi(1.50-1.342)=\Phi(-0.158)\), approximately \(0.437\). The lower probability arises from less information at the look, not from a change in the assumed true treatment effect.
Delay can also affect final success if fewer final events are available at the final analysis. For example, with \(\theta_F=0.30\), 160 events give mean \(0.30\sqrt{160}/2=1.897\); the approximate probability of exceeding a final boundary of \(1.96\) is \(1-\Phi(1.96-1.897)=0.475\). With 120 events, the mean is \(0.30\sqrt{120}/2=1.643\), and the probability is \(1-\Phi(1.96-1.643)=0.376\). These final probabilities are marginal approximations; they do not account for dependence on the intermediate selection statistic.
Planning and Interpreting a Sensitivity Analysis
A defensible analysis changes one assumption at a time to explain its role, then examines selected combinations to assess their joint consequences. Ranges should be justified by prior evidence, measurement properties, clinical expertise, or explicit uncertainty scenarios. For each scenario, retain the actual selection threshold and final-testing rule, and report at least the selection probability and the probability of final success after selection. When useful, also report the joint probability \(P(A\cap F)\). A scenario table makes assumptions and consequences easy to compare.
| Assumption varied | Possible consequence | Useful quantity to recalculate |
|---|---|---|
| Statistic correlation \(\rho\) | Changes how an intermediate result predicts the final statistic | \(P(F\mid A)\) and \(P(A\cap F)\) |
| Surrogacy slope \(b\) or residual variation \(\tau\) | Changes predicted final effects for a given intermediate effect | Predictive final-success probability |
| Measurement attenuation or error variance | Changes the location and/or spread of the selection statistic | \(P(A)\), followed by final success after selection |
| Events available at a look | Changes precision and may change the statistic’s expected signal | Selection probability and final power at the available information |
These approaches answer different questions. A joint-normal calculation is useful when dependence between analysis statistics is the main uncertainty. A treatment-effect prediction model is useful when transfer from intermediate to final effects is central. Directly recalculating the specified analysis under alternative event counts or measurement-error assumptions addresses information and measurement concerns. If probabilities are highly sensitive to an assumption, a tipping-point analysis can report how extreme that assumption must be to change the design conclusion. None of these approaches makes a weakly supported relationship certain.
Important failure modes include varying \(\rho\) while silently holding a treatment-effect surrogacy model fixed, assuming that delay only changes event counts when it may also change who has observable outcomes, and applying an attenuation factor without considering its effect on variance. Another error is to present a conditional probability among selected arms as if it were the probability of selecting and succeeding. If multiple arms compete for selection, the calculations must also reflect that competition and the prespecified multiplicity procedure.
Check Your Understanding
Use the definitions and assumptions in this tutorial. Each solution follows its question.
If the correlation between \(Z_I\) and \(Z_F\) increases, does that necessarily establish stronger treatment-effect surrogacy?
Solution. No. Correlation describes dependence between the two statistics under the specified model. Surrogacy concerns how treatment effects on the intermediate outcome predict treatment effects on the final outcome; it requires its own model, such as a slope and residual variation.
Under the bivariate-normal model, let \(\mu_I=\mu_F=1\), \(Z_I=2\), and \(\rho=0\). What is the conditional probability that \(Z_F>1.96\)?
Solution. The conditional mean is \(1+0(2-1)=1\), and the conditional standard deviation is \(\sqrt{1-0^2}=1\). Thus \(P(Z_F>1.96\mid Z_I=2)=1-\Phi(0.96)\), approximately \(0.1685\).
In the attenuation example, what is the selection probability if \(\lambda=1\), with all other assumptions unchanged?
Solution. The measured-effect mean is \(1(0.40)=0.40\). Thus \(P(Y_I>0.30)=1-\Phi((0.30-0.40)/0.20)=\Phi(0.50)\), approximately \(0.6915\).
For \(\theta_I=0.40\), how many events are needed for the approximate mean statistic to equal \(1.50\) under the equal-allocation formula?
Solution. Solve \(0.40\sqrt{E}/2=1.50\), giving \(\sqrt{E}=7.50\) and \(E=56.25\). Since events are counted in whole numbers, about 56 or 57 events give a mean near the boundary; this is a calculation of the mean statistic, not a guarantee of selection.
Why should a report distinguish \(P(F\mid A)\) from \(P(A\cap F)\)?
Solution. \(P(F\mid A)\) is the chance of final success among scenarios in which the arm passes selection. The joint probability \(P(A\cap F)=P(A)P(F\mid A)\) also accounts for the chance that the arm is selected at all. A high conditional success probability can coexist with a low joint probability if selection is uncommon.