Calibration Diagnostics for Probability-of-Success Models

Why Check Calibration?

A probability-of-success (PoS) model gives a numerical probability for an event such as a development program meeting a prespecified success definition. A model can rank programs well—assigning higher probabilities to programs more likely to succeed—yet systematically overpredict or underpredict their success rates. Calibration concerns agreement between predicted probabilities and observed event frequencies. It is therefore a direct diagnostic when probabilities will inform development planning.

Calibration is meaningful only when predictions and outcomes refer to the same event, population, and time point. For example, a prediction of success through the end of Phase II cannot be assessed against an outcome defined as approval after Phase III. The earlier material on defining success by estimand and endpoint and on predictive probability of trial success provides the context for making these targets explicit.

Calibration differs from discrimination. Discrimination concerns whether successes tend to receive higher predictions than failures; calibration concerns whether the stated probability levels match event frequencies. A model may have one property without the other. Neither is a substitute for checking whether the predictions are relevant to the decision being considered.

A Small Development-Program Dataset

Consider the following entirely hypothetical validation dataset. Ten programs received predictions before their outcomes were known. Five received predicted PoS \(0.20\); of these, one succeeded. Five received predicted PoS \(0.60\); of these, four succeeded. The success definition is assumed to be identical for all ten programs, and predictions are assumed to be genuinely prospective or otherwise obtained without using these outcomes to fit the model.

Grouping is useful for communication, but grouping choices affect the plot. Many narrow groups can leave very few programs per group; a few broad groups can hide local miscalibration. State how groups were formed and show group sizes. Where possible, display individual predictions and outcomes alongside summaries, and show uncertainty around observed proportions. A visual pattern is a diagnostic signal, not a formal guarantee of future performance.

Calibration Intercept and Slope

A common parametric diagnostic models the observed binary outcome using the logit of each prediction. Define \(\operatorname{logit}(p)=\log\{p/(1-p)\}\). The joint calibration model is

Here \(a\) is the calibration intercept and \(b\) is the calibration slope. Perfect calibration under this model corresponds to \(a=0\) and \(b=1\): the fitted success probability then equals \(p_i\). A positive or negative intercept indicates an overall shift on the log-odds scale, conditional on the slope used. A slope below 1 commonly indicates predictions that are too extreme (too close to 0 or 1) in relation to outcomes; a slope above 1 commonly indicates predictions that are not extreme enough. Interpret both together, and do not treat these patterns as proof of a particular cause.

A related quantity, often called the calibration-in-the-large intercept, is estimated with slope fixed at 1. It is the intercept \(c\) in

The joint-model intercept \(a\) and the calibration-in-the-large intercept \(c\) answer different questions. The former is estimated while the slope is also free to change; the latter asks whether a common log-odds shift is needed when the prediction scale is held fixed. Always state which one is reported.

The model’s likelihood also gives a short justification for what the intercept estimates. For independent binary outcomes, the log-likelihood for parameters \(a,b\) is \(\ell=\sum_i[y_i\eta_i-\log\{1+\exp(\eta_i)\}]\), where \(\eta_i=a+b\operatorname{logit}(p_i)\). Differentiating with respect to \(a\) gives

At a fitted maximum, this score is zero, so the fitted probabilities sum to the observed number of successes. This matching of totals does not ensure calibration at every probability level: the slope and the distribution of predictions matter too. The derivation assumes the logistic calibration model and independent Bernoulli outcomes; dependence among programs or model misspecification can make ordinary uncertainty estimates misleading.

Brier Score: Overall Probability Accuracy

The Brier score is the mean squared difference between predicted probabilities and binary outcomes. For \(n\) programs it is

The score ranges from 0 to 1, with lower values indicating smaller average squared probability errors in the evaluated dataset. It reflects both calibration and the ability to distinguish outcomes; it is not a pure calibration measure. Its value depends on the event frequency and on the population being evaluated, so comparisons should use the same outcome definition and preferably the same validation cases.

Assumptions, Diagnostics, and Failure Modes

These diagnostics require outcomes that are defined consistently and predictions recorded before observing those outcomes, or evaluated using a valid design that prevents information leakage. The logistic calibration model additionally assumes that the log-odds of the outcome are adequately described by a straight line in \(\operatorname{logit}(p_i)\). The simple likelihood derivation and usual model-based standard errors assume independent outcomes. Development programs may share mechanisms, sponsors, trial designs, or historical data; such dependence reduces the effective information and can invalidate standard uncertainty calculations.

• Check data alignment. Confirm the predicted probability, endpoint, success criterion, and assessment horizon match for every program. Missing outcomes or changed definitions can distort the comparison.
• Inspect coverage. Note the range and distribution of predictions. A dataset concentrated near \(0.5\) says little about calibration near \(0.05\) or \(0.95\).
• Assess uncertainty. Small samples and few successes or failures produce unstable estimates. Logistic fits can fail or become extreme when outcomes are separated by prediction values. Report uncertainty and the number of programs, not just point estimates.
• Check functional form. A single intercept and slope summarize a linear relationship on the logit scale. Curvature or subgroup differences can remain hidden; observed-versus-predicted summaries can help reveal patterns but are themselves noisy.
• Check dependence and reuse. If the same evidence informed both model construction and evaluation, apparent calibration may be optimistic. Related programs may also require validation and uncertainty methods that account for their dependence.

No single diagnostic settles model adequacy. A plot is intuitive but sensitive to grouping and sample size. Intercept and slope offer compact summaries but rely on a parametric form and may be unstable. The Brier score uses individual predictions and outcomes but combines multiple aspects of probability accuracy into one number. Use the diagnostics together, explain their limitations, and avoid interpreting small-sample differences as definitive evidence.

Using Calibration Evidence Responsibly

Calibration evidence is most useful when its scope is explicit: which programs were assessed, how success was defined, when predictions were made, and whether those programs represent the future decisions of interest. A model may be acceptably calibrated in one development setting but not transport to a different disease area, trial design, or evidence base. Recalibration can be considered when a persistent discrepancy is supported, but estimating adjustments and judging their performance on the same small dataset risks overfitting.

For a practical report, give the observed-versus-predicted display with group sizes, the calibration intercept and slope with their definitions and uncertainty, and the Brier score alongside a clearly specified comparison. State limitations that affect interpretation, including small sample size, outcome dependence, restricted prediction range, and any reuse of development data. These are diagnostics of probability performance, not guarantees about the outcome of an individual program.