Bayesian Updating of Program Success Probabilities

Why Update a Program Success Probability?

A development team may begin with a probability that a program will meet a specified technical success criterion. New evidence, such as a trial result, can make success more or less plausible. Bayesian updating provides a coherent calculation: it combines the probability assigned before observing the evidence with how likely that evidence is under success and non-success.

The result is conditional on the success event, the evidence event, and the probabilities used in the calculation. It is not a guarantee that the program will succeed, and it does not by itself determine whether the program should continue. The focus here is the probability update.

For a well-defined update, \(S\) and \(\neg S\) must be mutually exclusive and exhaustive, and \(E\) must be specified precisely enough that its likelihood can be assessed under both states. For example, “the trial was positive” needs a defined endpoint and decision rule; an ambiguous description can lead to inconsistent likelihoods.

Bayes’ Theorem and the Odds Form

When \(P(E)>0\), Bayes’ theorem gives the posterior probability directly. Since \(S\) and \(\neg S\) partition the possible states, the law of total probability gives \(P(E)=P(E\mid S)P(S)+P(E\mid\neg S)P(\neg S)\). Substituting this expression into the definition of conditional probability yields:

This expression shows the two contributions to the probability of observing \(E\): evidence generated in the success state and evidence generated in the non-success state. The posterior is the success-state contribution divided by their sum.

When the prior is strictly between zero and one, odds provide an equivalent way to express the update. Prior odds are \(P(S)/P(\neg S)\); posterior odds are \(P(S\mid E)/P(\neg S\mid E)\). If \(P(E\mid\neg S)>0\), the likelihood ratio, also called the Bayes factor for \(E\) in favor of \(S\), is defined by the following ratio. Under these conditions, the posterior-odds identity follows by dividing the posterior probability by its complement.

The likelihood ratio compares how well the evidence is predicted by success versus non-success. A value above one shifts the odds toward success; a value below one shifts them toward non-success; a value of one leaves the odds unchanged. The size of the shift depends on both the likelihood ratio and the prior odds.

Worked Updates in Development Settings

A Short Derivation of the Odds Identity

The direct Bayes expression can be written alongside the probability of non-success after observing the same evidence:

When both quantities are defined and \(P(E\mid\neg S)>0\), divide the first equation by the second. The common factor \(P(E)\) cancels, leaving:

This derivation explains why evidence multiplies odds rather than probabilities. It also identifies the condition that matters for the denominator of the likelihood ratio. The direct probability form should be used to examine boundary cases instead of cancelling or dividing by a term that may be zero.

Using Evidence Carefully

A Bayesian update is only as meaningful as its inputs. In a development setting, the prior, the state definition, and the evidence likelihoods must all refer to the same program and success criterion. For example, likelihoods for one endpoint or analysis rule cannot automatically be used to update a prior defined for a different technical outcome.

When evidence arrives in stages, a new update can use likelihoods conditional on the evidence already observed. For two items \(E_1\) and \(E_2\), the relevant second-stage likelihoods are \(P(E_2\mid S,E_1)\) and \(P(E_2\mid\neg S,E_1)\). Multiplying two likelihood ratios based on marginal probabilities is generally unjustified unless the assumptions make that multiplication valid. In particular, conditional independence of the evidence items within each state would permit the product; without it, this shortcut can count the same information more than once.

Useful checks before reporting an update include:

• Confirm that \(S\) and \(\neg S\) cover the success outcome of interest, rather than mixing different endpoints or definitions.
• Check that the evidence event is stated explicitly and that each conditional probability refers to that same event.
• Verify \(0\leq P(E\mid S)\leq 1\), \(0\leq P(E\mid\neg S)\leq 1\), and that the prior lies between zero and one.
• Calculate \(P(E)\) and confirm it is positive before interpreting \(P(S\mid E)\).
• Check that evidence shifts the probability in the direction implied by its likelihood ratio, and that the posterior remains between zero and one.

Common Pitfalls and Limitations

• Multiplying the prior probability by the likelihood ratio. This generally gives the wrong answer. The likelihood ratio multiplies prior odds, which must then be converted back to a probability.
• Calling the posterior an unconditional success rate. \(P(S\mid E)\) is conditional on the specified evidence and assumptions. It is not automatically the probability for a different program or a different evidence pattern.
• Using an undefined likelihood ratio. If \(P(E\mid\neg S)=0\), the ordinary ratio is undefined, not a finite number. Check the direct Bayes formula; when the evidence has positive probability only under success, the posterior is one.
• Treating uncertain inputs as facts. A calculated posterior can be numerically precise while the prior or likelihoods are uncertain. It is appropriate to examine how the result changes across plausible inputs rather than imply more certainty than the inputs support.
• Counting correlated evidence twice. Combining evidence requires the correct conditional likelihoods. Reusing information in multiple updates without accounting for dependence can overstate the evidence for or against success.

Bayesian updating is useful because it makes transparent how a prior belief and a specified observation combine. It does not establish that the prior or likelihood assumptions are correct, resolve uncertainty about how evidence was generated, or replace a decision analysis. The conclusion should therefore report the posterior together with the evidence definition and key probabilities that produced it.