Why Numerical Verification Matters
A Bayesian escalation program can produce a plausible-looking dose recommendation even when its likelihood, prior, dose indexing, or decision rule is implemented incorrectly. Numerical verification asks a narrower question than whether the model is clinically appropriate: does the software calculate the quantities specified by the model and apply the stated rule correctly? The distinction matters. A correct calculation cannot repair a poorly chosen model, while a suitable model is not useful if its computations are wrong.
A reliable check uses test cases whose answers can be obtained independently. Exact conjugate calculations are useful for validating posterior probabilities and predictions; small simulated examples can test Monte Carlo output; and deliberately simple boundary datasets can reveal errors that ordinary trial-like data may conceal. Checks should examine intermediate quantities, not only the final action.
A Conjugate Reference Calculation
Consider a single dose with an unknown toxicity probability \(p\). For a transparent test fixture, assume each participant has a binary toxicity outcome, outcomes are conditionally independent given \(p\), and all participants have completed the same assessment window. Let \(Y\) be the number of toxicities among \(n\) participants. Use a \(\mathrm{Beta}(a,b)\) prior, with \(a>0\) and \(b>0\). This is a deliberately simple reference model, not a claim that dose levels in a clinical trial should be modeled independently.
Under these assumptions, \(Y\mid p\) is binomial and the posterior distribution is \(\mathrm{Beta}(a+Y,b+n-Y)\). The result follows by multiplying the prior density, proportional to \(p^{a-1}(1-p)^{b-1}\), by the likelihood, proportional to \(p^Y(1-p)^{n-Y}\), and collecting powers of \(p\) and \(1-p\).
A correct implementation should reproduce this posterior for a test case, regardless of whether the production model is more complex. For a dose-toxicity model with a shared parameter or a monotonicity constraint, the same exact answer will not generally apply; the reference fixture should be chosen to match the calculations being checked.
Worked Example: Checking a Posterior Toxicity Probability
Suppose a hypothetical test fixture uses a uniform prior, \(p\sim\mathrm{Beta}(1,1)\), and observes 1 toxicity among 3 fully assessed participants at a dose. Verify the posterior probability that \(p\) exceeds a toxicity threshold of 0.30.
The posterior is \(\mathrm{Beta}(2,3)\). Its density is \(12p(1-p)^2\), so its cumulative distribution function is \(6p^2-8p^3+3p^4\). Therefore,
A software result near 0.6517 is a useful check of the posterior calculation and tail-probability convention. A result near 0.3483 suggests that the code may have returned \(P(p\leq0.30\mid Y)\) instead. This reference calculation checks only the stated fixture; it does not validate other prior specifications or dose-response structures.
Posterior Predictive Checks as Software Tests
A posterior predictive distribution describes outcomes in a new group under the fitted posterior model. In this fixture, let \(K\) be the number of toxicities in a future group of \(m\) participants at the same dose. Conditional on \(p\), \(K\) is binomial. Averaging over the posterior uncertainty in \(p\) gives a beta-binomial distribution:
This expression follows by integrating the conditional binomial probability against the posterior density of \(p\). The probabilities must be nonnegative and sum to one. The posterior predictive mean is \(m(a+Y)/(a+b+n)\). These provide checks independent of the software's own simulation output.
Worked Example: Verifying a Predictive Distribution
Use the previous hypothetical data, so the posterior is \(\mathrm{Beta}(2,3)\), and predict toxicities in a future group of 2 participants. Calculate each probability directly.
For \(K=0\), integrating \((1-p)^2\) against the \(\mathrm{Beta}(2,3)\) posterior gives \(3\cdot4/(5\cdot6)=0.4000\). For \(K=1\), the probability is \(2(2)(3)/(5\cdot6)=0.4000\). For \(K=2\), it is \(2\cdot3/(5\cdot6)=0.2000\). Thus the distribution is \((0.4000,0.4000,0.2000)\), in order \(k=0,1,2\), and its probabilities sum to 1.
The expected count is \(2(2/5)=0.8000\), also obtained from \(0(0.4)+1(0.4)+2(0.2)=0.8000\). A predictive routine that returns probabilities with a different sum, or a mean inconsistent with this value, should be investigated before trusting its simulated predictions.
A posterior predictive comparison can also help check data-flow and replicate generation. For example, with the original cohort size \(m=3\), the probabilities for \(K=0,1,2,3\) are \(60/210,72/210,54/210,24/210\), respectively. They sum to one, and the expected replicated count is \(3(2/5)=1.2\). The observed count of 1 is not surprising under this reference distribution: \(P(K\geq1\mid Y)=1-60/210=0.7143\). This is a computational consistency check, not proof that the toxicity model fits clinical reality.
Monte Carlo Convergence and Decision Stability
Complex escalation models may require numerical integration or Markov chain Monte Carlo (MCMC), rather than a closed-form posterior. In those cases, test both the computation and the decision it feeds. For MCMC, inspect multiple chains started from dispersed initial values, trace plots for persistent trends or poor mixing, and autocorrelation. The potential scale reduction statistic \(\hat{R}\) compares between-chain and within-chain variation; values near 1 are reassuring, but do not by themselves establish convergence. Effective sample size (ESS) accounts for autocorrelation and is more informative than the raw number of draws when estimating posterior probabilities.
For a posterior probability estimated as \(\hat{q}\) from \(N\) effectively independent draws, an approximate Monte Carlo standard error is \(\sqrt{\hat{q}(1-\hat{q})/N}\). For correlated MCMC draws, use an ESS-based estimate rather than treating every draw as independent. A small Monte Carlo error is important when a decision threshold is close: accurate estimation of a probability that is far from the cutoff may be sufficient even when a more precise estimate is needed near it.
Worked Example: Checking a Simulation Estimate Near a Cutoff
Suppose a hypothetical MCMC run estimates \(P(p>0.30\mid Y)\) as 0.654 from an effective sample size of 10,000. The exact reference value from the earlier fixture is 0.6517. Check the simulation precision and the consequence for an illustrative rule that permits escalation when this probability is below 0.70.
Using the ESS-based approximation, the Monte Carlo standard error is
The difference between the simulated and exact probabilities is \(0.654-0.6517=0.0023\), about \(0.48\) Monte Carlo standard errors. That discrepancy is consistent with ordinary simulation variation. Both values are below 0.70, so this illustrative decision is unchanged. If the cutoff were 0.653, the two values would fall on opposite sides of it; the computation would need greater precision or an exact reference before making a stable decision. The cutoff here is only a test rule, not clinical guidance.
Boundary Cases, Diagnostics, and Alternatives
Boundary-case datasets are purposeful tests, not realistic trial scenarios. For the beta-binomial fixture with a \(\mathrm{Beta}(1,1)\) prior, no participants at a dose must leave the posterior unchanged. If all \(n\) outcomes are non-toxic, the posterior is \(\mathrm{Beta}(1,n+1)\); if all \(n\) are toxic, it is \(\mathrm{Beta}(n+1,1)\). For example, with \(n=4\), these become \(\mathrm{Beta}(1,5)\) and \(\mathrm{Beta}(5,1)\). The first concentrates toward low toxicity probabilities and the second toward high values. These checks can expose reversed outcome coding, incorrect cohort counts, or an indexing error.
Additional tests should include probabilities very near 0 or 1, an empty dose-level record, and inputs that should be rejected, such as negative cohort sizes or toxicity counts greater than the number assessed. Check that probabilities remain finite, that posterior predictive probabilities sum to one within a stated numerical tolerance, and that dose labels map to the intended model inputs. In delayed-outcome settings, separately test whether pending observations are handled according to the specified likelihood and follow-up rule; do not silently treat incomplete outcomes as completed non-toxicities.
Compare methods where possible. Exact conjugate calculations give strong reference values for simple fixtures. Numerical quadrature provides an independent integration check for low-dimensional models. MCMC can handle more complex posterior distributions, but requires convergence and Monte Carlo error assessment. A second software implementation can expose coding differences, although two implementations that share the same mistaken model specification may agree and still be wrong. Testing each intermediate quantity—likelihood contribution, posterior summary, predictive probability, and decision input—helps localize discrepancies.
Diagnostics are signals to investigate, not automatic guarantees. A stable trace plot can miss a chain trapped in one mode; a large ESS does not check whether toxicity outcomes were coded correctly; and matching one reference probability does not validate every branch of the decision logic. Conversely, small numerical differences may be harmless if they are within an understood tolerance and cannot change the action. The relevant tolerance should be tied to numerical accuracy and decision sensitivity, rather than selected after seeing an inconvenient result.
State the prior, likelihood, observed outcomes, dose indexing, and decision rule precisely.
Use exact algebra where available, or a separately implemented numerical method.
Check posterior parameters or summaries, predictive probabilities, and Monte Carlo diagnostics before checking the final action.
Use no-event, all-event, empty, and invalid inputs, then assess whether small numerical changes can cross a decision cutoff.
Common Mistake
A common mistake is to treat a posterior predictive check as proof that the model is correct because simulated outcomes resemble the observed outcomes. The wrong reasoning is: “The observed count is plausible under simulations, so the model is validated.” The correction is to separate two questions. A predictive calculation can verify that the program generates outcomes from its stated posterior model, and a comparison can reveal discrepancies worth investigating. Neither establishes that the model's assumptions describe the trial or that the decision rule is appropriate. Model adequacy and numerical implementation require distinct evidence.
Check Your Understanding
Answer each question, then compare with the solutions below.
- With a \(\mathrm{Beta}(1,1)\) prior and 0 toxicities among 4 fully assessed participants, what posterior should a test fixture return?
- For a future group of 2 participants and posterior \(\mathrm{Beta}(2,3)\), what should the predictive mean toxicity count be?
- Why is an MCMC estimate of 0.654 not automatically a failure when the exact reference value is 0.6517?
- What does a posterior predictive check establish, and what does it not establish?
Solutions. (1) The posterior is \(\mathrm{Beta}(1,5)\), by adding the observed toxicity count to the first prior parameter and the four non-toxic outcomes to the second. (2) The mean is \(2(2/(2+3))=0.8000\) toxicities. (3) The difference is 0.0023, smaller than the approximate Monte Carlo standard error of 0.00476 for the stated effective sample size, so it is compatible with simulation variation. (4) It can check the predictive computation and help identify discrepancies between observed and replicated data; it does not prove that the model assumptions are scientifically adequate or that the decision rule is appropriate.