Posterior Predictive Checks For Joint Dose Models

Why Check Predictions From a Joint Dose Model?

A joint dose model compresses observed patient outcomes into a probability model that supports inference across doses. A model can nevertheless produce a plausible estimate of benefit or toxicity while failing to represent some feature of the data: for example, how often efficacy and toxicity occur together, whether efficacy rises sharply only at the highest dose, or whether an unusual number of severe events is plausible. Posterior predictive checks ask a direct question: if the fitted model generated another data set like this one, would important features of that data look like the outcomes we observed?

The check is especially useful when outcomes are joint. Modeling efficacy and toxicity separately can miss their association, even if each marginal rate looks reasonable. Posterior predictive checks do not establish that a model is true or select a dose. They help identify aspects of model fit that deserve scrutiny before model-based dose conclusions are relied upon.

Definition: Let $y$ denote the observed joint outcomes, $\theta$ the model parameters, and $y^{rep}$ a replicated data set. The posterior predictive distribution is $$ p(y^{rep}\mid y)=\int p(y^{rep}\mid \theta)\,p(\theta\mid y)\,d\theta. $$ It describes replicated outcomes by first drawing parameters from their posterior distribution given the observed data, then drawing outcomes from the model at those parameter values.

For a dose-finding model, replication should normally preserve the observed dose assignments and any covariates whose distribution is being held fixed. Thus, a replicated patient at dose $d$ is generated from the model’s joint outcome distribution for that dose and, where applicable, that covariate profile. The joint likelihood and dose structure are specified as in the earlier material on likelihood construction for joint dose models. Here the emphasis is on checking the consequences of that model, not choosing its form.

What Counts as a Discrepancy?

A check starts by choosing a discrepancy statistic $T(y)$: a summary that would be important if poorly reproduced. Compare its observed value $T(y)$ with values $T(y^{rep})$ from posterior predictive replicates. The statistic may target the full pair distribution, a trend across doses, or a clinically important tail. It should be chosen before interpreting its result and should reflect the scientific concern, rather than being selected only because it makes the model look poor or good.

For binary efficacy $E$ and binary toxicity $T$, the joint pair has four categories: neither event, efficacy without toxicity, toxicity without efficacy, and both events. The marginal efficacy probability alone cannot tell whether efficacy tends to occur in patients who also experience toxicity. A useful check can compare observed and replicated counts in all four categories, either at each dose or across doses.

If $Y_{dk}$ is the number of patients at dose $d$ in joint category $k$, and the model assigns category probability $\pi_{dk}$, then conditional on the parameters the expected replicated count is $n_d\pi_{dk}$, where $n_d$ is the number observed at that dose. Averaging over posterior uncertainty gives

E(Y_{dk}^{rep}\mid y)=n_d E(\pi_{dk}\mid y).

This identity follows from iterated expectation: first, conditional on $\theta$, a replicated multinomial count has mean $n_d\pi_{dk}$; then average that conditional mean over $p(\theta\mid y)$. It explains why posterior mean fitted counts are useful summaries, but not sufficient checks by themselves: the distribution of replicated counts also matters.

Worked Example: Checking the Joint Pair Distribution at One Dose

Suppose, hypothetically, that 20 evaluable patients received dose 2. Their observed counts are 8 with neither event, 6 with efficacy only, 3 with toxicity only, and 3 with both. For an intentionally simple illustration, suppose the dose-specific category probabilities have a Dirichlet prior with parameters $(1,1,1,1)$. This saturated model does not borrow information across doses; it is used here to make the predictive calculation transparent.

The posterior parameters are the prior parameters plus the observed category counts: $(9,7,4,4)$, with total 24. The posterior mean probabilities are therefore $9/24=0.3750$, $7/24=0.2917$, $4/24=0.1667$, and $4/24=0.1667$. For example, the posterior predictive expected count in the efficacy-only category is

E(Y_{\text{efficacy only}}^{rep}\mid y) =20\left(\frac{7}{24}\right) =\frac{140}{24} =5.833.

The corresponding expected counts for the four categories are 7.500, 5.833, 3.333, and 3.333. The observed counts, 8, 6, 3, and 3, are close to those expectations. The marginal posterior mean efficacy probability is $(7+4)/24=0.4583$, while the toxicity probability is $(4+4)/24=0.3333$. These margins alone do not replace checking the four categories: the count with both outcomes is specifically relevant to their joint pattern.

A complete check would draw posterior probability vectors from the Dirichlet posterior, generate replicated multinomial counts of size 20, and compare a chosen discrepancy—such as the largest absolute difference between observed and expected category counts—with its replicated distribution. Similar expected counts are reassuring descriptively, but they do not prove that the joint distribution is adequate.

Checking Dose Trends

A model may fit pooled outcome counts yet miss the way outcomes change with dose. Dose-trend checks compare a trend statistic calculated from the observed data with the same statistic calculated from every replicated data set. Suitable summaries include the observed rate at each dose, the difference between adjacent-dose rates, or a regression slope of a rate against a prespecified dose score. For binary outcomes, the rate at dose $d$ is the number with the outcome divided by $n_d$.

The statistic should match the model feature of interest. If monotone efficacy is scientifically expected, a statistic based on adjacent increases may be informative. If the concern is an abrupt increase in toxicity, a highest-dose rate or maximum adjacent increase may be more revealing than a single overall slope. With few dose levels, a slope summarizes a pattern but can conceal curvature or a local reversal. Replicates should use the same dose scores and calculation as the observed data.

Worked Example: Comparing an Observed Efficacy Trend With Replicates

Consider hypothetical efficacy counts of 1, 4, and 9 among 20 patients at dose scores 0, 1, and 2. The observed rates are $0.05$, $0.20$, and $0.45$. The successive increases are $0.20-0.05=0.15$ and $0.45-0.20=0.25$.

The ordinary least-squares slope across the three points is a separate summary. The mean dose score is 1, and the mean observed rate is $(0.05+0.20+0.45)/3=0.2333$. Thus

\hat{\beta} =\frac{\sum_d(x_d-\bar{x})(\hat{p}_d-\bar{p})} {\sum_d(x_d-\bar{x})^2} =\frac{(-1)(-0.1833)+(0)(-0.0333)+(1)(0.2167)}{(-1)^2+0^2+1^2} =\frac{0.4000}{2} =0.2000.

The fitted OLS trend has slope 0.20, or 20 percentage points per one-unit increase in dose score. The observed rates increase by 0.15 and then 0.25; they do not increase by 0.20 at each step.

Suppose, for illustration, that 1,000 posterior predictive replicates from a fitted joint dose model yield an efficacy-slope statistic at least as large as 0.200 in 426 replicates. The estimated upper-tail posterior predictive proportion is $426/1000=0.426$. The observed slope is not unusual relative to these simulated slopes. This finding is a check on that trend statistic, not evidence that every aspect of the joint model fits. The replicate output is hypothetical and must come from the model actually being evaluated.

Checking Clinically Important Tails

A model can match average rates and still poorly reproduce rare but consequential outcomes. A tail check focuses on a clinically meaningful event or threshold, such as the number of severe toxicities at a dose, the number of patients experiencing both efficacy and severe toxicity, or whether any dose has an excessive event count. The event definition and threshold should be clinically justified and fixed independently of the observed discrepancy where possible.

For a statistic $T$, a one-sided posterior predictive proportion can be estimated as

\widehat{P}_{pp}\{T(y^{rep})\geq T(y)\mid y\} =\frac{1}{M}\sum_{m=1}^{M} I\{T(y^{rep,m})\geq T(y)\},

where $M$ is the number of replicated data sets and $I\{\cdot\}$ is 1 when the condition holds and 0 otherwise. For an unusual low count, the direction can be reversed; for a two-sided concern, define a discrepancy that measures distance from the model’s expected pattern. A posterior predictive proportion near 0 or 1 signals that the observed statistic lies toward a tail of its replicated distribution. It is not a frequentist p-value, and it is not the probability that the model is true.

Worked Example: A Joint-Event Tail Check

Suppose, hypothetically, that at a dose with 20 patients, 6 experienced both efficacy and severe toxicity. The protocol team has identified 8 or more such joint outcomes as a clinically concerning count for a cohort of this size. Define $T(y)$ as the count with both events, and check whether the fitted joint model frequently generates $T(y^{rep})\geq 8$.

Assume 1,000 posterior predictive replicates produce counts of at least 8 in 72 replicates. The estimated predictive tail probability for the concerning threshold is $72/1000=0.0720$. The observed count of 6 is below the stated threshold, and this model generates threshold-level counts in about 7.2% of these hypothetical replicates. That probability is a model-based predictive frequency conditional on the observed data and the chosen model; it is not a guarantee about the next cohort.

The check should also inspect the distribution of counts below 8 and any relevant dose pattern. A low frequency of threshold exceedance alone could be misleading if the model systematically underpredicts joint events at other doses or covariate profiles.

Assumptions, Diagnostics, and Alternatives

A posterior predictive check is only as meaningful as the model and replication scheme behind it. The observed outcomes must correspond to the outcome definitions used by the likelihood, and the model must specify a valid joint distribution at the doses being checked. The posterior calculation must adequately represent uncertainty in the parameters. Replication should preserve relevant design features, especially the doses assigned; if covariate-specific predictions are being checked, retain the corresponding covariate values or make the intended covariate-generating assumptions explicit.

In an adaptive trial, doses were assigned using information available during the trial. For a conditional outcome-model check, it is usually appropriate to hold the realized dose assignments fixed and regenerate outcomes at those doses. Re-running the allocation process answers a different, design-level question because it also regenerates dose assignments. The allocation process should not be silently mixed into a check intended to assess the outcome model.

Use several focused discrepancies rather than one omnibus number. Useful diagnostics include:

Observed and replicated counts in all joint outcome categories, by dose.
Observed and replicated marginal efficacy and toxicity rates, to locate which outcome contributes to a joint discrepancy.
Adjacent-dose changes, overall slopes, and extreme dose-specific rates, selected to reveal distinct trend features.
Counts of clinically important events and joint events, especially where small probabilities make averages uninformative.

Posterior predictive checks use the data both to estimate the posterior and to judge replicated outcomes. This reuse can make checks conservative: a model may look adequate because parameters were adjusted to the same data being checked. A small posterior predictive tail proportion is a warning to investigate, not a formal calibrated rejection probability. Conversely, a non-extreme proportion does not prove adequacy; a statistic that ignores the model’s actual weakness may have little ability to detect it.

Prior predictive checks answer a different question: before observing the trial data, do parameter values and outcomes implied by the prior look plausible? Posterior predictive checks ask whether the model, after updating on the observed data, reproduces selected features. Parameter credible intervals summarize uncertainty about model parameters; they do not establish that the model reproduces the observed joint outcomes. These approaches complement one another rather than replace one another.

Warning signs include systematic discrepancies across adjacent doses, poor reproduction of the both-events category despite reasonable marginal rates, or repeated underprediction of a clinically important tail. Also examine whether conclusions depend on the statistic or dose subset selected. Such findings motivate checking outcome definitions, data handling, and model assumptions. A posterior predictive check alone does not identify which component is responsible, nor does it quantify the operating characteristics of a dose-finding design.

Common Mistake

Common mistake: “The observed efficacy rate is close to its posterior mean, so the joint model fits.” This conclusion is too strong. A close marginal rate says little about whether the model reproduces toxicity, the association between efficacy and toxicity, dose-specific patterns, or clinically important tails. Check the joint categories and prespecified trend and tail statistics using replicated data, and interpret each result as evidence about that feature only.

Key takeaway: A posterior predictive check compares an observed feature of joint outcomes with the distribution of the same feature under replicated data generated from the posterior. Its value depends on choosing clinically relevant discrepancies, reproducing the trial’s relevant conditions, and treating results as targeted diagnostics rather than proof of model adequacy.

Check Your Understanding

Answer each question, then compare with the solutions below.

What is the difference between a posterior predictive replicate and a new draw of a parameter from the posterior?
At one dose, observed efficacy and toxicity rates appear well reproduced, but the observed both-events count is much larger than typical replicated counts. What aspect of fit is concerning?
Observed efficacy rates at three equally spaced dose scores are 0.10, 0.30, and 0.40. What are the adjacent increases, and why should they not be replaced by the fitted slope?
In 2,000 replicates, 118 produce at least as many severe events as observed. Calculate and interpret the estimated upper-tail predictive proportion.
Why might a conditional outcome-model check hold observed dose assignments fixed in an adaptive trial?

Solutions. (1) A parameter draw is a value of $\theta$ from $p(\theta\mid y)$; a posterior predictive replicate additionally draws outcomes from $p(y^{rep}\mid\theta)$. (2) The joint association or frequency of the both-events category may be poorly reproduced even though marginal rates look reasonable. (3) The increases are $0.30-0.10=0.20$ and $0.40-0.30=0.10$. A fitted slope is an overall linear summary, not either adjacent increment. (4) $118/2000=0.0590$, so 5.90% of these replicates have at least the observed severe-event count; this is a model-based predictive frequency, not the probability the model is true. (5) Holding doses fixed isolates how well the outcome model reproduces outcomes at the doses actually assigned; regenerating assignments would also assess the adaptive allocation process.