Conflict Evidence from Predictive Densities

Why Predictive Density Is Relevant to Historical Borrowing

When a clinical trial borrows information from historical data, the current observations can be compared with what the historical-data-informed model predicts. If the observed data are difficult to reconcile with that predictive distribution, the model may be indicating prior-data conflict. A predictive density provides a model-based measure of how much predictive support is assigned near the observed data.

This assessment is useful, but it requires careful interpretation. For discrete data, the predictive probability of the observed outcome is a probability mass. For continuous data, the predictive density at an exact observation is not a probability. Its size depends on the measurement scale and the dimension of the observation, so a small value alone does not establish that the historical information is incompatible with the current trial.

Definition: Let $D_H$ denote historical data, let $\theta$ be a model parameter, and let $Y$ denote current-trial data. If the historical-data-informed posterior is $\pi_H(\theta\mid D_H)$, and the current-data sampling model has density or probability mass function $f(y\mid\theta)$, the posterior predictive density or mass is $$ m_H(y\mid D_H)=\int f(y\mid\theta)\pi_H(\theta\mid D_H)\,d\theta. $$ It averages the probability model for current data over uncertainty about $\theta$ after observing $D_H$.

Assumptions and Interpretation

This calculation relies on a specified statistical model. The historical data are used to form $\pi_H(\theta\mid D_H)$; the current data are then assumed to follow $f(y\mid\theta)$ conditional on $\theta$. The likelihood and parameter must describe a sufficiently comparable population, endpoint, measurement process, and study design for this predictive comparison to be meaningful. If these assumptions do not hold, the calculation can still be performed, but its interpretation as evidence of conflict with relevant historical information is weakened.

The posterior predictive distribution incorporates uncertainty in $\theta$, rather than treating an estimate from the historical data as known. In particular, it is generally wider than the sampling distribution evaluated at a single fixed estimate. For a discrete outcome, $m_H(y\mid D_H)$ is the probability of that exact outcome. For a continuous outcome, it is a density evaluated at $y$, measured in inverse units of the outcome. Probabilities arise by integrating the density over a region.

The predictive density also depends on what is defined as the data. A density for a trial-level mean is not the same as a joint density for every participant’s outcome. Joint densities can become numerically small as the dimension grows, even when each observation is reasonably compatible with the model. The data summary, model, and scale therefore need to be stated when reporting a density.

Worked Examples

Worked Example: A Beta-Binomial Predictive Probability

Suppose, hypothetically, that historical response data lead to a posterior response probability $p\mid D_H\sim\operatorname{Beta}(8,12)$. In a current trial of 10 patients, let $Y$ be the number of responses. Assume conditional on $p$, the current responses are independent Bernoulli outcomes with common probability $p$, so $Y\mid p\sim\operatorname{Binomial}(10,p)$. These assumptions define the predictive calculation; they do not assert that the hypothetical historical and current populations are in fact exchangeable.

Integrating the binomial probability over the beta posterior gives the beta-binomial predictive mass:

P(Y=y\mid D_H) ={10\choose y}\frac{B(8+y,12+10-y)}{B(8,12)}.

For exactly nine responses,

P(Y=9\mid D_H) =10\frac{B(17,13)}{B(8,12)} \approx 0.006854.

For ten responses,

P(Y=10\mid D_H) =\frac{B(18,12)}{B(8,12)} \approx 0.0009709.

Thus the predictive probability of nine or ten responses is $0.006854+0.0009709=0.007825$, or about $0.7825\%$. Under this specified historical model, that upper-tail outcome is unusual. This is a probability for the event $Y\geq 9$, not a probability that the historical model is false. The tail probability is also not the same quantity as the predictive mass of the single observed value $Y=9$.

Worked Example: A Continuous Predictive Density and a Robust Mixture

Consider a hypothetical current-trial sample mean $\bar{x}$, measured in a continuous outcome’s units. Under the historical component, suppose the posterior for the population mean is $\theta\mid D_H\sim N(0.10,0.10^2)$, and the sampling model is $\bar{x}\mid\theta\sim N(\theta,0.10^2)$. Assume the current observations are conditionally independent under this normal sampling model and that the variances are known for this illustration. The predictive distribution is normal with mean $0.10$ and variance $0.10^2+0.10^2=0.0200$.

Suppose the observed sample mean is $0.50$. The historical-component predictive density is

m_H(0.50\mid D_H) =\frac{1}{\sqrt{2\pi(0.0200)}} \exp\left\{-\frac{(0.50-0.10)^2}{2(0.0200)}\right\} \approx 0.05167.

This density is in inverse units of the sample mean. It is not a $5.167\%$ probability. For comparison, suppose a protective component gives $\theta\sim N(0,0.50^2)$ and uses the same sampling variance $0.10^2$. Its predictive variance is $0.2600$, and its density at $0.50$ is

m_0(0.50) =\frac{1}{\sqrt{2\pi(0.2600)}} \exp\left\{-\frac{(0.50-0)^2}{2(0.2600)}\right\} \approx 0.4838.

For a robust mixture with historical-component weight $0.80$ and protective-component weight $0.20$, the predictive density is the same weighted average of the component predictive densities:

m(0.50) =0.80(0.05167)+0.20(0.4838) \approx 0.1381.

The observation has greater density under the protective component than under the historical component, and the mixture assigns more predictive density to it than the historical component alone. This comparison concerns predictive support for this observation under these specified models. It does not by itself establish why the observation differs, or which component is scientifically correct.

Worked Example: Why the Units Change a Density

Use the historical predictive distribution in the preceding example, $X\sim N(0.10,0.0200)$, where $X$ is measured in the original outcome units. At $x=0.50$, its density is approximately $0.05167$ per original unit. Now report the same measurement in units 100 times as large, defining $Z=100X$. The event $X$ lies between $0.49$ and $0.51$ is exactly the event $Z$ lies between $49$ and $51$.

The density transformation is $m_Z(z)=m_X(z/100)/100$. Consequently, the density at $z=50$ is $0.05167/100=0.0005167$ per rescaled unit. Yet the probability of the corresponding interval is unchanged: approximately $0.05167(0.02)=0.001033$ under the original scale, and approximately $0.0005167(2)=0.001033$ under the rescaled scale. The point density changed by a factor of 100 while the probability of the same outcomes did not. A density’s numerical magnitude therefore cannot be interpreted without its units and scale.

Why the Predictive Formula Holds

The key justification is the law of total probability. Consider a small region $A$ of possible current outcomes. Conditional on $\theta$, its probability is obtained from the sampling model. Averaging that probability over the historical posterior gives the predictive probability:

P(Y\in A\mid D_H) =\int P(Y\in A\mid\theta)\pi_H(\theta\mid D_H)\,d\theta =\int_A\left\{\int f(y\mid\theta)\pi_H(\theta\mid D_H)\,d\theta\right\}dy.

The expression in braces is $m_H(y\mid D_H)$, so integrating the predictive density over $A$ gives the probability of $A$. For a discrete outcome, the integral over outcome values is replaced by a sum, giving the predictive probability mass. This also explains why parameter uncertainty is included: each possible $\theta$ contributes according to its posterior probability given the historical data.

For a finite mixture with components indexed by $j$, weights $w_j$ that are nonnegative and sum to one, and component predictive densities $m_j(y)$, the mixture predictive density is

m(y)=\sum_j w_jm_j(y).

This follows by integrating the mixture model component by component. It is a weighted average of predictive densities, not a choice of whichever component gives the largest density. The mixture weights and component definitions must be specified as part of the model.

How to Use the Evidence—and Its Limits

A low predictive probability for a discrete observed outcome, or low predictive density near a continuous observation relative to other plausible observations, can flag that the current data are not well anticipated by the historical-data-informed model. It is most informative when interpreted in context: identify the observation or region being assessed, state the predictive model, and compare with scientifically relevant alternatives or benchmarks. A small value can motivate examination of the model’s assumptions and the comparability of historical and current data.

There is no universal threshold at which a continuous density becomes “small.” Its value depends on measurement units, the chosen data representation, and the number of outcomes being modeled. A density can also be greater than one without difficulty: densities are not probabilities and need only integrate to one. Moreover, an exact point has probability zero under a continuous distribution, whether its density is high or low.

A tail probability instead assigns probability to a specified set of outcomes, but the tail region must be chosen and justified; in multiple dimensions, there may be no unique ordering of outcomes as more or less extreme. Posterior predictive checks examine discrepancies between observed data and replicated data, while Bayes factors compare marginal support under models. These approaches answer related but distinct questions. A predictive density at the observation is not automatically a tail probability, a posterior probability of model adequacy, or a Bayes factor.

Diagnostics should therefore include checking that the outcome definition and scale are correct, that the predictive model reflects the trial design and sampling process, and that the summary being evaluated is the one actually observed. If results are extremely sensitive to an arbitrary change of units or data encoding, that is a warning against interpreting the raw density numerically. Sensitivity to reasonable model specifications also matters: a low density may reflect a narrow model or a misspecified likelihood rather than a genuine historical-current discrepancy. Predictive evidence cannot identify the cause of an unexpected result by itself.

Common pitfall: “The density at the observed outcome is 0.05, so there is a 5% probability that the historical model generated the data.” This is incorrect for continuous data: a density is measured in inverse units, not a probability, and it is not a probability that a model is true. State the density with its units, or calculate the probability of a clearly defined region; then interpret it only under the stated model.

Check Your Understanding

Answer each question, then compare with the brief solutions below.

What is averaged in a posterior predictive density, and what distribution provides the averaging weights?
For a continuous outcome, why is a predictive density of $0.02$ not necessarily a 2% probability?
In a two-component mixture with weights $0.7$ and $0.3$, the component densities at an observation are $0.04$ and $0.20$. What is the mixture density there?
Why can the density at a fixed observed value change when the outcome is reported in different units even though event probabilities do not change?

Solutions. (1) The current-data sampling distribution is averaged over parameter values using the posterior informed by historical data. (2) A continuous density has inverse-outcome units; probability is obtained by integrating over a region. (3) The mixture density is $0.7(0.04)+0.3(0.20)=0.088$. (4) A change of units rescales the density by the inverse Jacobian, while the corresponding interval changes width so its probability is preserved.