Posterior Predictive Checks for a Borrowing Model

Why Check a Borrowing Model?

External controls can contribute information when concurrent control data are limited, but that contribution depends on assumptions: in particular, that the external and concurrent control outcomes are sufficiently compatible for the intended analysis. A model may produce a precise estimate while fitting an important feature of the observed control data poorly. Posterior predictive checks make that risk visible by comparing observed control outcomes with outcomes replicated from the fitted model.

A check is not a rule that automatically accepts or rejects a borrowing model. It is a way to ask a specific question about model-data agreement. The question might concern the overall response count, a difference between sites, or a trend across calendar periods. The chosen diagnostic must match the feature of the data that could undermine borrowing.

A Robust Borrowing Model

Suppose the external controls have $x_E$ responses among $n_E$ participants, and the concurrent controls have $y_C$ responses among $n_C$ participants. Let $\theta_C$ be the response probability for concurrent controls. For the worked examples, outcomes are binary, participants are independent within each group, and each group has a common response probability. These assumptions may be unsuitable if outcomes are clustered, follow-up differs, or response probabilities vary across relevant strata.

Use a robust mixture prior for $\theta_C$. Its borrowing component incorporates the external observations; its weak component allows the current control data to carry more weight if they conflict with the external information. For simplicity, let the external prior be $\theta_E\sim\operatorname{Beta}(1,1)$, and set the borrowing prior for $\theta_C$ to the posterior distribution based on the external controls. Let the weak component also be $\operatorname{Beta}(1,1)$. The prior probability of the borrowing component is $\pi$.

Definition: In this tutorial, $Z=1$ denotes the borrowing component and $Z=0$ the weak component. Conditional on the observed external data, the mixture prior is $$ \theta_C\mid Z=1,x_E \sim \operatorname{Beta}(1+x_E,1+n_E-x_E),\qquad \theta_C\mid Z=0 \sim \operatorname{Beta}(1,1), $$ with $P(Z=1)=\pi$. The concurrent-control likelihood is $Y_C\mid\theta_C\sim\operatorname{Bin}(n_C,\theta_C)$. A posterior predictive replication is a new control outcome $Y_C^{\mathrm{rep}}$ drawn from the fitted model, with parameters drawn from their posterior distribution.

This is a deliberately simple robust borrowing model, not a claim that a mixture prior resolves every transportability problem. The binomial likelihood assumes a common probability within the concurrent control group and independent outcomes. The external data are treated as fixed when constructing this conditional prior. A different external-data model or prior changes the resulting check.

For a component with $\theta_C\sim\operatorname{Beta}(a,b)$, integrating over $\theta_C$ gives the beta-binomial probability $$ P(Y_C=y\mid n_C,a,b)=\binom{n_C}{y}\frac{B(a+y,b+n_C-y)}{B(a,b)}. $$ Consequently, the posterior probability of borrowing is $$ P(Z=1\mid y_C,x_E)= \frac{\pi\,P(y_C\mid n_C,1+x_E,1+n_E-x_E)} {\pi\,P(y_C\mid n_C,1+x_E,1+n_E-x_E)+(1-\pi)\,P(y_C\mid n_C,1,1)}. $$ Within component $z$, the posterior for $\theta_C$ is beta with its two shape parameters increased by the concurrent response and nonresponse counts. The posterior predictive distribution is a mixture of the beta-binomial distributions from those updated components, weighted by the posterior probabilities of $Z$.

Building and Interpreting a Predictive Check

Choose a discrepancy $T$ that captures a potential failure relevant to the borrowing assumption. For an overall response-rate check, one choice is the absolute difference between concurrent and external rates: $T(y_C,x_E)=|y_C/n_C-x_E/n_E|$. Hold the observed external data fixed, draw $\theta_C$ from its posterior, and generate $Y_C^{\mathrm{rep}}\sim\operatorname{Bin}(n_C,\theta_C)$. Compare $T(Y_C^{\mathrm{rep}},x_E)$ with $T(y_C,x_E)$.

p_{\mathrm{ppc}} = P\!\left\{T(Y_C^{\mathrm{rep}},x_E)\geq T(y_C,x_E)\mid y_C,x_E\right\}.

This tail area measures how often replications from the fitted model are at least as discrepant by the chosen statistic as the observed data. A small value can indicate that the model rarely reproduces the observed feature. It is not a frequentist p-value with a guaranteed reference distribution, nor is it the probability that the model is true. Because the posterior uses the observed data, posterior predictive checks can also be conservative: fitting the model to the same data being checked may make discrepancies look less unusual.

A practical simulation proceeds as follows.

State the feature to check.
Choose a statistic in advance where possible, and explain why that feature matters for borrowing.

Draw from the posterior.
For each iteration, draw the mixture component and then draw $\theta_C$ from its component-specific posterior.

Replicate control outcomes.
Generate $Y_C^{\mathrm{rep}}$ under the stated likelihood, using the drawn value of $\theta_C$.

Compare and diagnose.
Calculate the discrepancy for the replication and the observed data; inspect the tail area and the posterior borrowing weight together.

Worked Example: A Moderate Difference in Response Rates

Suppose the external controls have 10 responses among 30 participants, the concurrent controls have 2 responses among 10 participants, and $\pi=0.5$. Under the borrowing component, the prior for $\theta_C$ is $\operatorname{Beta}(11,21)$; under the weak component it is $\operatorname{Beta}(1,1)$. Both groups are assumed to follow the stated binomial model.

For a beta-binomial distribution, successive probabilities have ratio $$ \frac{P(Y_C=k+1)}{P(Y_C=k)} =\frac{n_C-k}{k+1}\frac{a+k}{b+n_C-k-1}. $$ For the borrowing component, $n_C=10$, $a=11$, and $b=21$. Starting with relative weight $w_0=1$, the recurrence gives $w_1=3.667$ and $w_2=6.828$. Continuing the recurrence through $k=10$ gives a total relative weight of approximately 37.31. Thus the predictive probability of the observed count is approximately $6.828/37.31=0.1830$. Under the weak $\operatorname{Beta}(1,1)$ component, each possible count from 0 through 10 has probability $1/11=0.09091$.

The posterior probability of borrowing is therefore $$ \frac{0.5(0.1830)}{0.5(0.1830)+0.5(0.09091)}=0.6681. $$ The concurrent data increase the posterior weight on borrowing from 0.5 to about 0.668. The component-specific posterior means are $13/42=0.3095$ in the borrowing component and $3/12=0.2500$ in the weak component. The mixture posterior mean is approximately $0.668(0.3095)+0.332(0.2500)=0.2898$.

The observed absolute rate difference is $|2/10-10/30|=0.1333$. To check this feature, replicate $Y_C$ from the posterior predictive mixture and calculate $|Y_C^{\mathrm{rep}}/10-10/30|$ on each draw. A large proportion of replications at least this discrepant would suggest that this check finds no striking rate mismatch. That conclusion would apply only to this statistic; it would not establish compatibility in other respects.

Worked Example: A Severe Difference and the Correct Predictive Calculation

Keep the same external data and model, but suppose the concurrent controls have 9 responses among 10 participants. The observed absolute rate difference is now $|0.9-1/3|=0.5667$. This is a hypothetical illustration of strong conflict between the observed rates.

For the borrowing component, the predictive probability of 10 responses relative to 0 responses is the product of the successive probability ratios: $$ \frac{P(Y_C=10)}{P(Y_C=0)} =\prod_{k=0}^{9}\frac{10-k}{k+1}\frac{11+k}{30-k} \approx 0.006149. $$ This is a relative weight for the count 10 compared with the count 0; it is not a tail probability. Summing the relative weights for counts 0 through 10 gives approximately 37.31, so $P(Y_C=10\mid Z=1)\approx0.006149/37.31=0.0001648$. Under the weak component, $P(Y_C=9)=1/11=0.09091$.

The posterior borrowing weight is consequently $$ \frac{0.5(0.001731)}{0.5(0.001731)+0.5(0.09091)} \approx 0.0187. $$ The data have greatly reduced the borrowing component's weight. The posterior distribution is therefore dominated by the weak component, whose update is $\operatorname{Beta}(10,2)$, with mean $10/12=0.8333$. The borrowing component updates to $\operatorname{Beta}(20,22)$, with mean $20/42=0.4762$.

For the statistic $T=|Y_C/10-1/3|$, a replicated count is at least as discrepant as the observed count exactly when $Y_C^{\mathrm{rep}}\geq9$. Under the weak-component posterior $\operatorname{Beta}(10,2)$, the beta-binomial probabilities are $P(Y_C^{\mathrm{rep}}=10)=0.2619$ and $P(Y_C^{\mathrm{rep}}=9)=0.2757$, giving a tail probability of $0.5376$. Under the borrowing-component posterior $\operatorname{Beta}(20,22)$, the tail probability is approximately $0.01346$. Combining these with the posterior borrowing weight of approximately $0.0187$, the full mixture tail probability is approximately $0.5278$. This check does not flag the observed discrepancy as rare under the fitted posterior predictive distribution, largely because the model has responded to the conflict by greatly reducing borrowing. The low posterior borrowing weight is itself an essential diagnostic.

Check the Features That Matter

An overall rate check can miss structure. If exchangeability is assumed across sites or calendar periods, inspect whether outcomes are also compatible across those groups. For example, a pooled count can match the external rate while responses are concentrated in one site and absent in another. A posterior predictive check should reproduce the structure assumed by the model: if site-specific information matters, retain site labels in the likelihood and use a statistic that can detect site differences.

Worked Example: A Pooled Fit Hides Site Heterogeneity

Suppose 20 concurrent controls are split equally between two sites. Site 1 has 0 responses among 10 participants and site 2 has 8 among 10. The external controls have 8 responses among 20 participants. The pooled concurrent rate, $8/20=0.4000$, equals the external rate, but the site-specific rates are 0 and 0.8000.

Under a model that assumes one common concurrent-control probability, pooling the external and concurrent counts gives a $\operatorname{Beta}(17,25)$ posterior: there are 16 responses and 24 nonresponses in total, added to the $\operatorname{Beta}(1,1)$ prior. Its mean is $17/42=0.4048$. As a descriptive discrepancy, compare the observed site counts with the expected counts at that mean. The expected count is $10(0.4048)=4.048$ at each site, and the Pearson-type discrepancy is $$ T_{\mathrm{obs}}=\frac{(0-4.048)^2}{4.048}+\frac{(8-4.048)^2}{4.048}\approx7.91. $$

A posterior predictive check would draw $\theta_C\sim\operatorname{Beta}(17,25)$ and generate independent replicated counts $Y_1^{\mathrm{rep}},Y_2^{\mathrm{rep}}\sim\operatorname{Bin}(10,\theta_C)$. For each draw, calculate both the observed-data discrepancy $T_{\mathrm{obs}}(\theta_C)=\frac{(0-10\theta_C)^2}{10\theta_C}+\frac{(8-10\theta_C)^2}{10\theta_C}$ and the replicated discrepancy against the same expected counts $10\theta_C$, then compare these paired values. The value of about 7.91, calculated at the posterior mean, is descriptive only and is not the threshold for this check. If the observed discrepancy is in the extreme tail of these paired comparisons, the common-rate assumption is questionable despite the matching pooled rates. Site-specific differences could then make pooled borrowing misleading, so the model structure and the resulting treatment comparison should be examined before relying on the pooled estimate.

Alternatives, Limitations, and Common Mistakes

Posterior predictive checks are one part of model assessment, not a substitute for it. A prior predictive check asks whether data generated before observing the trial look plausible under the prior and likelihood. It is useful for detecting implausible prior implications, but does not assess fit to the observed trial in the same way. A sensitivity analysis instead compares conclusions under different defensible borrowing assumptions; it assesses dependence on modeling choices, not whether a particular discrepancy is reproduced. Leave-one-out or other predictive validation approaches can reduce the reuse of an observation in fitting and checking, but require a suitable design and interpretation. These approaches answer related but distinct questions.

A posterior predictive check can fail to identify a problem if its statistic does not target the relevant feature, if the model is flexible enough to adapt to it, or if the data are too sparse to reveal it. A reassuring tail area cannot establish that external and concurrent controls are exchangeable. Conversely, a small tail area may identify a mismatch without showing which model component caused it. Examine graphical replications, component weights, and sensitivity to plausible model changes together.

Mistake: treating the predictive tail area as the probability the model is correct. It is a conditional comparison for a selected discrepancy, not a model-truth probability.
Mistake: reporting only an overall rate check. Matching pooled counts can conceal site, calendar-time, or other structured differences.
Mistake: calling a relative weight a probability. In the second example, 0.006149 is $P(Y_C=10)/P(Y_C=0)$ under the borrowing component. Normalizing across all possible counts is required to obtain $P(Y_C=10)$.
Mistake: interpreting a reassuring check as proof of safe borrowing. The check addresses only its chosen statistic under its stated model; also report how the posterior borrowing weight and conclusions change.

Key takeaway: Replicate the control outcomes under the fitted borrowing model, compare a clinically and structurally relevant discrepancy, and interpret that comparison alongside posterior borrowing weights and sensitivity analyses. A check can reveal a model's weakness, but cannot certify transportability.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

With $x_E=6$, $n_E=20$, $y_C=3$, and $n_C=10$, what are the two component-specific posterior distributions for $\theta_C$ under the model in this tutorial?
Why is a posterior predictive tail area not the probability that the borrowing model is true?
An overall control-rate check looks reassuring, but a site-level check is extreme. Give one reason the two findings can coexist.
In the second worked example, why is 0.006149 not the predictive probability of 10 responses?

Solutions. (1) The borrowing prior is $\operatorname{Beta}(7,15)$; after 3 responses and 7 nonresponses it becomes $\operatorname{Beta}(10,22)$. The weak prior becomes $\operatorname{Beta}(4,8)$. (2) The tail area is the probability, conditional on the fitted posterior predictive model, of a replication at least as discrepant by one selected statistic; it is not a probability over competing model-truth states. (3) The pooled rate can agree while responses are distributed unevenly across sites, so a statistic that ignores site labels misses heterogeneity. (4) It is the ratio $P(Y_C=10)/P(Y_C=0)$; the probability for 10 responses requires normalizing the relative weights over all possible counts.