Comparing Competing Evidence Chains

Why Compare Evidence Chains?

An integrated forecast can be plausible at each modeling step and still be unreliable at the end. A pharmacokinetic model may predict exposure well, and an exposure-response model may fit its data, yet the handoff between them can rely on an unsupported assumption about dose, population, or endpoint. Comparing complete evidence chains helps identify whether their trial forecasts are robust to different ways of representing those handoffs and their uncertainty.

The comparison is meaningful only when the chains address the same development question. The target population, treatment regimen, endpoint, follow-up time, comparator, and trial analysis must be held fixed. Otherwise, different forecasts may reflect different estimands rather than different ways of integrating evidence. The earlier material on model discrepancy, shared-data dependence, predictive checks, and interface sensitivity provides useful tools for this comparison; here the emphasis is on judging the competing workflows together.

Two common workflows differ in how they carry information across interfaces. A plug-in chain passes selected point estimates from one component to the next and treats them as fixed downstream. This is computationally straightforward, but its uncertainty can be too small if it omits uncertainty in upstream estimates, interface discrepancies, or dependence between components. An uncertainty-propagating chain carries joint draws or an equivalent joint distribution through the linked components, preserving relevant uncertainty and dependence. Neither workflow is automatically correct: both depend on whether their models, data, and interface assumptions represent the prediction task adequately.

Set Up a Comparable Forecast

Consider a hypothetical development question: what distribution of responders should be expected in a future 100-patient trial at a specified regimen? Both chains use the same endpoint and target population. For illustration, the exposure-response relationship is logistic:

Here \(E\) is exposure in a prespecified unit, \(\delta\) is an interface discrepancy on the log-odds scale, \(p\) is the response probability, and \(Y\) is the future responder count. This simplified model assumes conditionally independent patient outcomes with a common response probability, a correctly specified logistic relationship for the task, and a consistent definition of exposure and response across model components. These assumptions are not guaranteed by good fit to the component data; they must be examined in context.

The plug-in workflow uses \(E=10\) and \(\delta=0\) as fixed values. The uncertainty-propagating workflow represents uncertainty in the linked exposure and response prediction with three illustrative, weighted joint draws. The draws preserve the same central exposure and allow modest interface variation:

These draws are a deliberately small illustrative approximation to joint predictive uncertainty, not empirical estimates from a real drug program. A real comparison would use enough joint draws to represent the uncertainty relevant to the decision and would document how shared data and dependence were handled.

Why Predictive Uncertainty Can Differ

A future trial outcome varies because of patient-level sampling, even if the response probability were known. It also varies because the integrated chain does not determine that probability with certainty. The total predictive variance includes both sources. For a random response probability \(p\) and \(Y\mid p\sim\operatorname{Binomial}(n,p)\), the law of total variance gives:

The first term is average binomial sampling variance. The second is variance in the expected trial count across the uncertain probability distribution. The identity follows by conditioning on \(p\): \(\operatorname{Var}(Y\mid p)=np(1-p)\) and \(E[Y\mid p]=np\), so \(\operatorname{Var}(E[Y\mid p])=\operatorname{Var}(np)=n^2\operatorname{Var}(p)\). A plug-in workflow that treats \(p\) as fixed includes only the first term.

Check Predictions Against Held-Out Outcomes

Predictive checks ask whether outcomes not used to fit or tune a chain are reasonably compatible with its predictions. A check can reveal systematic underprediction, overprediction, or excessive certainty, but a small set of outcomes cannot prove that one chain is true. Checks should match the forecast target: a trial-level prediction should be assessed using trial-level outcomes or an appropriate aggregation, not only component-level fit statistics.

The following hypothetical held-out cohorts each contain 100 patients. The plug-in chain predicts \(p=0.7311\) for all three. The uncertainty-propagating chain makes cohort-specific predictions associated with the low, central, and high draws above. For an initial descriptive check, a conditional standardized residual is \((y-np)/\sqrt{np(1-p)}\). It uses binomial sampling variability at the stated probability; it does not include the full uncertainty of the integrated chain.

Useful diagnostics include predictive intervals and coverage across held-out settings, plots of observed versus predicted outcomes, calibration across risk or exposure strata, and residual patterns by site, dose, or population. For an uncertainty-propagating workflow, simulate complete trial outcomes from joint draws and inspect whether the observed outcomes look unusual relative to those simulations. If the plug-in chain produces markedly narrower predictions, check whether that is a consequence of omitted uncertainty or a genuine difference in its modeled assumptions. Poor calibration, repeated residual trends, or incompatible predictive distributions are signals to investigate; they do not identify the cause by themselves.

Compare Assumptions, Not Just Outputs

A useful comparison makes each workflow’s handoffs explicit. At every interface, ask what quantity is passed forward, what population and units it represents, whether the downstream model expects the same quantity, and how uncertainty and dependence are carried. For example, a prediction for average exposure cannot silently be treated as an individual-level exposure distribution if the response model depends nonlinearly on individual exposure. Likewise, uncertainty from evidence shared by two components should not be counted as independent information.

The workflows have different practical strengths and vulnerabilities:

Neither approach replaces the other in every setting. A plug-in result can be a useful reference forecast, especially when uncertainty is small or the decision only requires a rough central estimate. The propagated chain is more informative when uncertainty at the interfaces could change a decision, provided its joint assumptions are defensible. Sensitivity analyses can isolate which assumptions drive the difference; an average across chains is a separate strategy and should not be used to conceal unresolved incompatibilities.

Common Mistake and Practical Limits

Common mistake: “The two chains predict nearly the same mean responder count, so their forecasts are interchangeable.” This ignores the trial-level distribution. In the worked example, the means are close, but the propagated chain has a substantially larger predictive standard deviation because it includes uncertainty in \(p\). Decisions involving a threshold, such as whether the trial is likely to exceed a minimum response count, can therefore differ even when mean forecasts are similar.

A second limit is that a predictive check cannot compensate for an ill-defined target or data leakage. If a held-out cohort helped select the model or tune an interface, it is no longer an independent check in the intended sense. Even a chain that predicts held-out outcomes well may fail under a new regimen or population if the transport assumptions do not hold. Report the target conditions, the uncertainty represented, and the conditions under which the comparison should not be extrapolated.