Why Assess a Synthesis Before Interpreting It?
A Bayesian synthesis can combine information from several trials, but a calculation that combines numbers is not automatically a defensible combination of evidence. Trials may differ in who was enrolled, what treatment contrast was studied, how outcomes were defined, or how follow-up and missing data were handled. These differences can make a single pooled treatment effect hard to interpret, even when the statistical model produces a precise posterior distribution.
Regulatory assessment therefore asks two linked questions: are the data sources relevant to the same clinical question, and does the model represent their relationship credibly? The answer requires examining the trials’ design and conduct as well as the synthesis model. A close numerical fit cannot repair a mismatch in clinical meaning; conversely, differences between trial estimates do not alone show that synthesis is inappropriate.
Begin With the Clinical Question and the Sources
Before examining pooled estimates, specify the target population, treatment contrast, endpoint, time point, and estimand. For example, “the mean difference in a symptom score at week 12 for adults with moderate disease receiving treatment A rather than control” is more precise than “the treatment effect.” A synthesis that combines estimates for different time points or outcome definitions must explain how those estimates correspond to the target question.
Assess each source against that target. Relevant questions include whether eligibility criteria and disease severity are comparable; whether background care, dose, comparator, and follow-up differ; and whether the same outcome construct was measured using compatible scales. Differences need not rule out synthesis. They do, however, need a reasoned account of why the data remain informative for the same target, or a model that explicitly represents the differences.
- Population: Compare disease stage, prior treatment, age, geography, and other factors that may modify treatment effect. A difference in average baseline risk does not necessarily imply a different relative treatment effect, but that assumption should be justified.
- Endpoint and estimand: Check endpoint definitions, assessment times, handling of intercurrent events, and missing-data assumptions. A standardized scale does not by itself make distinct clinical constructs interchangeable.
- Treatment contrast: Confirm that dose, regimen, comparator, and concomitant care align with the target contrast. Evidence for a different dose is not automatically evidence for the proposed regimen.
- Design and data integrity: Review randomization, conduct, outcome ascertainment, analysis populations, and data quality. Identify overlapping participants or shared control groups, since treating correlated estimates as independent can overstate information.
The assumptions behind the original trials also matter. For example, an estimate based on a particular missing-data assumption may not be directly comparable with one using a different assumption. This assessment builds on the earlier material on missing-data assumptions in Bayesian evidence packages; the key point here is to check whether those assumptions affect the comparability of sources in the synthesis.
Worked Example: Are Two Symptom-Score Trials Clinically Comparable?
Suppose two hypothetical randomized trials compare the same daily dose of a new medicine with placebo in adults with moderate disease. Trial A measures a validated symptom score at week 12. Trial B measures the same score at week 12, but enrolls only patients who have failed two previous therapies. Investigators propose a synthesis for the effect in adults with moderate disease generally.
What aligns? Both trials use the same treatment contrast, endpoint, scale, and assessment time. These features support comparison of their week-12 treatment effects.
What differs? Trial B’s treatment-experienced population is narrower than the proposed target population. If prior treatment failure is associated with a different treatment response, Trial B’s estimate may not represent the same effect as Trial A’s estimate.
Assessment. The trials are not automatically incompatible, but the synthesis needs an explicit justification. Investigators could examine whether the effect is plausibly similar across prior-treatment groups, use a model that allows population-related variation if the evidence supports it, or limit the target population. Simply pooling the estimates and describing the result as applying to all adults with moderate disease would not resolve the transportability question.
What the Synthesis Model Assumes
A useful starting point is a normal approximation for trial-level treatment-effect estimates. Let \(y_j\) be the estimated effect from trial \(j\), measured on a common scale, and let \(s_j\) be its standard error. Under a common-effect model, the assumptions are:
Here \(\theta\) is the effect shared by all trials. This model assumes that each estimate is approximately normally distributed around the same effect, that the reported standard errors are suitable, and that estimates are independent conditional on \(\theta\). It also assumes that any important variation in true effects across trials is negligible for the purpose of the analysis. A prior distribution for \(\theta\) must also be specified; conclusions can depend on it, especially when the data are limited.
The common-effect assumption is strong. A hierarchical alternative allows trial effects to vary around a distribution:
In this model, \(\mu\) represents the mean of the distribution of trial effects and \(\tau\) represents their between-trial standard deviation. When \(\tau\) is near zero, the model approaches a common-effect model. When \(\tau\) is larger, trial effects can differ more, and borrowing across trials is reduced. The hierarchical model does not make incompatible populations or outcomes interchangeable: its distributional assumptions and the meaning of \(\mu\) still require clinical justification. With few trials, the data may provide little information about \(\tau\), so its prior can materially affect the degree of borrowing.
When estimates are correlated, for example because two contrasts share a control group, the independent likelihood above is not appropriate as written. The joint sampling model must represent their covariance. Treating correlated estimates as independent counts some information more than once and can produce unjustified precision.
How Trial Contributions Arise
The common-effect normal model makes the role of each source explicit. Suppose the prior is \(\theta \sim N(m_0,v_0)\), and the trial estimates are conditionally independent with known standard errors. Multiplying the prior density by the trial likelihoods gives a normal posterior. Completing the square in the exponent gives:
Each trial contributes precision \(s_j^{-2}\), and its estimate contributes \(y_j s_j^{-2}\) to the precision-weighted numerator. Thus a more precise estimate has more influence under this model. This is not a universal rule that the largest trial should always dominate: the formula applies only when the common-effect model, independence, normal approximation, and prior are defensible.
Worked Example: Calculating a Common-Effect Posterior
Consider two hypothetical trials estimating the same treatment effect in points on a symptom scale, with positive values favoring treatment. Suppose \(y_1=4\) points and \(y_2=5\) points, with \(s_1=s_2=2\) points. Use the prior \(\theta \sim N(0,4)\), where 4 is the prior variance in squared points. Assume the effects have the same clinical meaning, the normal approximations are reasonable, and the estimates are independent.
The prior precision is \(1/4=0.25\), and each trial precision is \(1/2^2=0.25\). Therefore,
The posterior mean is:
Under these assumptions, the posterior is \(N(3.000,1.333)\), with posterior standard deviation \(\sqrt{1.333}=1.155\) points. Each trial has the same precision as the prior, so the prior contributes meaningfully rather than being negligible. The result is a model-based synthesis for the shared effect; its numerical precision does not establish that the trials truly address the same population or estimand.
Check Agreement, Influence, and Sensitivity
A regulator should inspect trial-specific estimates and uncertainty alongside the synthesis. Consider whether differences are larger than expected from sampling variation, whether they have a plausible clinical explanation, and whether conclusions change under reasonable alternatives. A diagnostic can flag tension, but no single statistic can decide clinical compatibility.
For the common-effect model, one descriptive measure is the weighted residual statistic:
Large residual contributions identify trials that disagree with the weighted mean relative to their standard errors. This is a diagnostic of model fit, not proof that a trial is invalid or that its population differs. Review the data, protocol, endpoint definition, and effect modifiers before attributing a discrepancy.
Worked Example: A Conflicting Trial and Its Influence
Suppose two hypothetical trials estimate a treatment effect in the same units, with \(y_1=4\), \(y_2=-2\), and \(s_1=s_2=2\). Assume for this calculation that the common-effect model and independent normal approximations apply. The precision-weighted mean is:
Each trial’s contribution to \(Q\) is its squared residual divided by its variance:
The estimates differ by 6 units, while the standard error of their difference under independence is \(\sqrt{2^2+2^2}=\sqrt{8}=2.828\) units. The discrepancy is therefore about \(6/2.828=2.121\) standard errors. This flags disagreement for investigation; it does not identify its cause.
For a direct influence check using the same prior as before, the posterior mean with both trials is \((1+(-0.5))/0.75=0.667\) units. If Trial 2 is omitted, the posterior mean is \(1/0.5=2.000\) units. Omitting that trial changes the mean by \(-1.333\) units. This large change shows that Trial 2 materially affects the synthesis. It is not, by itself, a reason to exclude the trial: the assessment should ask whether the difference reflects chance, a population or design difference, a data issue, or a failure of the common-effect assumption.
Useful checks include fitting a hierarchical model where justified, comparing the synthesis with no pooling, and repeating the analysis after omitting or down-weighting each source. If a hierarchical model is used, assess sensitivity to plausible prior distributions for \(\tau\), especially when few trials are available. Investigate whether the conclusions depend on a particular endpoint, time point, population, or external data source. These checks address how much each source matters and whether the substantive conclusion is robust; they do not replace the clinical assessment of whether that source belongs in the target question.
| Approach | What it assumes or provides | Regulatory assessment |
|---|---|---|
| Common-effect synthesis | One shared treatment effect across trials | Efficient if the common-effect assumption is credible; potentially misleading if meaningful variation is ignored |
| Hierarchical synthesis | Trial effects vary around a distribution governed by \(\mu\) and \(\tau\) | Allows partial pooling, but requires defensible exchangeability and careful assessment of the prior on heterogeneity |
| Separate trial analyses | No shared treatment-effect parameter is imposed across trials | Preserves trial-specific results but does not provide a pooled estimate; useful as a comparator and sensitivity check |
Common Pitfalls and Practical Limits
- Pooling because estimates are on the same numerical scale. A shared scale does not guarantee a shared clinical question. Check the population, endpoint, time point, and estimand.
- Treating a hierarchical model as a cure for incompatibility. Partial pooling represents variation under a model; it cannot make unrelated endpoints or treatment contrasts answer the same question.
- Calling a discrepant study an outlier and deleting it. First check data and analysis, then seek clinical explanations and report sensitivity to inclusion. Exclusion should not be selected merely because it makes the result more favorable.
- Interpreting precision as validity. A narrow posterior can result from many sources or strong assumptions. Review source quality, dependence, model fit, and prior influence as well as posterior uncertainty.
Assessment has limits. With few trials, heterogeneity is hard to estimate; trial-level summaries may conceal important participant-level differences; and sensitivity analyses cannot prove that exchangeability holds. A synthesis is strongest when the target is explicit, source-level differences are transparent, assumptions are plausible, and the main conclusion is not driven by an unexamined modeling choice or single source.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Two trials use the same endpoint scale but assess outcomes at weeks 8 and 16. Why is combining their estimates not automatically justified?
- In a common-effect normal model, Trial A has standard error 1 and Trial B has standard error 2. What are their relative precision contributions?
- Two independent trial estimates are 3 and 7 units, each with standard error 2. Calculate their precision-weighted mean and \(Q\).
- Why does a large change after omitting one trial call for investigation but not automatically justify excluding that trial?
Solutions. (1) The assessment times may correspond to different treatment effects and estimands. Investigators need to justify that both estimates inform the same target or model the time difference. (2) Precision is \(1/s_j^2\): Trial A contributes \(1\), Trial B contributes \(1/4=0.25\). Trial A therefore contributes four times the precision of Trial B under the stated model. (3) Equal standard errors give the mean \((3+7)/2=5\) units. Each residual is 2 units, so \(Q=(3-5)^2/4+(7-5)^2/4=1+1=2\). (4) The change establishes influence, not invalidity. The trial may be valid and relevant; investigate its conduct, population, endpoint, and possible effect modifiers, then report whether conclusions are sensitive to its inclusion.