Covariate Imbalance as Apparent Conflict

Why Covariate Imbalance Can Look Like Conflict

Historical and current trials may enroll patients with different covariate distributions. If treatment benefit varies across covariate values, each trial’s average treatment effect depends partly on which patients it enrolled. The resulting marginal effects can differ even when the treatment effects within comparable covariate groups are the same. Conversely, if the treatment effect is constant across covariates, a prognostic covariate imbalance alone does not change a within-trial treatment contrast, although it can change the outcome levels in both arms.

The distinction matters when historical data inform a current treatment effect. A commensurate model that compares unadjusted marginal effects may interpret a difference in patient composition as evidence that the treatment effects themselves conflict. Covariate adjustment can clarify the comparison, but it does not automatically make the studies comparable: it depends on measuring relevant covariates, having adequate overlap, and specifying an appropriate effect model.

A Model for Composition and Treatment-Effect Modification

For a continuous outcome, consider the following mean model in each trial:

Here \(\alpha_s\) allows the baseline outcome level to differ by trial, \(\beta\) describes the covariate’s prognostic association, \(\theta_s\) is the treatment effect at \(X=0\), and \(\gamma\) describes how the treatment effect changes with \(X\). Thus the conditional treatment effect is \(\theta_s+\gamma X\). For binary \(X\), let \(p_s=P(X=1\mid s)\). Averaging over the covariate distribution in trial \(s\) gives:

The trial-specific intercept and the prognostic covariate term cancel in this within-trial contrast. If the conditional effect parameters are shared across trials, so that \(\theta_C=\theta_H\) and \(\gamma\) is common, then:

This identity isolates the source of apparent conflict: when \(\gamma\ne0\), different covariate prevalences produce different marginal effects even without a difference in the conditional effect at either covariate value. If \(\gamma=0\), the marginal treatment effects agree under this model, regardless of \(p_C\) and \(p_H\).

The model makes substantive assumptions. Treatment is randomized within each trial, outcomes and covariates use compatible definitions and scales, and the specified conditional mean form describes the relevant treatment-effect modification. The common-\(\gamma\) assumption says that the way treatment effect changes with \(X\) is the same in both trials. If that assumption is implausible, a shared treatment effect at \(X=0\) does not by itself imply agreement at other values of \(X\). For more general covariates, standardization averages the fitted conditional effect over the chosen target distribution rather than over \(p_s\).

Adjusting and Standardizing Before Borrowing

A direct way to compare studies is to estimate treatment effects conditional on the same covariates, then average those effects over one common target distribution. For a binary \(X\), define the target prevalence as \(p_*\). Under the model above, the standardized effect for trial \(s\) is:

Choosing the target distribution is part of the estimand, not a cosmetic calculation. A current-trial target answers what the treatment effect would be for the current trial’s covariate mix. A prespecified clinical target may answer a broader question. Whichever target is used, both trials must be standardized to the same distribution for their effects to be directly comparable.

A covariate-adjusted commensurate model places the borrowing relationship on corresponding adjusted treatment-effect parameters rather than on raw marginal contrasts. For example, with a fixed commensurate precision \(\kappa>0\), a simple specification is:

This prior expresses a preference for the current and historical effects at the same reference covariate value to be close, with the strength of that preference controlled by \(\kappa\). Its interpretation depends on the chosen reference value, covariate coding, and outcome units. If treatment-effect modification is present, borrowing on \(\theta_s\) alone does not determine agreement in \(\gamma\); that parameter must be modeled and assessed as well. A commensurate prior on adjusted effects is not a substitute for checking whether the adjustment model is credible.

What Covariate Adjustment Can and Cannot Resolve

Adjustment separates composition from conditional effect differences only to the extent that the model captures relevant effect modification. A useful analysis therefore examines both the adjusted effect and the assumptions behind it. Randomization supports an unbiased within-trial comparison, but it does not guarantee that the historical and current trials have the same conditional treatment effect, nor does it repair differences in outcome measurement, adherence, follow-up, or co-interventions.

A concise justification for the standardization step follows directly from averaging over the target distribution. For binary \(X\), the conditional effects are \(\theta_s\) when \(X=0\) and \(\theta_s+\gamma\) when \(X=1\). The target population assigns probabilities \(1-p_*\) and \(p_*\) to those strata, so:

Because \(p_*\) is identical for both trials, a difference in their standardized effects under this model is:

Thus standardization removes the contribution of differing covariate prevalence, while retaining a difference in the reference conditional effect. This result relies on the common-\(\gamma\) model; if treatment-effect modification differs by trial, the standardized contrast also depends on that difference.

Diagnostics, Alternatives, and Limitations

Before interpreting adjusted commensurability, inspect the covariate distributions and the model’s fit. In particular, check whether the covariates are measured in the same way, whether treatment-effect modification is clinically plausible, and whether the trials contain patients in the covariate regions used for standardization. If one trial has virtually no patients in a target stratum, its standardized effect there is driven mainly by extrapolation rather than direct information. Large changes in the adjusted effect under reasonable model specifications are a warning that the apparent resolution of conflict is fragile.

The main alternatives answer related but different questions. Comparing raw trial-level marginal effects is simple, but confounds effect differences with covariate composition when effects vary across covariates. Separate analyses avoid borrowing, but do not use historical information to estimate the current effect. Stratified comparisons can make effect modification visible when strata are sufficiently populated; regression adjustment can use covariates more efficiently but relies on the chosen model. Standardization makes the target population explicit. None of these approaches resolves unmeasured differences or guarantees transportability.

A Bayesian commensurate prior also adds assumptions beyond covariate adjustment: it specifies how strongly adjusted historical and current parameters should be related. The posterior may appear more precise because information is borrowed, but that precision is conditional on the prior and the likelihood model. A substantial residual difference between adjusted effects, poor overlap, or sensitivity to the adjustment specification should not be dismissed merely because the model produces a pooled estimate.

• Wrong reasoning: “The historical and current marginal effects differ, so their treatment effects must conflict.”
• Correction: First check whether treatment-effect modification and differing covariate distributions can explain the marginal difference. Compare conditional effects or effects standardized to a common target before judging commensurability.
• Another common error: “The baseline covariate distribution differs, so the within-trial randomized treatment contrast must be biased.”
• Correction: Under a constant additive treatment effect, a prognostic covariate imbalance changes outcome levels but cancels in the within-trial contrast. It changes the marginal treatment effect when treatment effects vary with the covariate, or can mislead when investigators compare outcome levels rather than within-trial contrasts.