Triangulating Evidence Across Designs with Different Biases

Why Compare Different Designs?

A causal estimate is only as credible as the assumptions that connect the observed data to the causal effect of interest. An adjusted observational analysis may be vulnerable to an unmeasured cause of both treatment and outcome. A randomized trial avoids that particular source of confounding for treatment assignment, but may face nonadherence or limited generalizability. An instrumental-variable analysis can address some treatment-selection problems, yet relies on assumptions such as exclusion that are not needed in the same form for a randomized comparison.

Triangulation means comparing evidence from designs whose important biases and assumptions differ. The aim is not to count how many estimates point in one direction. It is to ask whether their differences make sense given what each design estimates and how each could be biased. Agreement is more informative when designs have genuinely different vulnerabilities; it is not proof that all their assumptions hold.

Map the Assumptions Before Comparing Estimates

An observational analysis commonly compares outcomes under treatment \(D=1\) and no treatment \(D=0\), adjusting for measured covariates \(X\). For a causal interpretation, the measured covariates must suffice to control confounding: conditional on \(X\), treatment must be independent of the relevant potential outcomes. Consistency requires that each person’s observed outcome under the treatment received equal the corresponding potential outcome; positivity requires that each treatment option have a positive probability at covariate levels represented in the target population. If an unmeasured variable \(U\) affects both treatment and outcome, conditional exchangeability may fail even after adjustment for \(X\).

Random assignment makes treatment assignment independent of potential outcomes, on average, in the randomized population. Thus, a comparison by assigned group identifies the intention-to-treat effect under standard conditions such as consistent outcome measurement and appropriate handling of missing outcomes. When people do not adhere to assigned treatment, that effect is the effect of assignment, not automatically the effect of receiving treatment. Randomization also does not guarantee that the trial population represents every population to which one might wish to generalize.

An instrumental-variable analysis uses an instrument \(Z\) that shifts treatment uptake \(D\). For the usual Wald ratio to identify a complier average treatment effect, the instrument must be relevant, independent of the joint vector of potential outcomes and potential treatment statuses, satisfy exclusion, and satisfy monotonicity. Relevance means assignment to different instrument levels changes treatment uptake. Independence means \(Z\) is independent of \((Y(0),Y(1),D(0),D(1))\), where \(D(z)\) is the treatment status under instrument level \(z\). Exclusion means \(Z\) affects the outcome only through treatment. Monotonicity means the instrument does not cause anyone to move opposite to its treatment-encouraging direction. The resulting effect is for compliers, not necessarily the average treatment effect in the full population.

Negative-control outcomes and exposures can reveal patterns that raise concern about hidden bias, but a negative-control association is not itself a direct estimate of the causal effect under study. Its diagnostic value depends on whether it shares relevant causes or measurement processes with the primary analysis and on whether the control is truly unaffected by the treatment or exposure of interest.

Bias Direction Depends on the Causal Structure

The direction of confounding is not determined by the word “unmeasured.” It depends on how the omitted variable relates to treatment and outcome, and on the chosen effect scale. For example, on an additive outcome scale, suppose the conditional mean follows the simplified model \(E[Y\mid D,U]=\beta_0+\tau D+\gamma U\), with no other confounding and no interaction. Here \(\tau\) is the treatment effect and \(\gamma\) is the outcome difference associated with \(U\). Let \(q_d=P(U=1\mid D=d)\). The unadjusted treated-minus-untreated contrast is then

The second term is the confounding bias in this simplified model. If \(U\) increases the outcome (\(\gamma>0\)) and is more common among treated people (\(q_1>q_0\)), the bias is positive. For a beneficial treatment whose effect is negative on this scale, this positive bias makes the observed effect look less beneficial than the causal effect. If the treated group has less of the harmful \(U\), the bias instead points in the negative direction and can exaggerate apparent benefit. This sign argument is specific to the stated additive model; interactions or another outcome scale can change it.

Worked Comparisons Across Designs

How to Triangulate Without Overclaiming

A disciplined comparison begins by writing down what each estimate represents and what could distort it. The following sequence keeps the comparison focused on causal structure rather than on whether point estimates happen to agree.

Why the Distinctions Matter

Triangulation is useful because no single design is automatically decisive. Observational adjustment may control rich measured covariates while leaving hidden confounding. Randomization protects the assignment comparison from baseline confounding, but nonadherence changes what that comparison estimates. An instrument can address treatment selection under its assumptions, but an invalid exclusion restriction can undermine its interpretation. Negative controls can expose warning signs, but they do not repair a primary analysis by themselves.

A common mistake is to call several estimates “independent confirmation” simply because they come from different analyses. If they share the same unmeasured cause, outcome measurement problem, or selection mechanism, their errors may also be shared. Another mistake is to average estimates with different targets, such as a population average effect and a complier effect. The correction is to map each design’s assumptions and vulnerabilities first, check whether the estimands are comparable, and then judge whether the pattern of results is coherent under plausible bias scenarios.