Sensitivity Analysis for Unmeasured Effect Modifiers

Why quantify an unmeasured modifier?

A transported treatment effect is conditional on assumptions about how treatment effects compare across populations. Even after measured population differences have been addressed, a characteristic that was not measured—or was not represented in the model—could modify the treatment effect and be distributed differently in the target population. A useful sensitivity analysis does not claim to discover that characteristic. It asks a more focused question: how strong would its effect modification have to be, given a specified population difference, to change the conclusion?

The calculation depends on the effect scale and on the decision being examined. This tutorial uses an additive benefit scale: \(B\) is the control-group event risk minus the treatment-group event risk, so positive values favor treatment. A benefit of 6 percentage points means that the event risk is estimated to be 6 percentage points lower with treatment than with control. The same approach can be defined on another appropriate scale, but its parameters and interpretation must then be changed accordingly.

A one-modifier bias model

Let \(B_S\) denote the treatment benefit in the source population, either estimated directly or after standardization to a defined source population. Let \(U\) be an unmeasured modifier. For a binary modifier, suppose its effect on benefit is additive: \(B(U)=a+\gamma U\), where \(U=0\) or \(1\), and \(\gamma\) is the difference in treatment benefit between the \(U=1\) and \(U=0\) groups. Thus, \(\gamma\) has the same units as \(B\). It is not the effect of \(U\) on outcome risk irrespective of treatment; it is the difference in treatment effect across levels of \(U\).

Let \(q_S=P(U=1\mid S)\) and \(q_T=P(U=1\mid T)\) be the modifier prevalences in the source and target populations. If the source and target differ only in this modifier’s distribution for the purpose of the sensitivity calculation, then the corresponding population-average benefits are \(a+\gamma q_S\) and \(a+\gamma q_T\). Subtracting the source average from the target average gives the adjustment:

The quantity \(\delta=q_T-q_S\) is the population difference in modifier prevalence, and the bias adjustment is \(\gamma\delta\). This derivation also explains the direction of the adjustment: a modifier can change the target benefit only if it both modifies treatment effect (\(\gamma\ne 0\)) and differs in prevalence (\(\delta\ne 0\)). For a continuous modifier with a linear effect-modification relationship, the same derivation gives \(B_T=B_S+\gamma(\mu_T-\mu_S)\), where \(\mu_S\) and \(\mu_T\) are the source and target means of that modifier. In that case \(\gamma\) is the change in benefit per unit of \(U\).

The model is deliberately narrow. It assumes the benefit varies linearly with \(U\) on the chosen scale, the source estimate and target contrast refer to compatible outcomes and treatment definitions, and the population difference in \(U\) is represented by the specified prevalence or mean difference. It also assumes that other relevant differences have already been handled or are being held fixed for this calculation. These conditions are not established by the sensitivity analysis; they define what its numbers mean.

Choosing the tipping-point threshold

A conclusion may be defined by more than whether the point estimate is positive. For example, a decision rule might require a target benefit of at least 2 percentage points. If \(\tau\) is the chosen threshold, the tipping point satisfies \(B_S+\gamma_{\mathrm{tip}}\delta=\tau\). Provided \(\delta\ne 0\), this gives:

The threshold must be stated before interpreting the result. A zero-benefit threshold asks when the point estimate changes sign. A minimum-benefit threshold asks when the estimate falls below a specified clinically or operationally meaningful value. If the question concerns an interval-based decision, the interval boundary and the uncertainty calculation also need to be specified; a point-estimate tipping point does not answer that different question.

Uncertainty and combined modifiers

A sensitivity adjustment to an estimate does not automatically include all uncertainty. If the source estimate has a confidence interval and \(\gamma\) and \(\delta\) are held fixed, adding the same \(\gamma\delta\) to both interval endpoints shifts that interval. This conditional calculation does not account for uncertainty in the sensitivity parameters, nor does it assert that those parameters are known. When the parameters themselves are uncertain, analysts can examine ranges or distributions for them and show how conclusions vary; the resulting analysis is still conditional on the chosen ranges or distributions.

With multiple unmeasured modifiers, a simple additive specification is \(B_T=B_S+\sum_j\gamma_j\delta_j\). Here each \(\gamma_j\delta_j\) is a separate bias contribution, with units of benefit. This form assumes the contributions can be added without interaction terms among the modifiers. It is useful to show a tipping frontier: the combinations of parameter values that just reach a decision threshold. It does not make each parameter separately identifiable from the transported data.

Assumptions, diagnostics, and alternatives

Before using a tipping point to support a bridging decision, check that its inputs and interpretation are coherent. The modifier contrast should be stated on an interpretable scale, with its direction and units. The assumed population difference should be plausible for the target and source populations after accounting for measured characteristics. The effect-modification parameter should represent a treatment-effect contrast, not a prognostic association with outcome alone. Finally, the treatment, endpoint, and follow-up context must be sufficiently aligned that a change in the treatment-effect contrast is meaningful on the selected scale.

A practical diagnostic is to calculate the adjustment at several positive and negative values of \(\gamma\), including zero, and verify that the resulting direction agrees with \(\gamma\delta\). For a binary modifier, prevalence differences must lie between \(-1\) and \(1\); for a continuous modifier, the units and plausible range of the mean difference need to be explicit. If several modifiers are considered, inspect whether the assumptions about their differences and additive contributions are jointly plausible. Correlation among modifiers, nonlinearity, or interaction between modifiers can make a simple sum a poor representation.

A tipping-point analysis is useful because it makes the required departure from the transport assumption visible. It is not a test of whether transportability holds, and it cannot identify an unmeasured modifier from data that do not measure it. A result that tips only under very large, implausible effect modification may be reassuring relative to the elicited range, but it is not proof of no bias. Conversely, a small tipping value signals fragility, not proof that the conclusion is wrong.

The main alternatives answer related but distinct questions. A single tipping-point calculation gives the threshold for a chosen decision rule. A grid or contour display shows how conclusions change over a range or combination of bias parameters. A probabilistic bias analysis assigns distributions to the sensitivity parameters and propagates that uncertainty, but its output depends on those distributions. Collecting local data that measure the suspected modifier can replace some assumptions with empirical information, though it does not remove all transport assumptions. These approaches complement rather than substitute for a clearly defined effect scale and target-population contrast.

Common pitfalls and interpretation

• Confusing prognostic effects with effect modification. A factor may predict outcomes in both treatment groups yet have no effect on the treatment contrast. The sensitivity parameter \(\gamma\) concerns the difference in treatment benefit across modifier levels.
• Ignoring the direction of the population difference. A negative \(\gamma\) does not always reduce target benefit; the sign of \(\delta\) matters because the adjustment is \(\gamma\delta\).
• Reporting a threshold without its scale or units. “The modifier must be strong” is not interpretable unless the effect contrast, modifier contrast, and outcome scale are stated.
• Treating a point-estimate tipping point as an interval conclusion. The threshold for changing the point estimate’s sign may differ from the threshold for changing an interval-based decision.
• Presenting a sensitivity value as an estimated fact. The calculation says what follows if the assumed values and model hold; it does not show that those values are true.

The central output is therefore not just a number. It is a conditional statement: for a stated difference in modifier distribution and a stated effect scale, an effect-modification parameter of a given magnitude would move the transported estimate to a specified decision boundary. That statement is most useful when the assumed values are compared with clinical and pharmacological knowledge, and when alternative plausible values are shown transparently.