Protocol Sensitivity Analyses For Ambiguous Definitions

Why Ambiguous Definitions Need Planned Alternatives

A target-trial protocol can be clear in principle while its operational translation to records remains uncertain. A diagnosis may be supported by one code or require two; treatment initiation may be identified from a dispensing record or an administration record; an outcome may include only hospitalizations or a broader set of acute-care events; and follow-up may end at a fixed day or at a different prespecified horizon. Each choice can change who enters the emulation, how treatment strategies are assigned, which events count, or how long outcomes are observed.

A protocol sensitivity analysis makes these choices visible before comparing results. It specifies a defensible primary definition and alternative operational definitions, then evaluates what changes when those alternatives are applied. The exercise is not a search for the version with the preferred result. Nor does stability across definitions prove that the causal assumptions hold. It assesses sensitivity to a particular source of operational uncertainty.

The key distinction is between a rule that is uncertain because records support more than one credible interpretation and a rule changed after seeing which result is more favorable. Sensitivity alternatives should be justified by clinical meaning, data provenance, or plausible measurement practice—not by their effect estimate.

Specify the Ambiguity and Its Consequences

For each uncertain definition, document the source of ambiguity, the primary rule, the alternative rule or rules, and the data needed to implement them. Also state whether the alternative changes the target population, treatment-strategy classification, outcome variable, follow-up horizon, or more than one of these. Varying one dimension at a time makes the consequence easier to interpret; joint alternatives can then examine plausible combinations when the uncertainties may interact.

An alternative is informative only if the data can implement it with reasonable fidelity. If records do not distinguish administrations from dispensings, an “administration-only” scenario may be impossible, not merely inconvenient. Feasibility should be assessed before the alternatives are finalized. Likewise, every scenario must preserve a coherent time origin and treatment assignment rule; otherwise the comparison can silently introduce a different design problem rather than test the stated ambiguity.

Why Definition Changes Can Change the Estimand

Let \(E\) denote eligibility under a specified rule, \(A\) the treatment strategy, and \(Y^a(\tau)\) the outcome by follow-up horizon \(\tau\) that would be observed under strategy \(a\). For a binary outcome, one possible causal contrast is the risk difference in the eligible population:

This notation makes two points explicit. First, changing the eligibility rule changes the conditioning population and can therefore change the target estimand. Second, changing the outcome definition or the follow-up horizon changes \(Y^a(\tau)\) itself. A change in an estimate is not automatically evidence that the analysis is unreliable: the alternative may be asking a different causal question.

For example, suppose a stricter eligibility rule selects a subset of a broader eligible population. The broader risk under strategy \(a\) can be written as a weighted average of the risk in that subset and the risk among those excluded by the stricter rule:

where \(w=P(E_{\text{strict}}=1\mid E_{\text{broad}}=1)\), provided strict eligibility is nested within broad eligibility. The equation follows by partitioning the broad eligible group into those meeting the strict rule and those not meeting it, then applying the law of total probability. Unless the two groups have the same outcome risk under each strategy, the broad and strict risks—and potentially their contrast—can differ. This is a population change, not necessarily an estimation error.

A causal interpretation of any estimate still requires the assumptions appropriate to the design and analysis, including valid treatment assignment, adequate control of confounding where needed, consistency, and sufficient support. Comparing operational definitions does not establish these assumptions. It asks a narrower question: what does this emulation produce under alternative defensible translations of the protocol into data?

Worked Alternatives for Exposure, Outcome, and Follow-Up

Diagnostics, Reporting, and Common Pitfalls

Useful diagnostics begin with implementation checks. For each scenario, report the number eligible, the number assigned to each strategy, the number with observed outcomes, and the amount of follow-up available. For nested eligibility rules, verify that the stricter group is contained in the broader group. For reclassified exposure, account for every person moved between groups. For outcome alternatives, check overlap and avoid double counting. For follow-up alternatives, confirm that event ascertainment and censoring rules are defined consistently at each horizon.

Large shifts in group size, outcome frequency, or follow-up should trigger investigation of data extraction and rule implementation as well as substantive interpretation. If an alternative excludes many people because a required field is missing, the difference may reflect data availability rather than the intended clinical definition. If a scenario cannot be applied to a meaningful portion of the cohort, report that limitation rather than treating the result as a clean comparison.

A strong report states the primary rule, gives the rationale for each alternative, shows the corresponding cohort and outcome summaries, and interprets each estimate as belonging to its own operationalized question. Prespecification should include which scenarios will be run and how they will be compared. If several definitions are uncertain, a small, scientifically motivated set is often more interpretable than every possible combination. Joint scenarios are warranted when two ambiguities plausibly interact, but their implications should be described separately.

• Do not select definitions by result. Choosing the outcome rule that yields the largest apparent benefit turns a sensitivity analysis into outcome-driven analysis.
• Do not call every difference a failure of robustness. An eligibility change can alter the target population; a follow-up change can alter the estimand.
• Do not infer causal validity from stability. Similar estimates across definitions do not rule out confounding, selection bias, measurement error, or violations of other causal assumptions.
• Do not confuse protocol sensitivity with a correction for bias. These comparisons expose dependence on operational choices; they do not by themselves correct misclassification or quantify unmeasured bias.

The main alternatives serve different purposes. A single primary protocol offers a focused answer to one prespecified question but may hide dependence on ambiguous records. A protocol sensitivity analysis shows how specified operational alternatives change the emulation. Quantitative bias analysis instead requires assumptions about a bias mechanism and quantifies its possible impact; it is not a substitute for defining the trial. When a definition is not genuinely ambiguous, repeatedly varying it can obscure the primary question rather than strengthen the analysis.