Why Examine Protocol Choices?
A target trial emulation requires decisions about who is eligible, how treatment strategies are defined, how long treatment may be initiated, how long outcomes are followed, and which events count as outcomes. These decisions are not merely analytic settings: they help define the causal question. Sensitivity analysis for protocol choices asks whether the substantive conclusion changes under other defensible specifications.
A changed estimate does not automatically show that one specification is biased or that the result is unreliable. For example, changing eligibility can change the target population, while extending follow-up can change the outcome from a short-term risk to a longer-term risk. The first task is therefore to state what each protocol estimates; only then can estimates be compared meaningfully.
This notation suppresses details such as censoring and treatment assignment mechanisms. Each protocol-specific estimate still requires an appropriate identification and estimation strategy: in particular, the relevant consistency, exchangeability, positivity, and outcome-observation assumptions must be plausible for that protocol. Changing a grace period can change the treatment strategies; it does not remove the need to define them precisely. The earlier material on grace periods, cloning, artificial censoring, and unequal outcome observation supplies the design concepts needed to specify those choices.
A useful sensitivity analysis begins with a primary protocol selected for substantive reasons and a small set of alternatives that are also clinically or operationally defensible. Vary one decision at a time first. If the conclusion changes, joint variations can help determine whether the change is driven by one choice or by an interaction among choices.
Compare Like With Like—And Name What Changed
For each alternative, report the protocol change, the resulting target population or outcome, and the corresponding estimates. If eligibility changes, estimates generally refer to different populations. If the horizon or outcome changes, the measured event probability changes. It is reasonable to show those estimates side by side, but the comparison should not be described as if it were a repeated measurement of one unchanged estimand.
When the purpose is to isolate one design choice, keep the others fixed and, where appropriate, estimate both protocols in a common population. For example, one can apply two outcome definitions to the same eligible participants and horizon. That makes the effect of changing the outcome definition easier to interpret. It does not make the outcomes identical: it holds the population and horizon constant while changing what counts as an event.
Worked Example: Narrowing the Eligibility Rule
Suppose the primary protocol includes adults with a recorded baseline measurement. Among 1,000 eligible people, the estimated one-year risks are 0.18 under strategy 0 and 0.12 under strategy 1. Thus \(\hat\Delta=-0.06\), a 6-percentage-point lower estimated risk under strategy 1. A defensible narrower rule requires an additional baseline clinical assessment. Among the 700 people meeting that rule, the estimated risks are 0.15 and 0.10.
For the narrower eligible population, \(\hat\Delta=-0.05\). The risk difference changes by \((-0.05)-(-0.06)=0.01\), or one percentage point toward zero. The estimated risk under strategy 0 changes by \(0.15-0.18=-0.03\), and under strategy 1 by \(0.10-0.12=-0.02\).
The risk difference remains negative under both rules, but the two risk differences concern different eligible populations. The changes could reflect different treatment effects across those populations, differences in estimation, or both. These summaries alone do not identify which explanation is responsible. Report the population change and check whether the narrower rule leaves adequate treatment overlap and outcome information.
Worked Example: Changing the Grace Period
Consider two well-defined strategies that differ in their permitted initiation window: initiate by day 30 versus initiate by day 60. Suppose each strategy is compared with the same specified alternative strategy, and each emulation uses a common one-year outcome horizon. Estimated one-year risks under the two strategies are 0.20 and 0.14 for the 30-day protocol, giving \(\hat\Delta_{30}=0.14-0.20=-0.06\). For the 60-day protocol, the estimates are 0.19 and 0.12, giving \(\hat\Delta_{60}=0.12-0.19=-0.07\).
The risk difference changes by \(-0.07-(-0.06)=-0.01\), one percentage point farther below zero. The strategy-specific risks also change: the comparator risk decreases by 0.01, while the other strategy's risk decreases by 0.02. Because the permitted initiation windows define different strategies, this is not simply a check of whether the same treatment effect was calculated twice. It asks about effects of alternative treatment strategies. Verify that initiation dates, assignment time, and follow-up are aligned under each specification; otherwise the comparison may reintroduce misclassification or immortal time.
Worked Example: Changing the Outcome Definition
In a common eligible population followed for one year, suppose the primary outcome is any hospitalization: 120 of 1,000 people under strategy 0 and 90 of 1,000 under strategy 1. The estimated risks are \(120/1000=0.12\) and \(90/1000=0.09\), so \(\hat\Delta=-0.03\).
Now define the outcome more narrowly as an unplanned hospitalization. Suppose 80 and 65 people, respectively, meet that definition. The risks are \(80/1000=0.08\) and \(65/1000=0.065\), giving \(\hat\Delta=-0.015\). The risk difference changes by \(-0.015-(-0.03)=0.015\), or 1.5 percentage points toward zero.
The narrower outcome has fewer events and a different meaning. The difference in risk differences is not evidence that one definition is correct; clinical relevance and reliable ascertainment determine which definition best addresses the question. Also check that outcome measurement is comparably complete across strategies: a narrower definition is not useful if it is recorded differently by treatment group.
Summarize the Change Along a Protocol Path
When several protocol components differ, a path through the specifications gives a transparent accounting of how the estimate changes. The numerical contributions depend on the order in which components are changed, so this calculation is descriptive rather than a unique causal attribution to each design decision.
Proof. Expand the right-hand side. Every intermediate term appears once with a positive sign and once with a negative sign:
This identity is useful for organizing a comparison, but it does not show that a component has an independent effect. If changing eligibility first changes who is included, then changing the outcome afterward is evaluated in that newly selected population. Reversing the order can yield different intermediate changes even though the final difference is the same. \(\square\)
Account for Sampling Uncertainty in the Comparison
Estimates from alternative protocols often use many of the same participants, so their sampling errors are dependent. A comparison should use the uncertainty of the difference, not treat the two estimates as independent by default.
Proof. Write the difference as a sum, \(\hat\psi_B-\hat\psi_A\). For any finite-variance random variables \(X\) and \(Y\), expanding the centered square gives
Substituting \(X=\hat\psi_B\) and \(Y=\hat\psi_A\) proves the result. \(\square\)
For example, suppose the two estimates have variances 0.0009 and 0.0004, and their covariance is 0.0003. The variance of their difference is \(0.0009+0.0004-2(0.0003)=0.0007\), with standard error \(\sqrt{0.0007}\approx0.0265\). Ignoring the covariance would instead give \(\sqrt{0.0013}\approx0.0361\). In practice, a suitable joint variance estimator or resampling procedure should preserve the shared-data dependence and repeat the protocol-specific estimation. The calculation does not make a meaningful difference in estimands into a common estimand; it quantifies uncertainty in the numerical contrast between estimates.
Diagnostics, Alternatives, and Limits
A useful comparison reports both effect measures and the information supporting them. For every specification, inspect the number of eligible participants, treatment-strategy adherence, outcome counts, follow-up completeness, and evidence of treatment overlap. The earlier material on design-based assessment of treatment overlap is relevant here: a protocol that sharply restricts eligibility or lengthens a grace period may leave some strategies poorly represented in parts of the covariate distribution. Estimates can then be unstable even if the overall sample size appears large.
Other warning signs include very small event counts, marked differences in outcome observation, and a result driven by a few observations with extreme influence. Record how estimates and uncertainty change, not just whether a significance threshold is crossed. A confidence interval spanning zero does not by itself mean that two protocol estimates agree; the uncertainty of their difference and the estimands being compared both matter.
The main alternative is to choose one protocol and report only its estimate. That can be appropriate when the question and design are firmly specified in advance, but it gives readers little information about dependence on reasonable design choices. A broad grid of every possible specification is not automatically better: it can obscure the primary question and encourage selective emphasis. A prespecified, clinically justified set of alternatives is more interpretable.
Sensitivity analysis cannot establish that an alternative protocol is correct, repair unmeasured confounding, or prove that a protocol-specific identification assumption holds. Nor is it appropriate to choose the specification that produces the most favorable estimate. State the rationale for each alternative, distinguish estimand changes from estimation variability, and describe limitations that remain under all specifications.
Check Your Understanding
Answer each question, then compare with the worked solutions below.
- A protocol changes eligibility and its estimated risk difference changes from \(-0.04\) to \(-0.02\). What must be clarified before interpreting that change?
- Two protocol estimates have variances 0.002 and 0.001 and covariance 0.0006. Calculate the variance and standard error of their difference.
- Why can changing a grace period change the causal question rather than merely test the stability of one estimate?
- Give two diagnostics that are especially important when an alternative eligibility rule leaves far fewer participants.
Solutions. (1) State how the eligibility rule changed and which population each estimate targets; then consider whether the assumptions and estimation quality are plausible in both populations. (2) The variance is \(0.002+0.001-2(0.0006)=0.0018\); the standard error is \(\sqrt{0.0018}\approx0.0424\). (3) The permitted initiation window is part of the treatment strategy, so different grace periods define different strategies. (4) Check treatment overlap and the number of outcome events; also examine follow-up and outcome ascertainment, since reduced eligibility can make each less adequate.