Why Audit a Protocol Before Estimating an Effect?
An emulation can use sophisticated analysis and still answer the wrong question if its protocol puts eligibility, treatment classification, and follow-up on different timelines. A basic audit asks whether a person could enter the target trial at a time when they were eligible, assigned to a strategy, and at risk of having the outcome counted. If these moments do not line up, the analysis may exclude early outcomes, select survivors, or compare groups observed over different periods.
This tutorial treats the draft protocol as a set of dated rules. The audit is not a substitute for defining the causal question or assessing confounding. It is a way to check whether the rules for time zero, eligibility, strategy classification, and outcome ascertainment describe a coherent trial.
Write the draft protocol as a timeline for an individual. Let \(t_0\) denote time zero, \(E(t_0)=1\) indicate eligibility at that time, and \(g\) denote a treatment strategy. Let \(Y(t)\) indicate that the outcome has occurred by time \(t\). Treatment may be initiated or changed after \(t_0\), but a baseline strategy group must not be defined by a person's future outcome, survival, or treatment history unless the protocol explicitly defines and handles a later assignment time.
Write down when every inclusion and exclusion criterion is assessed. Ask whether each criterion could be known at the proposed \(t_0\).
State what information determines a person's strategy group, and whether that information is available at \(t_0\) or only later.
Record when outcome counting starts and stops, including what happens after treatment changes, loss to follow-up, or competing events.
For each strategy, ask whether early outcomes can be counted and whether the same eligibility and follow-up rules apply.
Future-Defined Treatment Groups and Immortal Time
A common warning sign is a baseline group defined by what a person does in the future: for example, “initiated treatment within 30 days” versus “did not initiate.” If follow-up starts at \(t_0\), someone classified as an initiator must remain eligible and able to initiate until initiation. When an outcome or death prevents initiation, the interval before initiation is necessarily outcome-free for people later classified as initiators. Calling that interval exposed time gives the initiator group guaranteed event-free time.
Justification. The classification rule requires \(S<T\). Since initiation occurs after \(t_0\), the two inequalities give \(t_0<S<T\). Therefore the outcome time cannot fall in \([t_0,S]\). This is not evidence that treatment protected the person during the pre-initiation interval: treatment had not yet begun. The problem is the retrospective group definition and the allocation of that interval to the initiator group. \(\square\)
Worked Example: “Initiated Within 10 Days”
A draft study starts follow-up on day 0 and compares people who initiate treatment by day 10 with everyone else. Among 100 eligible people, 40 initiate on day 10. Of the remaining 60, 10 experience the outcome on days 1–9, so cannot initiate; 12 more experience the outcome after day 10. Among the 40 initiators, 4 experience the outcome after initiation.
Under the draft's retrospective classification, the initiator group has 4 outcomes among 40 people: \(4/40=0.10\), or 10%. The “did not initiate” group contains the 10 early outcomes as well as the 12 later outcomes: \(22/60\approx0.367\), or 36.7%. The initiator group is guaranteed to have no outcome before its day-10 initiation, because an early outcome rules out eventual initiation under this definition.
The arithmetic exposes the protocol problem; these crude percentages do not establish the size of a causal bias. The groups have been defined using future treatment and outcome-related survival. A coherent alternative would define strategies at baseline, including any grace period, and use an appropriate emulation design to represent those strategies. Simply moving the analysis start to day 10 would instead target people who remain eligible at day 10, a different population.
Post-Baseline Eligibility and Selection
An eligibility rule assessed after the proposed time zero can silently change who enters the comparison. Rules such as “survived 14 days,” “had a laboratory test in the next month,” or “remained enrolled through treatment initiation” use post-baseline information. If the outcome can occur during that interval, excluding people who experience it removes early outcomes from the target population. If eligibility is affected by prognostic factors, selection can also change the distribution of risk factors among those retained.
The distinction can be stated directly. Let \(S=1\) indicate passing a post-baseline eligibility rule, and let \(Y=1\) indicate an outcome by a fixed horizon. The full baseline population's risk decomposes as
This identity follows by partitioning the event \(Y=1\) into the disjoint events \(\{Y=1,S=1\}\) and \(\{Y=1,S=0\}\), then applying \(P(Y=1,S=s)=P(Y=1\mid S=s)P(S=s)\). If the protocol keeps only \(S=1\), it estimates a quantity conditional on passing the rule, not automatically the risk in the baseline-eligible population. In particular, if early outcomes imply \(S=0\), the selected risk excludes those outcomes by construction. \(\square\)
Worked Example: Eligibility Requires 14-Day Survival
A protocol says participants must be eligible on day 0, survive without the outcome through day 14, and then enter follow-up. There are 100 people eligible on day 0. Twenty experience the outcome by day 14; the other 80 survive outcome-free to day 14. Of these 80, 16 experience the outcome between day 14 and day 365.
For the original baseline population, the day-365 risk is \((20+16)/100=36\%\). Among those selected by the 14-day survival condition, the conditional risk of an outcome after day 14 is \(16/80=20\%\). These are different quantities: the latter excludes the 20 early outcomes and conditions on surviving outcome-free to day 14.
The protocol could deliberately define a trial among day-14 survivors, but then day 14 must be the stated time zero, eligibility must be assessed there, and the target population must be described as people who reach that point outcome-free. It cannot claim to estimate the day-0 risk in all 100 people from the selected 80 without additional assumptions and a design that addresses the selection.
Strategy-Dependent Follow-Up
The follow-up rules also need an audit. If outcome counting stops at treatment discontinuation for one strategy but continues for another, the groups have different observation windows tied to strategy. A contrast in observed outcomes can then reflect unequal opportunity to detect the outcome as well as any treatment effect. The issue is not that all strategies must have identical observed data in practice; it is that the protocol should specify a common outcome horizon and explain how deviations from observation are handled.
For each strategy, trace the start of follow-up, intended end, administrative end, and any rule for censoring or stopping outcome ascertainment. Check especially whether a treatment change is being confused with the end of follow-up. A person can deviate from a strategy while still being followed for outcomes, depending on the target trial question and analysis plan.
Worked Example: Outcomes Counted Only While Treatment Continues
A 12-month protocol compares two strategies, with 50 people assigned to each. In the first group, outcome follow-up continues for all 50 people through 12 months; 10 outcomes are observed. In the second group, outcome counting stops when treatment is discontinued. Fifteen people discontinue before 12 months, and 4 outcomes are recorded among the 35 whose outcomes continue to be counted.
The observed proportions are \(10/50=20\%\) and \(4/35\approx11.4\%\). The second denominator is not 50 followed for the same 12-month outcome horizon: 15 people have their outcome status after discontinuation uncounted under this rule. The proportions therefore do not compare the same ascertainment period. One cannot conclude from these numbers alone that the second strategy has lower 12-month risk.
The audit asks the protocol to specify what the intended outcome is after discontinuation. If the target trial follows everyone for 12 months regardless of treatment changes, the emulation needs a way to preserve that follow-up or account for missing outcome information. If the intended estimand instead concerns outcomes only while on treatment, that different question must be stated explicitly, including how the strategy-specific observation period is defined.
Compare the Draft With Plausible Alternatives
A protocol defect is not repaired merely by changing a date in the analysis. Instead, identify which trial the revised rules would represent. The main alternatives differ in their target populations and assignment moments.
| Draft feature | Audit question | Possible protocol clarification |
|---|---|---|
| Group defined by later initiation | Could an outcome before initiation prevent membership in the initiator group? | Define strategies at a common baseline with an explicit grace period and a design that handles the period before initiation. |
| Eligibility requires later survival or measurement | Is this a baseline criterion, or does it select people after follow-up should have begun? | Move time zero to the later eligibility assessment and describe the survivor population, or retain baseline eligibility and address the later process explicitly. |
| Follow-up ends differently by strategy | Are outcomes ascertained over a common horizon and under comparable rules? | Specify the same intended horizon and clarify how treatment changes, censoring, and unavailable outcomes are handled. |
These revisions answer different questions. For example, moving time zero to day 14 may remove the mismatch between eligibility and follow-up, but it changes the target to those alive and outcome-free on day 14. Conversely, keeping day 0 as time zero requires a design that does not erase early outcomes simply because future treatment is unknown at baseline. Neither choice is automatically preferable; the protocol must make its target population and strategy definitions explicit.
Common Audit Mistakes and Limitations
- Calling future behavior baseline assignment. “Ever treated” or “treated within 30 days” is not known at baseline. Correct the protocol's strategy definition or use a design that represents the later decision without assigning pre-treatment time as exposed.
- Assuming a later time zero is a harmless fix. Starting on day 14 changes eligibility and the target population. State that population rather than treating the date change as a purely technical repair.
- Equating treatment discontinuation with loss to follow-up. These are different events. A protocol should say whether outcomes remain observed after a treatment change.
- Reading a timeline as proof of causal validity. Temporal alignment can prevent specific design errors, but it does not establish exchangeability, positivity, consistency, accurate measurement, or absence of other selection processes.
A timeline audit is a diagnostic, not a guarantee. It can reveal impossible ordering, future-defined groups, excluded early events, and strategy-dependent observation, but it cannot by itself determine whether every confounder is measured or whether the chosen target trial is substantively appropriate.
Check Your Understanding
Answer each question by naming the timeline issue and stating what the protocol would need to clarify.
- A study starts follow-up on day 0 but labels people “treated” if they fill a prescription by day 21. What is the key risk if an outcome can occur before a fill?
- Eligibility requires a test result measured on day 30. What two coherent choices could the protocol make about time zero?
- In a study with a 1-year horizon, one strategy stops outcome counting at discontinuation while the other continues. Why is a direct comparison of recorded proportions problematic?
- Why does moving time zero from day 0 to day 14 not automatically recover the original day-0 target population?
Solutions. (1) If the outcome prevents a fill, people classified as treated must have remained outcome-free until filling. Counting their pre-fill time as treated creates immortal-time selection; define baseline strategies and handle the grace period appropriately. (2) The protocol can set time zero at day 30 and target people eligible then, or retain day 0 and avoid using the future test as an unqualified baseline eligibility criterion, specifying how that later information enters the design. (3) The groups have different outcome observation windows, so their recorded proportions mix outcome risk with unequal ascertainment; specify a common intended horizon and how outcomes after discontinuation are handled. (4) A day-14 start excludes people who did not survive outcome-free to day 14 and therefore targets a selected survivor population, not everyone eligible on day 0.