From a Study Description to an Executable Specification
A protocol can sound precise while leaving consequential choices to whoever writes the code. “Include new users,” for example, does not say how far back to search for prior treatment, which records count as treatment, or how to handle a dispensing on the index date. If two analysts resolve those details differently, they may assemble different cohorts and answer different questions while claiming to implement the same study.
A reproducible emulation protocol therefore has two connected forms: a scientific specification of the target comparison and an operational specification that can be implemented against identified data. The operational rules should preserve the study question, not merely make the software run. A protocol is reproducible when another analyst can identify the exact version, inputs, rules, and analysis that produced a result, and can distinguish a faithful re-run from a substantive change.
This definition does not guarantee that the study is free of bias or that its data measure every concept well. Reproducibility concerns whether the stated design can be recovered and checked. Validity still depends on the design, data, and assumptions appropriate to the causal question.
Specify Decisions at the Level Where Code Can Apply Them
For each decision, write both the scientific meaning and the operational rule. A useful specification says what population or strategy is intended, how records identify it, the time period over which the rule applies, and what happens at boundary cases or with missing information. “Use a one-year washout” is not yet executable: the protocol must say which treatments count, how dates are compared, and whether the index date itself is in the washout window.
Version definitions, not just files. Give each protocol a stable identifier, version number, date, and change log. A change from “any prior dispensing” to “a prior dispensing in the past 365 days” changes cohort membership and may change the target question. It is not merely a code cleanup. By contrast, correcting a program bug so that it implements an already approved rule should be documented as an implementation correction, with evidence that the intended rule did not change.
Worked Example: Making a Washout Rule Executable
Suppose a hypothetical study compares initiation of Drug A with initiation of Drug B. The protocol defines a new initiator as someone with no dispensing of either drug during the 365 days before the index date. The index date is the first qualifying dispensing date. Define the lookback interval as the 365 calendar days before, but not including, that date. In date notation, a prior dispensing qualifies for the washout check when its date \(d\) satisfies \(t_0-365\text{ days} \leq d < t_0\), where \(t_0\) is the index date.
For an index date of 2024-04-09, the lower boundary is 2023-04-10. A dispensing on 2023-04-10 is exactly 365 days before the index date and is inside the washout interval. A dispensing on 2023-04-09 is 366 days before the index date and is outside that interval. This calculation accounts for February 29, 2024; treating the interval as “the same date last year” would produce the wrong boundary.
The executable rule is therefore: reject an otherwise eligible person if either drug has a recorded dispensing on any date from 2023-04-10 through 2024-04-08, inclusive. Do not reject solely for a dispensing on 2023-04-09, and do not count the index-date dispensing as a prior dispensing. An audit should test records on both boundaries, as well as a person with no dispensing in the interval. Those test cases check whether the code implements the wording rather than an analyst’s informal interpretation.
An implementation-ready eligibility rule should state the required baseline data, age calculation date, diagnoses or procedures used, exclusions, and handling of ambiguous or missing values. Treatment rules should identify the qualifying event, whether one or multiple records establish initiation, and how same-day records are resolved. Time zero must be defined consistently with treatment assignment and follow-up. When eligibility can recur, the protocol must say whether each eligible date creates a separate trial entry or only the first one does.
Preserve Provenance and Make the Analysis Recoverable
Data provenance identifies where the analytic records came from and how they were transformed. Record the data custodian and source, database release or extract identifier, extraction date, covered population and calendar period, coding-system versions, linkage procedures, and known coverage limitations. Record the analytic dataset’s construction steps, software and package versions, code repository revision, and any random seeds or nondeterministic settings. If privacy rules prevent distributing data, provenance and executable code can still make the process auditable within an authorized environment.
The analysis plan should be specific enough that the protocol does not leave the primary result to post hoc choice. At minimum, state the target contrast and its units, outcome definition and time horizon, follow-up and censoring rules, primary estimator, covariates or other inputs, uncertainty interval, and planned handling of missing data. Also distinguish the primary analysis from sensitivity analyses, and specify which alternative definitions each sensitivity analysis changes. Protocols for ambiguous definitions can be especially useful when they identify in advance which reasonable alternatives will be examined.
Worked Example: Recording Data Provenance
Consider a hypothetical analysis using a health-system electronic record and a linked vital-status file. A useful provenance record might identify the electronic-record extract as release H2025.03, extracted on 2025-04-15, and state that diagnosis codes follow the named coding-system release used by the system. It would also identify the vital-status file release, linkage method, covered dates, and the code revision that joined the files.
The record should distinguish what is known from what is assumed. For example, the team may know the extract date and the date coverage of the electronic-record system, but only have a data-custodian statement that outpatient encounters are nearly complete. That coverage statement is not proof of completeness. It belongs in a limitation and should prompt a defined check, such as comparing encounter counts over time or reviewing documentation of known system changes.
Suppose the protocol defines an outcome as the first inpatient admission with a specified diagnosis code during 180 days after index. The provenance record should point to the versioned code list, explain whether the code must be primary or may appear in any diagnosis position, and identify the field used to establish admission date. “Hospitalization was identified in the records” is not enough to reproduce the outcome definition.
Worked Example: Specifying the Primary Analysis
A hypothetical protocol compares Drug A initiation with Drug B initiation for the 180-day risk of a defined acute hospitalization. It specifies the contrast as the adjusted risk difference, Drug A minus Drug B, in percentage points. It defines follow-up as the 180 days beginning on index date, with the outcome counted if it occurs on or before day 179 after index. It identifies the diagnosis-code list and admission-date field, specifies the prespecified baseline covariates, and names the primary adjustment procedure and the method for obtaining a 95% confidence interval.
The protocol also says how records with incomplete outcome capture are handled, when follow-up ends, and which sensitivity analyses alter the outcome definition or the follow-up horizon. These details matter because changing the time horizon changes the risk being compared; changing the code list can change which admissions qualify. The analysis plan should not simply name a software package or say “adjust for confounding.” It should describe the actual inputs and operations sufficiently for an analyst to implement the same primary analysis.
Before analysis, the team checks that each required covariate is available before index, that dates can establish the 180-day window, and that the outcome code list is present in the pinned code release. If any element cannot be implemented as written, the protocol is revised and versioned before the primary result is generated, rather than silently replacing the intended analysis with a convenient one.
A Short Justification for Versioning and Audit Tests
The reason to pin definitions and inputs can be expressed as a simple reproducibility condition. Let \(v\) denote the protocol version, \(D\) the identified data release, and \(C\) the code revision. The analysis output is a function of those inputs and the specified computational environment:
If the protocol, data, code, and relevant environment are held fixed, a deterministic implementation should return the same cohort and analysis output, apart from explicitly documented numerical tolerances. If a rule changes, then \(v\) changes; if a data extract changes, then \(D\) changes. In either case, a difference in \(R\) cannot be attributed automatically to random fluctuation or a software error: the changed input must be examined. This is why version identifiers and provenance are part of the scientific record, not administrative decoration.
Audit tests connect the written rules to the implementation. A test case should be designed from a boundary or ambiguity in the protocol, with an expected result specified before inspecting the code output. For the washout rule above, the records dated 2023-04-09 and 2023-04-10 must produce different eligibility results. If both are treated alike, either the rule or the implementation is wrong and needs resolution before cohort construction.
Keep Unresolved Choices Visible
A protocol may be complete enough to proceed while still containing genuine uncertainty. The remedy is not to hide the uncertainty in a footnote or let an analyst make an unrecorded choice. Maintain a decision register with a unique item identifier, the competing options, the current status, the responsible decision-maker, a decision deadline, and the planned analysis consequence. If the choice remains unresolved, say whether analysis is paused, whether a prespecified sensitivity analysis will address it, or whether the uncertainty limits interpretation.
Worked Example: Registering an Unresolved Definition
Suppose a hypothetical diagnosis is recorded in two places: a problem list and an encounter diagnosis field. The protocol team has not established whether either source alone should qualify a prior diagnosis for eligibility. The register should identify the exact question, list the candidate rules, and state the evidence needed to choose between them, such as source documentation or a validation sample.
A useful entry might say: “D-04; unresolved: eligibility diagnosis from problem list only, encounter diagnosis only, or either source; owner: clinical coding lead; decision due before cohort extraction; consequence: cohort membership may differ; if not resolved by the deadline, use the prespecified primary definition and report the alternative-source rule as a sensitivity analysis.” The entry does not pretend that the alternatives are equivalent. It makes the uncertainty traceable and limits discretion after results are known.
The record should also separate decisions from assumptions. A decision is a choice among operational definitions, such as which diagnosis field to use. An assumption is a condition needed to interpret the resulting analysis, such as the assumption that the data capture the relevant treatment dispensings. Both should be written down, but they call for different responses: decisions need governance and versioning; assumptions need justification, diagnostics where possible, and appropriately cautious interpretation.
Diagnostics, Alternatives, and Limits
Several checks can reveal when an emulation is no longer following its protocol. Compare the implemented flow counts with the expected sequence of eligibility, exclusions, treatment assignment, and follow-up. Inspect date ranges, impossible or reversed intervals, duplicate records, missingness, and abrupt shifts around data-system changes. Confirm that the index date precedes outcome follow-up, that baseline variables are measured before index, and that treatment classification uses only the events allowed by the written rule. Re-run boundary test cases after code changes.
A narrative protocol is useful for explaining the scientific rationale, but it often leaves computational details implicit. An executable specification makes those details testable, but code alone can obscure why a rule was chosen. The strongest record keeps both: clear scientific prose and code-linked operational definitions. A registration or timestamped protocol helps show what was planned before outcome analysis; version control helps show how files changed. Neither substitutes for the other, and neither proves that the chosen design is unbiased.
Reproducibility also has limits. Data may be corrected or withdrawn, software environments may become unavailable, and a code list may fail to capture the intended clinical concept. A deterministic program can faithfully reproduce a poorly specified or invalid analysis. Versioning does not resolve substantive ambiguity, establish data quality, or guarantee causal identification. Its value is that it makes choices inspectable, makes changes legible, and supports a disciplined distinction between the planned study and later revisions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A protocol says “exclude recent users” but names neither the washout length nor which treatments count. Why is this not executable, and what information must be added?
- For an index date of 2024-04-09, does a dispensing dated 2023-04-10 fall within a 365-day pre-index washout? What about a dispensing dated 2023-04-09?
- A programmer changes the diagnosis code list after the first results are produced. What should be recorded to distinguish an implementation correction from a substantive protocol change?
- Name four items that belong in a data-provenance record and explain why recording only the database name is insufficient.
- A data field needed for a prespecified covariate is unavailable in the identified release. What is the reproducible response?
Solutions. (1) “Recent” has no fixed time boundary, and “users” does not identify the treatment records that count. Specify the washout duration, date boundaries, included treatment codes or concepts, and the rule for missing or conflicting records. (2) The 2023-04-10 dispensing is 365 days before index and is inside the defined interval; 2023-04-09 is 366 days before index and is outside it. (3) Record the old and new code lists, protocol and code versions, reason for the change, whether the intended clinical definition changed, and which results were generated under each version. If the definition changes, treat it as a substantive versioned change, not merely a code correction. (4) Examples include the extract identifier and date, source and covered population, coding-system version, linkage method, and code revision. The database name alone cannot identify which release, coding rules, or transformations produced the analytic records. (5) Do not silently omit or replace the covariate. Document the incompatibility, assess whether an alternative release can supply it, and decide and version any revised analysis before proceeding; report the deviation and its implications if the planned analysis cannot be carried out.