Why Audit Selection Before Estimation?
A target-trial protocol is not protected from selection bias merely because it names an eligibility rule, treatment strategies, and an outcome. The crucial audit question is which people contribute to each comparison, and what information determines whether they contribute? If entry into the analysis depends on treatment, prognosis, or events that occur after follow-up begins, the analyzed groups may no longer represent the groups defined at baseline.
This audit focuses on four recurring risks: eligibility assessed after baseline, time zero defined differently by treatment group, outcomes captured differently across groups, and exclusions based on future events. These are protocol-level issues. A sophisticated analysis cannot automatically repair them; first determine whether the protocol is estimating the intended contrast in the intended population.
Let \(A\) denote the treatment strategy assigned at a common baseline, and let \(Y^a\) be the outcome that would occur over a specified follow-up horizon under strategy \(a\). A simple target risk difference in the baseline-eligible population is \(E[Y^1]-E[Y^0]\), where \(Y^a\) is coded 1 if the outcome occurs and 0 otherwise. A protocol that removes people based on post-baseline information may instead compare risks in treatment-specific selected populations. Those quantities need not equal the target contrast.
The audit is not a substitute for assumptions needed for causal interpretation. Even with a well-aligned protocol, causal identification requires, among other conditions, well-defined strategies and outcomes, consistency, no interference, positivity, and adequate control of confounding for the observational data. The audit asks whether the protocol has created additional selection problems before those identification questions are addressed.
A Four-Part Audit
Write each eligibility condition and ask whether it can be determined using information available by the protocol’s time zero. A condition involving later survival, treatment response, adherence, or absence of an outcome is post-baseline.
For each strategy, identify when a person becomes eligible, when strategy assignment is defined, and when outcome follow-up begins. A treatment group defined by later initiation may receive guaranteed event-free time before group membership is known.
Check whether the same outcome definition, data sources, linkage rules, and observation horizon apply to both strategies. Distinguish an outcome not recorded from an outcome known not to have occurred.
For each exclusion, record when the determining event occurs and whether it is affected by treatment or prognosis. An exclusion based on later adherence, switching, survival, or a future clinical event changes who remains in the analysis.
A practical audit table can list each rule, the time at which it is evaluated, the people it removes, and the possible consequences for the target population and treatment-outcome comparison. The protocol should make these items explicit rather than leaving them to undocumented data-processing decisions.
Worked Examples of Protocol Selection
Worked Example: Eligibility That Depends on Early Outcomes
Suppose a hypothetical trial protocol enrolls 100 people to each of two strategies at day 0, when all meet the stated baseline eligibility criteria. The outcome is followed for one year. The completed analysis, however, excludes anyone who has the outcome during the first 14 days and then calculates risk among those remaining. In strategy 1, 4 people have an early outcome and 5 of the remaining 96 have an outcome later in the year. In strategy 0, 2 have an early outcome and 6 of the remaining 98 have a later outcome.
The protocol’s selected-sample risks are \(5/96=0.05208\), or 5.21%, for strategy 1 and \(6/98=0.06122\), or 6.12%, for strategy 0. The selected-sample risk difference is \(0.05208-0.06122=-0.00914\), or -0.914 percentage points.
But if the target outcome is any event during the full year from baseline, the risks in the original baseline groups are \((4+5)/100=0.0900\), or 9.00%, and \((2+6)/100=0.0800\), or 8.00%. Their risk difference is \(0.0900-0.0800=0.0100\), or 1.00 percentage point. The arithmetic shows that the analyzed contrast differs from the baseline-group contrast and even has the opposite sign in this hypothetical example.
The repair is not to call the early events “ineligible”: all 200 people met eligibility at day 0. If the question concerns one-year risk from baseline, retain early outcomes in that outcome definition. If the scientific question genuinely begins after day 14, define the target population as people alive and outcome-free at day 14 and specify that later landmark as time zero for both strategies. That is a different estimand and population, not a harmless data-cleaning choice.
Worked Example: Treatment Group Defined by Later Initiation
Consider a hypothetical cohort of 200 people eligible on day 0, followed through day 180. The analysis labels 100 people “treated” because they initiate a medication by day 30; all 100 initiators remain outcome-free until they initiate. The other 100 are labeled “untreated” because they do not initiate by day 30; 10 of these people have the outcome before day 30 and 5 have it later. Among the 100 initiators, 8 have the outcome after initiation and before day 180.
The reported risks, using the explicitly stated group sizes, are \(8/100=0.0800\), or 8.00%, among initiators and \((10+5)/100=0.150\), or 15.0%, among noninitiators. The reported risk difference is \(0.0800-0.150=-0.0700\), or -7.00 percentage points.
This comparison does not establish that treatment reduced risk. To enter the initiator group, a person had to remain outcome-free until starting treatment. A person who had an early outcome could not later become an initiator under this classification, so that person necessarily entered the noninitiator group. This guaranteed event-free interval creates a treatment-defined time zero and a selection mechanism tied to the outcome.
A protocol repair aligns eligibility, strategy assignment, and follow-up at a common time zero. Depending on the question, the protocol might compare initiation on day 0 with no initiation on day 0, or specify a grace period and use a design that accounts for the resulting treatment strategies. Simply comparing “ever initiates by day 30” with “never initiates by day 30” from day 0 leaves the timing problem unresolved.
Worked Example: Different Outcome Capture Across Strategies
Suppose a hypothetical one-year comparison has 100 people in each baseline strategy group. A validation process establishes that 20 people in strategy 1 and 10 people in strategy 0 truly experienced the outcome. The routine data source records 16 of the 20 strategy-1 outcomes but 9 of the 10 strategy-0 outcomes. Assume, for this illustration, that there are no false-positive records.
Using routine records alone, the observed risks are \(16/100=0.160\), or 16.0%, and \(9/100=0.0900\), or 9.00%. The observed risk difference is \(0.160-0.0900=0.0700\), or 7.00 percentage points. The validation information gives risks of \(20/100=0.200\), or 20.0%, and \(10/100=0.100\), or 10.0%, for a difference of 10.0 percentage points.
The difference arises because outcome capture is 80.0% in strategy 1 \((16/20)\) and 90.0% in strategy 0 \((9/10)\). In this particular example, the recorded difference is smaller than the difference based on validated outcomes. Unequal capture does not always move an estimate in the same direction; its effect depends on the capture process and underlying risks.
The audit should compare outcome sources, follow-up completeness, coding rules, and linkage success by strategy. A recorded absence is not evidence of no outcome when the person was not observable. If capture differs, options include improving measurement, explicitly defining an outcome available equally in both groups, or using an appropriate missing-outcome or censoring analysis under stated assumptions. Merely treating all unrecorded outcomes as zero does not solve the problem.
Worked Example: Exclusion Based on Future Adherence
A hypothetical protocol assigns 100 people at baseline to each of two medication strategies and follows outcomes for one year. The completed analysis excludes anyone who discontinues the assigned strategy during the first 90 days. In strategy 1, 20 people discontinue; 5 of them experience the outcome during the year. Among the 80 retained, 12 experience the outcome. In strategy 0, 10 discontinue; 2 experience the outcome, while 9 of the 90 retained experience it.
Among retained participants, the risks are \(12/80=0.150\), or 15.0%, and \(9/90=0.100\), or 10.0%, giving a selected-sample difference of 5.00 percentage points. In the original baseline groups, the specified outcome counts give risks of \((12+5)/100=0.170\), or 17.0%, and \((9+2)/100=0.110\), or 11.0%; the difference is 6.00 percentage points.
The retained-sample comparison describes people selected by a future adherence event, not everyone assigned to the baseline strategies. If discontinuation is related to prognosis or treatment response, the selected groups can differ in those factors. A per-protocol question may be scientifically useful, but simply deleting later discontinuers does not generally identify it. A suitable analysis must address deviations and their causes under explicit assumptions; otherwise, the protocol should retain its baseline strategy definition and state that it estimates a different treatment-policy contrast.
Why Conditioning on Selection Can Create Bias
The basic mechanism can be shown without assuming that treatment initially differs by prognosis. Let \(U\) be a binary prognostic factor, with \(P(U=1)=P(U=0)=0.5\), and suppose treatment \(A\) is initially independent of \(U\). Let \(S=1\) indicate inclusion in the completed analysis. Suppose selection probabilities are \(P(S=1\mid A=0,U=0)=0.9\), \(P(S=1\mid A=0,U=1)=0.3\), \(P(S=1\mid A=1,U=0)=0.3\), and \(P(S=1\mid A=1,U=1)=0.9\).
Among selected people with \(A=0\), the probability of \(U=1\) is
For \(A=1\), the corresponding probability is
Now suppose treatment has no effect on the outcome, but outcome risk is 0.50 for \(U=1\) and 0.10 for \(U=0\). The selected risk for \(A=0\) is \(0.25(0.50)+0.75(0.10)=0.200\). For \(A=1\), it is \(0.75(0.50)+0.25(0.10)=0.400\). The selected contrast is 20.0 percentage points even though treatment has no effect in this construction. Selection has made the prognostic composition of the treatment groups differ.
This calculation illustrates why the relevant question is not merely whether exclusions occurred. Ask whether selection probabilities vary with treatment and outcome-related factors. Selection that is independent of relevant variables may reduce precision without inducing this particular bias; selection connected to both treatment and prognosis can distort the contrast.
Diagnostics, Repairs, and Limitations
A protocol review should produce an auditable timeline for each strategy: eligibility assessment, treatment assignment or initiation, follow-up start, outcome ascertainment, censoring, and every exclusion. Then compare those timelines side by side. Look for treatment groups that require surviving or remaining event-free long enough to qualify, unequal observation windows, and exclusions determined using data recorded after baseline. Report the count and timing of exclusions by strategy where possible.
| Approach | What it addresses | Key limitation |
|---|---|---|
| Redesign with common baseline and baseline eligibility | Prevents group membership from depending on post-baseline events when the target question permits it | May require changing the strategy definition or target population |
| Landmark analysis | Defines a common later time zero for people meeting explicitly stated landmark conditions | Targets people who reach that landmark; does not answer the original baseline-population question |
| Modeling selection or censoring | Can address measured predictors of selection or loss of outcome observation | Requires adequate measurement, positivity, correct modeling, and assumptions about conditional selection or censoring |
| Complete-case analysis | Uses records with observed outcomes | Can be biased when observation depends on treatment or outcome-related variables; it is not automatically valid |
These approaches are not interchangeable. A redesign changes the protocol to better match the intended question. A landmark analysis deliberately changes the eligible population and time zero. Weighting or other adjustment may address selection only when the necessary predictors are measured and the relevant conditional assumptions hold. No statistical method can recover information about unobserved outcomes without assumptions, and lack of overlap in selection probabilities can make an adjustment unstable or impossible.
Common Mistakes
- Calling a future condition baseline eligibility. “Eligible if no outcome occurs in the first two weeks” is not a baseline rule; it uses future information. Define a later landmark population if that is the intended question.
- Starting follow-up at baseline but classifying treatment by later initiation. This can give initiators guaranteed event-free time. Align time zero and treatment strategy instead.
- Treating no record as no outcome. First establish that the person was observable and that capture rules were comparable.
- Deleting nonadherent participants and calling the result per protocol. The deletion selects on a future event. Per-protocol interpretation requires an appropriate strategy for deviations and assumptions, not just exclusion.
- Assuming every selection issue can be fixed by adjustment. Adjustment cannot compensate for unmeasured selection factors or absent support without additional assumptions.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A protocol excludes people who experience the outcome in the first week after treatment assignment. Is this a baseline eligibility rule? What is one defensible way to frame the question if investigators truly want to study risk after that week?
- In a hypothetical cohort, 120 people initiate treatment by day 30 and all are outcome-free until initiation. Another 80 people do not initiate, including 8 with an outcome before day 30. Why is a comparison from day 0 vulnerable to selection bias?
- In a hypothetical group of 200 people, 30 outcomes occur, but only 24 are captured. What capture proportion is observed? Why is this insufficient to establish unbiased outcome risk?
- A one-year analysis excludes participants who discontinue treatment during the first month. What population does the retained-sample risk describe, and why is it not automatically a per-protocol causal effect?
Solutions. (1) No. The rule uses post-assignment information. If the scientific question concerns people who are alive and outcome-free at day 7, define those conditions for both strategies and set the common follow-up start at day 7; this targets that landmark population, not the original baseline cohort. (2) Group membership depends on later initiation, and initiators must remain outcome-free until initiation. The 8 early events can only be in the noninitiator group under the stated classification, so the groups have unequal guaranteed event-free time. (3) The observed capture proportion is \(24/30=0.800\), or 80.0%. Without knowing whether capture is related to treatment, prognosis, or outcome occurrence—and without knowing the unobserved outcomes—the proportion alone cannot establish that the recorded risk is unbiased. (4) It describes people who remained on their assigned treatment through the first month, selected by future adherence. If discontinuation is related to prognosis or treatment response, the retained groups may differ; simply excluding discontinuers does not establish the causal effect of sustained adherence.