Estimand Drift From Changing Eligibility Criteria

Why Eligibility Changes Can Change the Question

A trial’s eligibility criteria do more than determine who may enroll. They help define the population to which the treatment effect refers. If a protocol is revised during an adaptive trial so that a baseline criterion becomes more restrictive, later participants may come from a different prognostic group than earlier participants. The treatment contrast can then change even when the treatments, endpoint, and follow-up remain unchanged.

This is a question about the target estimand, not automatically a flaw in randomization. Within each eligibility-defined group, valid randomization can still support a valid treatment comparison. The concern is that combining stages or reporting one overall effect may answer a different population-level question from the one originally specified.

A revised rule, \(E_1\), can define a different population from the original rule, \(E_0\). The corresponding effects, \(\Delta_{E_0}\) and \(\Delta_{E_1}\), need not be equal. A baseline characteristic can be prognostic if it predicts the outcome, and predictive of treatment effect if the treatment contrast differs across its values. A characteristic may be prognostic without being predictive on a particular effect scale.

The key comparison is therefore not simply whether the new criterion selects healthier or sicker participants. Ask whether the criterion changes the distribution of treatment effects relevant to the chosen estimand. The earlier material on standardizing effects to a fixed population provides one way to express a contrast for a specified baseline distribution; here the focus is diagnosing what eligibility revisions do to that distribution.

How a Prognostic Criterion Can Alter the Marginal Effect

Suppose a baseline characteristic partitions the original target population into strata \(g=1,\ldots,G\). Let \(w_g=P(g\mid E_0)\) be the proportion in stratum \(g\) under the original criteria, and let \(\Delta_g=E[Y(1)-Y(0)\mid g,E_0]\). When the strata partition the target, the law of total expectation gives:

If the revised rule changes which strata are eligible, or changes their relative representation, it changes the weights in this average. The effect changes when the revised weights put different emphasis on strata with different treatment effects. Even if the treatment effect stays the same, the population’s average outcome under either condition may change because the criterion selects people with a different prognosis.

This decomposition requires a well-defined treatment condition and outcome for each person in the target groups, and a consistent outcome definition and follow-up across stages. Baseline strata must be defined using measurements made before treatment assignment. For a causal interpretation within an eligibility-defined population, randomization must be implemented there as planned, and the outcomes must correspond to the specified treatment conditions. If the revised rule selects people outside the original target’s support, an effect for the original target cannot be recovered from the new group alone without additional assumptions or data.

When Prognosis Changes but the Effect Does Not

A criterion’s prognostic value alone does not establish effect drift on every scale. If the treatment effect is constant across the selected baseline strata on the estimand’s scale, reweighting those strata leaves the marginal effect unchanged on that scale. This distinction matters: a changed outcome distribution is not, by itself, evidence that the treatment contrast changed.

Auditing a Trial With Two Eligibility Versions

When eligibility changes after an interim decision, it is useful to describe each version explicitly rather than treating the protocol as if it had one timeless rule. Record the baseline criteria, the date each version took effect, which participants were screened under each version, and how many met each rule. Then examine whether the revised rule changes the distribution of prognostic baseline characteristics among those eligible.

The following calculation shows why a pooled contrast can be valid for its own combined population while failing to represent the original target. Assume the first-stage participants reflect the original 60% moderate and 40% high-severity mix in the first example, and the later stage enrolls only high-severity participants. For simplicity, suppose the two stages contribute equal numbers of participants and have the same treatment allocation proportions.

Diagnostics, Alternatives, and Limitations

Useful diagnostics begin with the protocol and enrollment process, not with a single significance test. Compare the old and revised eligibility rules in operational terms, identify which baseline ranges were removed or added, and summarize the characteristics and screening outcomes of people affected by the revision. Within each eligibility version, examine the distribution of the prognostic criterion and other important baseline variables. A large shift is evidence that the target composition changed; it does not by itself quantify the treatment-effect shift.

Where data permit, report treatment contrasts by relevant baseline strata and show the population weights used to obtain each marginal contrast. Check whether the treatment and outcome definitions, follow-up, and outcome handling remain comparable across stages. If a subgroup is absent under the revised rule, its treatment effect cannot be estimated from that stage. Sparse data or poor overlap can also make a standardized estimate unstable. Such limitations should be reported rather than hidden by a pooled summary.

There are several distinct responses to eligibility drift. Their suitability depends on the scientific question and the data available:

A common mistake is to say, “Randomization was preserved, so the overall treatment effect still applies to the original population.” Randomization protects the comparison within the population enrolled under each rule; it does not make a population recruited under a stricter rule representative of people excluded by it. The correction is to state the target population for each contrast and, if an original fixed target is needed, use data and a justified method that actually represent that target.

Another limitation is that eligibility can change for reasons beyond a measured prognostic variable. If revisions also alter treatment versions, follow-up, or the outcome definition, the observed difference cannot be attributed to population composition alone. An eligibility audit should therefore document what else changed and avoid interpreting every stage difference as evidence of treatment-effect modification.