Design Restrictions for Limited Overlap

When Eligibility Is Part of the Design

A treatment comparison may be difficult to support when some kinds of patients almost always receive one treatment and rarely receive the other. For example, patients with the most severe disease might all receive an intensive therapy, while less severe patients receive either option. The challenge is not simply how to analyze the data: a comparison for the full population requires information about both treatments across that population. A design restriction addresses this by defining a narrower population in which both options are relevant and represented.

This choice changes the question. Instead of asking about the effect for everyone who met the original broad criteria, the study asks about the effect for people who meet the new eligibility criteria. That can be a scientifically useful question, but the restricted effect should not be presented as though it automatically applied to the excluded patients.

Definition: Let \(A\) indicate treatment (\(A=1\) or \(A=0\)), \(X\) denote baseline covariates, and \(Y(1)\) and \(Y(0)\) denote the potential outcomes under each treatment. A design restriction defines eligibility using a rule \(S(X)\), where \(S(X)=1\) means eligible. The average treatment effect in the restricted population is \(E[Y(1)-Y(0)\mid S(X)=1]\).

The rule should use characteristics measured before treatment, such as baseline disease severity, age, or clinical history. It should describe a population in which both treatments are meaningful options. A rule based on treatment received or on outcomes observed after treatment does not define eligibility in this way and can create a different selection problem.

What Positivity Requires After Restriction

Restriction does not make positivity automatic. It changes the population within which positivity must hold. In the restricted population, for each covariate pattern occurring with positive probability, there must be a positive probability of receiving either treatment:

0 < P(A=1\mid X=x,S(X)=1) < 1

The probability of receiving treatment 0 is then also positive. In observational studies, the usual causal identification conditions must also hold in the restricted population: consistency (the observed outcome equals the potential outcome under the treatment actually received) and conditional exchangeability (within levels of \(X\), treatment assignment is independent of the potential outcomes). Under those assumptions and positivity, the restricted-population effect can be identified by standardizing conditional outcome means over the covariate distribution among eligible people:

E[Y(1)-Y(0)\mid S(X)=1] = E\left[ E(Y\mid A=1,X,S(X)=1) - E(Y\mid A=0,X,S(X)=1) \mid S(X)=1 \right].

The formula makes the target explicit: the outer expectation averages over covariate patterns in the eligible population, not over the original population. The restriction addresses a design limitation only if it removes patterns without comparable treatment options while retaining a population that answers a useful question.

Worked Example: Excluding a Stratum With No Treatment Comparators

Suppose a hypothetical observational study includes 500 patients in three baseline severity groups. In the severe group, all 100 patients receive treatment 1. In the moderate group, 100 of 200 receive each treatment. In the mild group, 40 of 200 receive treatment 1 and 160 receive treatment 0. Consider restricting eligibility to mild and moderate patients.

Check the original design. In the severe group, \(P(A=0\mid X=\text{severe})=0\) in the observed data. There is no treatment-0 group with which to compare severe patients. The mild and moderate groups each contain patients under both treatments.

Apply the eligibility rule. Define \(S(X)=1\) for mild or moderate severity and \(S(X)=0\) for severe severity. The restricted population contains \(200+200=400\) patients, so the retention proportion is \(400/500=0.80\), or 80%. It contains \(40+100=140\) treated patients and \(160+100=260\) control patients. The overall treatment proportion after restriction is \(140/400=0.35\).

Check the relevant support. Within mild patients, the observed treatment proportions are \(40/200=0.20\) and \(160/200=0.80\). Within moderate patients, they are \(100/200=0.50\) and \(100/200=0.50\). Both treatments are represented in both retained severity groups. This is a useful support check, though it does not establish exchangeability or prove support for every other covariate pattern.

Interpret the target. Any identified effect now describes mild and moderate patients who meet the rule, not all 500 patients. In particular, the restriction gives no direct estimate of the effect for the excluded severe patients.

Deriving the Restricted-Population Estimand

The change in estimand follows directly from conditioning on eligibility. Suppose consistency and conditional exchangeability hold among eligible patients, and positivity holds for their covariate patterns. For \(a\in\{0,1\}\), exchangeability allows the potential-outcome mean within a covariate pattern to be represented by the observed mean among eligible patients receiving \(a\):

E[Y(a)\mid X=x,S(X)=1] = E[Y\mid A=a,X=x,S(X)=1].

Average this equality over the covariate distribution among eligible patients for each treatment level, then subtract the treatment-0 mean from the treatment-1 mean. By the law of iterated expectations, the result is:

\begin{aligned} E[Y(1)-Y(0)\mid S(X)=1] &=E[E[Y(1)\mid X,S(X)=1]-E[Y(0)\mid X,S(X)=1]\mid S(X)=1]\\ &=E[E[Y\mid A=1,X,S(X)=1]-E[Y\mid A=0,X,S(X)=1]\mid S(X)=1]. \end{aligned}

This derivation also clarifies a key limitation. The restricted estimand is not the original population average with inconvenient observations removed from a calculation. It is a different population average, defined by the eligibility rule. The equality does not apply if exchangeability fails, and the conditional means cannot be compared where one treatment is absent.

Worked Example: Calculating an Effect for the Eligible Population

Continue the hypothetical mild/moderate study. Suppose the outcome is hospital days, with the following observed mean outcomes: among mild patients, the treatment-1 mean is 12 days and the treatment-0 mean is 13 days; among moderate patients, the corresponding means are 8 and 10 days. For this calculation, assume consistency, conditional exchangeability given the measured baseline covariates, and positivity in the eligible population.

The restricted population is half mild and half moderate: each group contains 200 of the 400 eligible patients. The standardized effect is therefore the weighted average of the two conditional mean differences:

\begin{aligned} \widehat{\tau}_{S=1} &=\frac{200}{400}(12-13)+\frac{200}{400}(8-10)\\ &=0.5(-1)+0.5(-2)\\ &=-1.5\text{ days}. \end{aligned}

Under the stated assumptions, this estimate describes an average reduction of 1.5 hospital days for the eligible mild/moderate population. The arithmetic does not provide an effect estimate for severe patients: that group was excluded, and the data supplied no treatment-0 outcomes for them. Nor does the calculation itself verify the causal assumptions.

Making Eligibility Rules Defensible

A defensible rule starts with the scientific question and clinical context, then uses baseline information to identify a population for which both treatments are plausible. The rule should be specific enough that another researcher could apply it without knowing treatment assignment or outcomes. It should also be checked against the treatment distribution: a seemingly reasonable clinical criterion may still leave a retained subgroup with no representation of one option.

In a prospective study, eligibility can be applied before treatment is assigned or chosen. This can prevent enrollment of people for whom a meaningful comparison cannot be made, while making the intended target population clear at the outset. In an analysis of existing data, the same kind of rule can define a restricted analytic population, but the investigator must report that the target is now conditional on meeting the rule. Either way, support should be examined across important covariate patterns, not only through an overall treatment proportion.

Worked Example: A Rule Based on Joint Clinical Characteristics

Consider a hypothetical study where treatment options are compared across joint baseline groups defined by age and severity. A broad eligibility rule includes 80 younger, lower-severity patients (40 per treatment), 120 older, lower-severity patients (60 per treatment), and 100 older, high-severity patients (100 receiving treatment 1 and none receiving treatment 0). A proposed restriction excludes the older, high-severity group.

The restricted group contains \(80+120=200\) patients out of \(300\), retaining \(200/300=0.667\), or about 66.7%. In the retained joint groups, both treatments are represented: the younger, lower-severity group has 40 in each arm, and the older, lower-severity group has 60 in each arm. The excluded group has no treatment-0 patients.

The calculation supports the proposed rule as a response to the stated joint-support problem. It also shows why checking age and severity separately can be insufficient: support in each marginal category does not guarantee treatment representation in every combination of categories. Whether the restriction is worthwhile still depends on whether an effect for younger and lower-severity patients is useful for the scientific question.

Diagnostics, Alternatives, and Limitations

Warning signs that a design restriction is not solving the intended problem include retained covariate groups with only one treatment represented, very few patients in one treatment arm within an important group, and eligibility criteria that are difficult to apply consistently. A rule may remove the most obvious unsupported subgroup yet leave sparse treatment cells in another covariate pattern. Check treatment counts and proportions within clinically important groups and assess the joint covariate patterns that matter to the comparison.

The main alternatives answer somewhat different questions. Common-support trimming excludes observations according to a defined support rule and therefore also changes the target population; design-based eligibility specifies the population through study criteria, ideally before treatment assignment. Matching or weighting can construct a comparison using observed participants, but cannot supply information for treatment options absent from a covariate pattern. Modeling may produce predictions in such regions, but predictions are not observed comparisons. These approaches are not interchangeable fixes: the appropriate choice depends on the target population, the available data, and the assumptions one is willing to make.

Restrictions have real costs. They reduce sample size and may exclude groups for whom the treatment decision is especially important. They can also limit generalizability: an effect for eligible patients need not equal the effect for excluded patients. Extending the result to a broader population requires additional evidence and assumptions; it does not follow merely from the fact that the eligible group was analyzed carefully.

Do not define eligibility using observed outcomes. Selecting patients based on a post-treatment result changes the comparison and can introduce selection bias.
Do not mistake a good overall treatment split for subgroup support. A 50/50 overall split can coexist with a covariate stratum containing only one treatment.
Do not claim the restricted result applies to those excluded. State the eligibility rule and name the population to which the effect pertains.

Key takeaway: A design restriction is a choice of target population, not a way to recover an effect for people who lack comparable treatment options. Define the eligible group using baseline characteristics, verify support within it, and report the resulting effect for that group.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a hypothetical study, all patients with a particular baseline condition receive treatment 1. What positivity problem does this create, and what could a restriction accomplish?
A rule retains 240 of 300 participants. What percentage is retained, and what must be checked beyond that overall percentage?
Why does restricting eligibility to a baseline-defined group change the causal estimand?
Can a good overall treatment split establish positivity within every joint covariate subgroup? Explain.

Solutions. (1) The condition-defined group has no observed treatment-0 patients, so a treatment comparison within that group is unsupported by the data. Excluding that group can define a narrower target in which both treatments are represented, if support holds across the retained covariate patterns. (2) The retained percentage is \(240/300=0.80\), or 80%. Treatment counts and proportions must also be checked within important subgroups and joint covariate patterns. (3) The effect is averaged over the covariate distribution among eligible people, so changing who is eligible changes the population over which the effect is defined. (4) No. An overall split can conceal a subgroup in which one treatment is absent; positivity must be considered within covariate patterns, not inferred from the marginal totals alone.