Design-Based Assessment of Treatment Overlap

Why Assess Overlap Before Comparing Outcomes?

A treatment comparison is most informative when people with similar, clinically important baseline characteristics are represented under both strategies. If nearly everyone with severe disease receives one strategy and nearly everyone with mild disease receives the other, an overall count showing equal numbers in each strategy does not solve the problem: the data contain little or no within-profile comparison. The gap matters at the design stage, before outcome differences are used to make a treatment claim.

Here, let $A=1$ denote one treatment strategy and $A=0$ the comparator, as defined by the emulation protocol. Let $G$ denote a prespecified baseline clinical profile, such as a combination of disease severity and age category. The profile must use information measured before strategy assignment and should reflect characteristics that clinicians consider relevant to treatment choice or to the population of interest.

Definition: For a profile $g$, let $n_g$ be the number of eligible people in that profile, $n_{g1}$ the number assigned to or initiating strategy $A=1$, and $n_{g0}$ the number assigned to or initiating strategy $A=0$, with $n_g=n_{g1}+n_{g0}$. The observed treatment frequencies are $\hat p_g=n_{g1}/n_g$ and $1-\hat p_g=n_{g0}/n_g$, when $n_g>0$. Observed overlap in profile $g$ means that both $n_{g1}>0$ and $n_{g0}>0$. It is thin when both strategies appear but one has few representatives relative to the comparison being planned.

This is a design-based assessment: it describes who is represented under each strategy across baseline profiles, not how outcomes differ. In a population formulation, treatment positivity for the profiles of interest requires $0<P(A=1\mid G=g)<1$. Sample frequencies are evidence about representation, not proof that the population probability is positive or zero. The earlier material on Feasibility Checks Before Emulation motivates checking whether the proposed study can be carried out; here the specific question is whether each clinically relevant profile includes people under both strategies.

Constructing and Reading the Profile Table

First specify the eligible population and the treatment definition from the protocol. Then choose a manageable set of clinically meaningful, pre-treatment characteristics and tabulate the number under each strategy in every resulting profile. Report counts alongside percentages: a frequency of 20% represents very different information when its denominator is 5 than when it is 500.

Profiles should not be invented after looking for a favourable result. They need to be clinically interpretable and detailed enough to reveal important treatment-selection patterns, while retaining enough people per cell to make the display useful. A highly refined cross-classification can create many tiny cells; an overly broad classification can merge people for whom treatment decisions differ substantially. This is a design choice to justify, not an automatic rule about how many categories to use.

Worked Example: Frequencies Across Severity Profiles

Suppose an eligible cohort includes 180 people, classified by baseline severity. The treatment counts are:

Baseline severity	Strategy $A=1$	Strategy $A=0$	Total	Frequency $A=1$
Mild	56	24	80	56/80 = 0.70
Moderate	30	30	60	30/60 = 0.50
Severe	8	32	40	8/40 = 0.20
Total	94	86	180	94/180 = 0.522

Both strategies have at least one representative in every severity profile, so there is no empty cell in this table. Representation is nevertheless uneven: only 8 of the 40 severe-profile participants received $A=1$, compared with 32 who received $A=0$. The overall frequency, $94/180$, does not communicate that difference. The appropriate design-stage conclusion is that the severe-profile comparison may be less well supported by the observed data than the mild- or moderate-profile comparisons. These counts alone do not establish whether the imbalance is acceptable for the intended analysis.

Worked Example: An Overall Split Hides a Missing Comparison

Consider 200 eligible people classified as either low or high baseline risk. All 100 low-risk people receive $A=1$, and all 100 high-risk people receive $A=0$. The overall counts are therefore $n_1=100$ and $n_0=100$, an apparently balanced 50–50 split.

Within the low-risk profile, however, $n_{\text{low},1}=100$ and $n_{\text{low},0}=0$. Within the high-risk profile, $n_{\text{high},1}=0$ and $n_{\text{high},0}=100$. Neither profile has observed overlap. The equal marginal counts arise because the two strategies occupy different profiles, not because both strategies are represented among comparable people.

Thus, a marginal treatment-frequency check is insufficient. The design supports no direct within-profile contrast for either risk group. An outcome difference could reflect differences between the risk profiles as well as treatment, and the table does not provide observed treatment alternatives within either profile.

Two Useful Results for Interpreting the Table

Result 1: Marginal frequencies are weighted averages of profile frequencies. For any exhaustive set of nonempty profiles, $$ \hat p=\frac{\sum_g n_{g1}}{\sum_g n_g} =\sum_g \frac{n_g}{n}\hat p_g, $$ where $n=\sum_g n_g$ and $\hat p$ is the overall frequency of $A=1$. Consequently, an overall frequency strictly between zero and one does not imply overlap in every profile.

Proof. By definition, $n_{g1}=n_g\hat p_g$ for each profile with $n_g>0$. Summing gives $\sum_g n_{g1}=\sum_g n_g\hat p_g$. Divide both sides by $n=\sum_g n_g$:

\frac{\sum_g n_{g1}}{n} =\sum_g \frac{n_g}{n}\hat p_g.

The weights $n_g/n$ are nonnegative and sum to one. They can combine a profile with $\hat p_g=1$ and another with $\hat p_g=0$ into a marginal frequency between zero and one. In the preceding example the weights are both $1/2$, so the overall frequency is $(1/2)(1)+(1/2)(0)=1/2$, even though both profiles lack overlap. $\square$

Result 2: A zero observed count does not by itself establish a zero population treatment probability. Suppose, within a fixed profile, $n_g=m$ people have independent treatment indicators with common probability $q=P(A=1\mid G=g)$. Then the probability of observing no $A=1$ people is $(1-q)^m$.

Proof. For each of the $m$ people, the probability of $A=0$ is $1-q$. Under the stated independence assumption, the probability that all $m$ people have $A=0$ is the product of their probabilities:

P(n_{g1}=0\mid m,q) =\prod_{i=1}^{m}P(A_i=0\mid G_i=g) =(1-q)^m.

For any $0<q<1$ and finite $m$, this probability is positive. Therefore, observing no treated people in a finite profile does not logically prove $q=0$; it signals a potentially serious support problem that needs substantive interpretation. $\square$

Worked Example: How a Small Profile Can Have an Empty Cell by Chance

Suppose the true probability of receiving $A=1$ among people with a particular profile is $q=0.05$, and the profile contains $m=20$ sampled people. Under the independent, common-probability assumption in Result 2, the chance that none receives $A=1$ is:

(1-0.05)^{20}=0.95^{20}\approx 0.3585.

So even with a positive treatment probability of 5%, a sample of 20 has about a 35.9% chance of displaying an empty $A=1$ cell. For a sense of the sample size needed to make this probability no more than 5%, solve $0.95^m\leq 0.05$. Taking logarithms gives $m\geq \log(0.05)/\log(0.95)\approx 58.4$, so at least 59 people are needed under this model. This calculation illustrates sampling uncertainty; it is not a universal minimum profile size or a clinical threshold.

What Poor Overlap Means for the Target Population

A profile without one of the strategies has no observed within-profile treatment contrast. A profile with only a few people under one strategy offers a fragile basis for a comparison in that part of the population. In either case, the data provide less direct support for comparing the strategies across the full eligible population. The limitation concerns the population represented by the comparison, not merely the precision of one summary statistic.

If investigators restrict the intended analysis population to profiles with adequate representation, they change the target population. The proportion of eligible people in such profiles can be reported descriptively, but choosing which profiles qualify requires a clinical and design rationale. It is not appropriate to silently discard unsupported profiles and describe the result as applying to everyone eligible. Nor does the table alone establish that a restricted comparison has all the conditions needed for a causal interpretation.

Cross-tabulations are transparent and preserve the clinical meaning of the profiles shown. A propensity-score summary can compress many baseline characteristics into one numerical scale, but its apparent overlap can conceal sparse combinations of characteristics; conversely, a cross-tabulation can become uninformative if it divides the cohort into too many tiny cells. These are complementary diagnostic views, not substitutes for a justified target population or for the assumptions required by a causal analysis.

Diagnostics, Pitfalls, and Limits

Show denominators. Always report $n_g$, $n_{g1}$, and $n_{g0}$, not only a percentage. Small denominators make frequencies unstable.
Look beyond the overall split. A near-50–50 marginal allocation can coexist with complete separation in important profiles.
Use pre-treatment characteristics. Defining profiles using variables measured after treatment begins can condition the design assessment on consequences of treatment, rather than describe baseline representation.
Keep the treatment definition consistent. Count people according to the protocol’s strategy definition. If a design creates cloned records, do not count artificial copies as independent people when describing actual baseline treatment representation.
Do not turn a threshold into a law. There is no universal frequency that makes a profile adequately represented for every clinical question. Sparse cells warrant investigation, not an automatic conclusion about a particular analysis.

A common mistake is to say, “The cohort has roughly equal numbers under the two strategies, so overlap is good.” The correction is to examine frequencies within clinically important baseline profiles. Overall balance is a weighted average, and—as the first result proves—it can hide profiles in which only one strategy appears. A second mistake is to interpret an empty sample cell as proof that the treatment is impossible for that profile. The finite-sample calculation shows why that inference is too strong, although an empty cell still deserves attention as a practical limitation.

Key takeaway: Assess treatment representation within prespecified, clinically meaningful baseline profiles. Counts and profile-specific frequencies reveal where comparisons are supported, where they are thin, and where the intended target population may exceed what the observed design can credibly compare.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In a profile of 40 people, 6 receive $A=1$ and 34 receive $A=0$. What are the two treatment frequencies, and what does the table establish about observed overlap?
Two equally sized profiles have $A=1$ frequencies of 0.8 and 0.2. Find the overall frequency and explain why it does not describe either profile fully.
For $q=0.10$ and $m=10$, calculate the probability of observing no $A=1$ people under the assumptions of Result 2.
Why does excluding profiles with poor overlap change the interpretation of the target population?

Solutions. (1) The frequencies are $6/40=0.15$ for $A=1$ and $34/40=0.85$ for $A=0$. Both are represented, so there is observed overlap, but representation is uneven; the counts alone do not determine whether it is adequate. (2) With equal profile sizes, the overall frequency is $(0.8+0.2)/2=0.5$. The marginal value averages the profile frequencies and does not reveal that treatment allocation differs substantially between them. (3) The probability is $(1-0.10)^{10}=0.9^{10}\approx0.3487$, or about 34.9%. (4) Exclusion removes people with specified baseline profiles, so the resulting comparison concerns a subset rather than the full eligible population; the restricted target must be stated explicitly.