Testing Nested Population Hypotheses

Why Nested Population Claims Need Precise Definitions

An adaptive trial may be designed to support a treatment claim for everyone eligible for the trial, for patients with a biomarker, or for a narrower group defined by additional biomarker criteria. These populations are related, but a treatment effect in one is not automatically the treatment effect in another. A hypothesis family makes the intended claims explicit and helps distinguish what follows from a result in one population from what would require separate evidence in another.

This tutorial focuses on the structure and implications of those claims. It does not specify a procedure for controlling error across multiple tests. A list of hypotheses is not, by itself, a complete testing strategy: the analysis must also address how evidence across the claims will be evaluated.

Define the Population Effects Before Writing Hypotheses

Let \(A\) denote the all-comer target population. Let \(B\) be a prespecified biomarker subgroup contained in \(A\), and let \(C\) be a narrower subgroup contained in \(B\). Thus \(C\subseteq B\subseteq A\). The set \(A\setminus B\) consists of people in the all-comer population who are not in \(B\); similarly, \(B\setminus C\) consists of people in \(B\) who are not in \(C\).

For clarity, suppose the endpoint is a clinical benefit score for which larger values are better. Let \(Y(1)\) be the outcome a participant would have under treatment and \(Y(0)\) the outcome under control. For any defined population \(S\), write its mean treatment effect as

A positive \(\Delta_S\) means higher average benefit under treatment than under control in population \(S\). The effect is expressed in the endpoint’s units, such as points on a symptom score. The definition must be adapted if the endpoint’s direction is reversed; the sign should always correspond to the stated clinical benefit.

A causal interpretation of \(\Delta_S\) requires a well-defined treatment and outcome, a prespecified target population, and conditions supporting identification of the treatment contrast. In a randomized trial, random assignment supports comparison of treatment groups within baseline-defined populations, subject to the trial’s conduct and analysis assumptions. Biomarker status used to define \(B\) or \(C\) should be measured before treatment assignment, or otherwise be defined in a way that does not depend on post-randomization outcomes. The outcome, follow-up time, and handling of intercurrent events should be aligned across populations. Otherwise, apparently related claims may refer to different estimands.

For a one-sided claim that the average benefit exceeds a threshold \(\delta\), the population-specific hypotheses are

The threshold is measured in the endpoint’s units. If \(\delta=0\), the alternative asserts positive average benefit. If \(\delta\) is a positive minimum clinically important effect, the alternative asserts benefit greater than that minimum. The null is a composite statement: it includes effects equal to or below the threshold. It is not necessarily a claim of exactly zero effect.

How Nested Effects Are Related

The effects in a nested population and its complement combine to give the effect in the broader population. Define \(q=P(B\mid A)\), the proportion of the target population \(A\) belonging to \(B\). For the same outcome and treatment contrast in each group,

This identity follows by conditioning the mean potential outcome under each treatment on membership in \(B\) or \(A\setminus B\). For \(z\in\{0,1\}\),

Subtract the expression for \(z=0\) from the expression for \(z=1\). The two conditional differences are \(\Delta_B\) and \(\Delta_{A\setminus B}\), giving the stated formula. The same reasoning applies to \(B\), \(C\), and \(B\setminus C\). The weight is the relevant population proportion, not necessarily the proportion in a particular randomized sample.

This identity describes a relationship among population effects; it does not say that a favorable effect in one subgroup guarantees a favorable effect in the broader population. The complement’s effect and the subgroup’s prevalence both matter. Nor does the identity equate a within-subgroup treatment claim with a claim that treatment effects differ between subgroups.

Implications of Claims Across Several Nested Populations

For populations \(C\subseteq B\subseteq A\), a family might include \(H_{0,A}\) versus \(H_{a,A}\), \(H_{0,B}\) versus \(H_{a,B}\), and \(H_{0,C}\) versus \(H_{a,C}\). Each alternative makes a distinct claim: treatment benefit above the threshold in all-comers, in \(B\), or in \(C\). Writing the family this way avoids treating “the treatment works in the biomarker population” as if it had a single unambiguous meaning.

With the same threshold \(\delta\) for \(B\) and its complement, there is a useful implication. If both \(\Delta_B>\delta\) and \(\Delta_{A\setminus B}>\delta\), then

provided \(0\leq q\leq1\). Thus both component effects exceeding the threshold entails that the broad effect exceeds it. But a claim about \(B\) alone does not entail the same claim for \(A\), because the effect in \(A\setminus B\) is unrestricted by that claim. The broad effect exceeding \(\delta\) also does not entail that the effect in \(B\) exceeds \(\delta\): a sufficiently large effect outside \(B\) can raise the weighted average.

The same reasoning applies to the narrower population \(C\). A claim that \(\Delta_C>\delta\) does not establish \(\Delta_B>\delta\), because the effect in \(B\setminus C\) could be small or unfavorable. Conversely, \(\Delta_B>\delta\) does not establish \(\Delta_C>\delta\). A claim about \(B\) is an average over its members, not a guarantee about every subgroup within it.

Implications depend on thresholds as well as nesting. If \(B\) and \(A\setminus B\) are judged against different thresholds, the common-threshold inequality above cannot be used without modification. More generally, if \(\Delta_B>\delta_B\) and \(\Delta_{A\setminus B}>\delta_{A\setminus B}\), the weighted lower bound is \(q\delta_B+(1-q)\delta_{A\setminus B}\). Whether that bound exceeds the all-comer threshold \(\delta_A\) must be checked explicitly.

Choosing a Hypothesis Family That Matches the Claim

The correct family depends on the intended scientific claims, not only on which populations can be defined from the data. For each population included, state the estimand and the claim in words before writing its hypothesis. For example, “average benefit exceeds 2 points in biomarker-positive patients” corresponds to \(H_{0,B}:\Delta_B\leq2\) versus \(H_{a,B}:\Delta_B>2\), assuming \(\Delta_B\) is measured in points on the specified endpoint.

A nested population claim is not the same as an interaction claim. “Treatment benefit exceeds 2 points in \(B\)” concerns the level of \(\Delta_B\). “The treatment effect differs between \(B\) and \(A\setminus B\)” concerns a contrast between effects, such as \(\Delta_B-\Delta_{A\setminus B}\). One claim may be true without the other. For instance, both populations could have large and similar benefits, or \(B\) could have a benefit while the complement has none. A family of within-population hypotheses should not be described as proof that treatment effects vary by biomarker status.

A useful comparison is between an all-comer-only claim and a collection of population-specific claims. An all-comer-only hypothesis directly addresses the average effect in \(A\), but it may mask different effects among its components. A set of subgroup hypotheses can state which groups are of interest, but it does not automatically provide a claim for \(A\). Neither formulation is universally preferable; the choice should reflect the trial objective and the populations for which conclusions are intended.

Diagnostics and Common Pitfalls

Before interpreting a nested family, verify the following:

• Membership is genuinely nested. Confirm that the definitions satisfy \(C\subseteq B\subseteq A\). Overlapping but non-nested groups do not have the simple subgroup-and-complement structure used here.
• Effects are comparable. Use the same endpoint, follow-up time, treatment contrast, and effect scale when applying the weighted-average identity.
• Weights match the target population. Use \(P(B\mid A)\) for the target population in the estimand. A sample proportion can differ because of sampling variation or the trial’s recruitment and allocation structure.
• The direction and threshold are explicit. State whether positive values mean benefit and express \(\delta\) in the outcome’s units. If thresholds differ across claims, do not assume common-threshold implications.
• Estimates are not confused with hypotheses. A sample estimate above a threshold does not by itself establish that the population effect exceeds it. The hypothesis states a parameter claim; a testing procedure and its error control determine how evidence is assessed.

A common mistake is to reason, “The biomarker subgroup has a favorable effect, so the treatment must benefit all-comers.” That conclusion ignores both subgroup prevalence and the effect outside the subgroup. The correction is to write the weighted identity and check the complement: \(\Delta_A=q\Delta_B+(1-q)\Delta_{A\setminus B}\). Another mistake is to treat a significant result in one subgroup and a nonsignificant result in another as proof that the subgroup effects differ. Separate within-population claims do not, by themselves, establish a difference between effects.