Inferential Targets Under Response-Adaptive Assignment

Why the Inferential Target Comes First

A response-adaptive trial changes assignment probabilities as information accumulates. That feature makes it especially important to say what a conclusion is about before choosing a statistic or reference distribution. A statement about the average effect among the people enrolled in this trial is not automatically a statement about future patients. Nor is a test of no effect for every participant equivalent to a test that the average effect is zero.

Three inferential targets are useful to keep separate. A finite-sample design-based estimand concerns treatment effects for a specified, fixed set of trial participants. A sharp-null test asks whether a fully specified no-effect claim is compatible with the assignment process and observed outcomes. A model-based claim relies on a statistical model for outcomes, participants, or both, and may target a broader population or a conditional contrast. These targets can coexist in one report, but they are not interchangeable.

Definition: For a fixed cohort of $N$ participants, let $Y_i(1)$ and $Y_i(0)$ denote participant $i$'s outcomes under treatment and control, respectively. Under a well-defined treatment and no interference between participants, the finite-sample average treatment effect is $$ \tau_F=\frac{1}{N}\sum_{i=1}^{N}\{Y_i(1)-Y_i(0)\}. $$ It describes the average contrast for this particular cohort. It does not, by definition, describe a larger population.

This definition uses potential-outcome notation; only the outcome under the assigned treatment is observed for each participant. The finite-sample target is therefore generally not known exactly. The assignment design determines which outcomes are observed and how much information they provide, but it does not change the definition of the target.

Three Targets, Three Kinds of Claims

Target or claim	What it concerns	What it requires
Finite-sample design-based estimand	The average treatment effect for the fixed trial cohort	A defined cohort, well-defined treatment conditions, and potential outcomes for its members
Sharp-null test	A fully specified claim about effects for every participant	A null that determines the missing outcomes under each assignment, plus a correctly represented assignment design
Model-based claim	Conditional effects, predictions, or effects for a broader target population	A stated outcome or sampling model and its assumptions, in addition to the observed data

A sharp null is “sharp” because it specifies enough to determine every participant’s missing potential outcome under every possible assignment. The familiar no-effect-for-anyone claim, $Y_i(1)=Y_i(0)$ for each $i$, is sharp. In contrast, the weak null $\tau_F=0$ allows participants to have nonzero effects that cancel in the average. That weaker claim does not, on its own, determine all missing outcomes.

A design-based statement treats the participants’ potential outcomes as fixed and the assignment as random according to the trial design. A model-based statement instead obtains its interpretation from additional assumptions—for example, that a specified regression model adequately describes outcome means. A model may be useful for precision or adjustment, but its conclusions are not design-based merely because the data came from a randomized trial.

Worked Example: Defining the Finite-Sample Target

Consider a hypothetical trial with four participants and a continuous outcome measured in points. Suppose the potential outcomes, known here only to illustrate the definition, are as follows.

Participant	Control outcome $Y_i(0)$	Treatment outcome $Y_i(1)$	Individual contrast
1	10	12	2
2	8	11	3
3	14	13	-1
4	6	9	3

The finite-sample average treatment effect for these four participants is

\tau_F=\frac{(12-10)+(11-8)+(13-14)+(9-6)}{4} =\frac{2+3-1+3}{4} =\frac{7}{4}=1.75\text{ points}.

The target is an average increase of 1.75 points for this specified cohort. In an actual parallel-group trial, each participant contributes only one observed outcome, so this calculation cannot usually be performed directly. Also, the cohort-specific result alone says nothing about the average effect among people who were not enrolled.

What a Randomization Analysis Treats as Fixed

Randomization inference asks what values a chosen statistic could take under assignments allowed by the design, while specified information is held fixed. In a response-adaptive trial, “hold the design fixed” does not mean freeze the realized sequence of treatment assignments. The assignments are the randomized quantities. It means preserve the trial’s rules and the features on which the analysis conditions.

A careful analysis typically specifies the following before drawing conclusions:

The cohort and its order. If the enrolled participants and their arrival order are treated as given, keep them fixed. If enrollment or eligibility depends on treatment or outcome history, that process may also matter and cannot be silently ignored.
Baseline information. Covariates measured before assignment are commonly held fixed for the enrolled cohort. The analysis should identify which covariates the allocation rule used and which covariates enter the statistic.
The allocation rule. Preserve the design’s initial conditions, constraints, and rule for updating assignment probabilities. A response-adaptive rule must be applied to the history generated under each assignment path being considered.
The outcome information allowed by the null. Under a sharp no-effect null, the observed outcome for each participant can be treated as that participant’s outcome under either assignment. If the allocation rule uses accumulating outcomes, the rule must use the corresponding history when assignments are varied.
Operational rules such as stopping. Preserve a prespecified stopping rule. Whether the realized stopping time can be conditioned on as fixed depends on the design and the question; if assignments could change when the rule stops, the rule may need to be represented rather than treating the observed final sample size as inevitable.

The key distinction is between fixing inputs and rules and fixing a realized random path. Under a sharp null, outcomes can be fixed in the sense that they do not change with assignment; nevertheless, an adaptive algorithm that uses interim outcomes may generate different assignment probabilities along different paths. A valid reference distribution must respect that dependence. The design must be sufficiently documented to determine which assignments and updates are possible.

Worked Example: A Sharp Null and a Weak Null Are Different

Suppose a hypothetical two-person cohort has treatment effects of $+2$ points for participant A and $-2$ points for participant B. The finite-sample average effect is

\tau_F=\frac{2+(-2)}{2}=0\text{ points}.

Thus the weak null $\tau_F=0$ holds for this cohort. But the sharp null of no effect for either participant does not hold: A’s outcome changes by 2 points and B’s by -2 points. The average cancels even though both individual effects are nonzero.

Under the sharp no-effect null, an observed outcome for A or B would be the same under either assignment, so the missing potential outcome is specified by the observed one. Under the weak null, knowing that the two effects average to zero does not reveal either individual effect. Therefore, a test whose reference distribution is justified by the sharp null does not automatically provide an exact test of the weak average-effect null.

A Short Justification: Why the Null Determines What Can Be Held Fixed

Consider an assignment $Z_i$, where $Z_i=1$ means treatment and $Z_i=0$ means control. The observed outcome is $Y_i=Z_iY_i(1)+(1-Z_i)Y_i(0)$. To evaluate a statistic under a different assignment, the outcomes that would have been observed under that assignment must be known or determined by the null.

\begin{aligned} H_0^{\text{sharp}} &: Y_i(1)=Y_i(0)\quad\text{for every }i,\\ &\Longrightarrow Y_i(1)=Y_i(0)=Y_i\quad\text{for every }i,\\ &\Longrightarrow \text{the outcome for each participant is determined under every assignment.} \end{aligned}

This is why a sharp null supports direct reassignment of the observed outcomes when defining a randomization distribution. The implication fails for a weak null: an average constraint on the differences does not specify each missing outcome. This distinction concerns what the null lets the analyst impute; it does not by itself guarantee that the assignment paths or adaptive updates have been represented correctly.

Worked Example: A Model-Based Standardized Contrast

In a hypothetical study, an analyst fits an outcome model using a baseline risk category $X$, with separate fitted mean outcomes for treatment and control in each category. The fitted means are 11 and 9 points for low-risk participants, and 21 and 18 points for high-risk participants. The target population for this illustration is defined to have equal proportions in the two categories.

The fitted treatment-control contrasts are $11-9=2$ points in the low-risk category and $21-18=3$ points in the high-risk category. Standardizing to the stated 50:50 target distribution gives

\widehat{\tau}_{\text{std}} =0.50(11-9)+0.50(21-18) =0.50(2)+0.50(3) =2.50\text{ points}.

This is a model-based standardized contrast for the specified target distribution, using the fitted category-specific means. Its interpretation relies on the adequacy of those fitted means and on the relevance of the chosen 50:50 distribution. It is not automatically the finite-sample effect for the enrolled cohort, and it is not automatically an effect for every member of a broader population. The target distribution and model assumptions must be stated to make the claim interpretable.

Choosing Among the Approaches

The main alternatives answer different questions. A design-based estimand keeps the focus on a fixed set of trial participants and the assignment mechanism. A sharp-null randomization test assesses a fully specified null using outcomes that the null makes imputable. A model-based analysis can make conditional or population-oriented claims, but those claims depend on its model and target-population definitions. One analysis cannot be substituted for another simply because all use the same observed data.

In practice, state the target before naming the method: for example, “the average treatment effect among the $N$ enrolled participants,” “the sharp no-effect null for every participant,” or “the model-standardized contrast for a population with the stated covariate distribution.” Then state the assumptions that connect the data and assignment process to that target. If the target is a broader patient population, explain how the trial cohort represents that population; random assignment within a trial does not, by itself, establish representativeness.

Diagnostics, Limitations, and a Common Pitfall

Several checks help reveal a mismatch between the intended target and the analysis:

Can the analyst state whether the target is the enrolled cohort, a conditional subgroup, or a broader population?
Does the stated null determine the unobserved outcomes needed for the proposed randomization calculation?
Are the adaptive allocation and stopping rules documented well enough to identify how they respond to each relevant history?
Does the model-based conclusion depend on extrapolation into covariate groups or treatment-history patterns with little or no data?
Are outcome definitions, missing-data handling, and any exclusions consistent with the target population and the design being analyzed?

A common mistake is to see a randomization test of “no effect,” then report it as an exact test of “zero average effect.” The reasoning is wrong because “no effect for every participant” is stronger than “effects average to zero”; only the former specifies all individual missing outcomes. The correction is to state the null precisely and match the test’s justification to that null. A second mistake is to keep the observed adaptive assignment probabilities fixed while imagining a different assignment path, even though the algorithm updates probabilities from the evolving history. The correction is to distinguish fixed design rules from path-specific quantities that those rules recalculate.

No inferential framework removes the need for careful design records. A design-based analysis can be limited by a poorly documented randomization algorithm or by weak support for relevant assignments. A model-based analysis can be sensitive to misspecification, sparse data, or a poorly defined target population. The conclusions should identify which limitations apply rather than presenting any one framework as universally preferable.

Key takeaway: Name the estimand, the null or model claim, and the features treated as fixed before interpreting an inferential result. In an adaptive trial, preserve the allocation and stopping rules while allowing assignment-dependent histories to evolve as the rules prescribe.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A trial reports an average effect for the participants enrolled. Is that automatically an effect for future patients? Why or why not?
Two participants have effects of $+4$ and $-4$ points. Which of the weak null $\tau_F=0$ and the sharp no-effect null holds?
Under a sharp no-effect null, why can observed outcomes be used to determine outcomes under alternative assignments?
An adaptive algorithm updates treatment probabilities using accumulating outcomes. What should remain fixed, and what may need to be recalculated?
A model-standardized effect is calculated for a population with 30% high-risk and 70% low-risk participants. Name two details needed to interpret the result.

Solutions. (1) No. The finite-sample estimand concerns the enrolled cohort; a claim for future patients requires a defined target population and assumptions connecting it to the trial. (2) The weak null holds because $(4-4)/2=0$ points, but the sharp null fails because neither individual effect is zero. (3) The sharp null states $Y_i(1)=Y_i(0)$ for every participant, so each participant’s observed outcome supplies both potential outcomes under that null. (4) Keep the cohort and relevant baseline information fixed if conditioning on them, and preserve the algorithm and its rules; recalculate probabilities and other history-dependent features as each assignment path generates its history. (5) The fitted outcome model and its adequacy, and the target population’s specified 30:70 covariate distribution; other relevant details include how uncertainty and sparse categories are handled.