Build Data-Generating Mechanisms For Adaptive Trials

Why the Data-Generating Mechanism Matters

A simulation comparison is only as informative as the trial conditions it represents. An allocation rule is evaluated against data generated under a specified population, outcome process, and treatment effect. If those ingredients are vague or inconsistent, the resulting operating characteristics may answer a different question from the one intended. A data-generating mechanism (DGM) makes those ingredients explicit: it describes how each participant’s covariates and potential outcomes arise.

The focus here is the construction of outcome, covariate, and treatment-effect models for adaptive trials. The allocation mechanism determines which potential outcome is observed, but it does not define the population response process. For a proposed DGM, specify that response process first, including what “no effect” means and how any effect varies across participants.

A common construction is to generate \(X_i\), then define potential outcomes \(Y_i(0)\) and \(Y_i(1)\), and finally record the outcome corresponding to the assigned treatment. This separates the response model from the allocation rule. It also forces a clear statement of the target effect: for example, a difference in mean outcome, a difference in event risk, or a subgroup-specific contrast.

Specify Covariates and Outcomes Before Effects

Begin by deciding what covariates matter to the trial question and how they are distributed. A binary prognostic covariate might represent a disease subgroup; a continuous covariate might represent baseline severity. The model should state whether covariates are generated independently, jointly, or with a specified association. It should also reflect the intended target population: for instance, a subgroup prevalence in a simulation should not be chosen accidentally through an arbitrary sampling scheme.

Next specify how outcomes depend on covariates in the absence of treatment. For a continuous outcome, a simple model is \(Y_i(0)=\beta_0+\beta_X X_i+\varepsilon_i\), where \(\varepsilon_i\) has mean zero and a stated variance. For a binary outcome, one standard choice is a logistic model, which maps a linear predictor to an event probability. These are modeling choices, not universal truths. Their assumptions include the chosen functional form, the error distribution or link function, and the covariate distribution.

The potential outcomes need a joint specification if the simulation will use individual treatment effects, paired potential outcomes, or quantities depending on their dependence. Their marginal distributions alone do not determine how \(Y_i(0)\) and \(Y_i(1)\) co-vary within a participant. One simple choice is to use a shared exogenous error or uniform random variable across treatment conditions. Another is to generate the potential outcomes independently conditional on covariates. Either choice can yield the same marginal treatment-specific outcome distributions, but the distribution of individual effects \(Y_i(1)-Y_i(0)\) may differ.

Null Effects and Heterogeneity

A sharp null of no treatment effect states that every participant’s potential outcomes are equal: \(Y_i(1)=Y_i(0)\) for all \(i\). A convenient way to generate it is to use exactly the same outcome-generating expression under both treatments. A weaker null can instead concern an average effect, such as \(E\{Y_i(1)-Y_i(0)\}=0\). That condition permits some participants to benefit and others to be harmed, provided the effects average to zero. Thus, a zero average effect does not by itself imply a sharp null.

Heterogeneity can be modeled by allowing the treatment contrast to depend on baseline covariates. In a continuous-outcome model, the treatment term may include an interaction with a subgroup indicator. In a binary-outcome model, an interaction on the log-odds scale describes heterogeneity in conditional log odds; it does not directly specify the same degree of heterogeneity in risk differences. Always calculate the effect on the scale that will be interpreted.

Represent Clinically Relevant Alternatives

A useful alternative scenario should be defined in an interpretable scale, not merely by choosing a nonzero coefficient. For a continuous endpoint, specify the treatment effect in outcome units and state how it varies by subgroup. For a binary endpoint, report treatment-specific risks and a marginal contrast when that is the clinical quantity of interest. An effect can be clinically relevant even if it is not constant across participants.

Construction Checks and Common Failure Modes

Before using a DGM, inspect what it implies. For continuous outcomes, calculate conditional means, treatment contrasts, and a plausible range of generated values. For binary outcomes, evaluate event probabilities at relevant covariate combinations and confirm that each lies strictly between zero and one. For either outcome type, check that subgroup prevalence and treatment-effect summaries match the intended population. A model can be mathematically valid yet clinically implausible.

A useful derivation is to obtain the marginal treatment effect by averaging conditional effects over the covariate distribution. If \(X_i\) takes values \(x\) with probabilities \(P(X_i=x)\), then for a mean outcome contrast,

This follows by partitioning the expectation over the mutually exclusive covariate values. It is the required calculation when subgroup effects are specified but the target is an overall mean effect. For binary outcomes, the same averaging applies to conditional risks to obtain marginal risks; subtract those two marginal risks to get the marginal risk difference.

Common errors include confusing a sharp null with a zero average effect, describing a conditional logistic coefficient as a marginal risk difference, and choosing a subgroup prevalence that is inconsistent with the intended population. Another mistake is specifying only the separate distributions of \(Y_i(0)\) and \(Y_i(1)\) while assuming that this uniquely determines individual treatment effects. It does not: the within-person dependence between potential outcomes must also be stated when it matters.

There are several reasonable alternatives when specifying a response model. A logistic model is convenient for conditional odds and ensures valid probabilities, while direct risk models may make marginal risk contrasts easier to target but need constraints to keep probabilities in range. Fixed covariate values can represent a deliberately balanced population; randomly generated covariates represent population sampling variation. Neither choice is universally preferable. The choice should match the scientific target and be stated. Simple models are often easier to interpret, while richer models can represent nonlinear prognostic effects or multiple subgroups—but require more parameters and more careful plausibility checks.

These mechanisms have limits. A chosen model is not evidence that the real response process follows it. Results can depend on the covariate distribution, the effect scale, the assumed outcome variability, and the coupling of potential outcomes. A finite collection of scenarios cannot cover every plausible trial population. The purpose is to make assumptions visible and to evaluate conclusions under clearly defined conditions, not to disguise assumptions as empirical facts.