From a Trial Scenario to a Probability Model
A scenario matrix identifies the trial conditions under which operating characteristics are to be assessed. To make those conditions mathematically usable, specify how a hypothetical trial’s data arise under each scenario. The central question is not only what the treatment effect is, but also what is generated for each participant, in what order, and under which dependencies.
A data-generating mechanism (DGM) is a probability model for the trial data under a stated set of assumptions and parameter values. It specifies the distribution of relevant baseline covariates, treatment assignments, outcomes, and follow-up or missingness indicators. In a calibration exercise, the DGM represents a possible truth against which the trial procedure is evaluated. It is distinct from the model used to analyze the trial: those two models may agree, but they need not.
A useful way to organize the specification is by the temporal and probabilistic order of events. For a simple parallel-group trial, let \(X_i\) denote baseline covariates for participant \(i\), \(A_i\) the assigned treatment, \(Y_i\) the outcome that would be measured, and \(R_i\) an indicator that the outcome is observed. A possible factorization is
This factorization is a description of conditional distributions, not a claim that the variables are independent. In particular, follow-up can depend on the outcome, treatment, or covariates. If the trial includes intermediate measurements or additional time-varying variables, their conditional distributions must also be included in an order consistent with when they occur.
Specify the Design Before the Outcome Model
Begin with what the protocol fixes: eligibility, sample size, allocation ratio, randomization method, endpoint definition, and follow-up schedule. Then state which quantities vary between participants and which are fixed by design. This prevents a common error: treating an allocation rule as if it were an outcome probability, or assuming that the planned sample size implies complete outcome ascertainment.
For example, under complete randomization with exactly \(n_T\) of \(n\) participants assigned to treatment, the assignment vector has exactly \(n_T\) treatment labels. Individual assignments are not independent: once all but one assignments are known, the last may be determined by the fixed total. By contrast, under Bernoulli assignment with probability \(q\), the \(A_i\) may be independent with \(P(A_i=1)=q\), and the number assigned to treatment is random. These are different designs and generally generate different trial data.
The outcome model should then state the response scale, its distribution, and the parameters that define the scenario. For a binary endpoint, one standard choice is \(Y_i\mid A_i,X_i\sim\text{Bernoulli}(p_i)\), with a specified model for \(p_i\). For a continuous endpoint, one might specify a normal distribution with a treatment- and covariate-dependent mean and a stated variance. Choosing a distribution is an assumption, not a consequence of knowing only a mean treatment effect.
Worked Example: Binary Endpoint Under Complete Randomization
Suppose a hypothetical superiority trial enrolls 200 eligible participants and uses complete randomization to assign exactly 100 to each arm. The endpoint is response by week 12, coded \(Y_i=1\) for response and \(Y_i=0\) otherwise. For this scenario, assume each participant’s response is Bernoulli with probability 0.30 under control and 0.45 under treatment, and assume responses are independent across participants conditional on their assigned groups. Assume all outcomes are observed.
The assignment rule fixes the arm sizes; it is not independent Bernoulli assignment. Conditional on the assignments, the outcome distributions are
The expected number of responses is \(100(0.30)=30\) in control and \(100(0.45)=45\) in treatment. The scenario’s response-probability difference is \(0.45-0.30=0.15\), or 15 percentage points. These are expectations and model parameters, not guaranteed realized counts or an analysis result. The specification also makes its limitations visible: it assumes no covariate variation, no loss to follow-up, and a common response probability within each arm.
Connect the Mechanism to the Estimand
An estimand describes the treatment effect or other target quantity of interest; the DGM describes how trial data arise under a scenario. Their connection must be explicit. For instance, if the target is a treatment-minus-control difference in response probabilities, then the scenario needs arm-specific response probabilities in the population relevant to that target. If treatment effects vary across baseline risk groups, a single average difference may depend on the population weights used to combine those groups.
Potential outcomes provide one way to state this connection. Let \(Y_i(1)\) and \(Y_i(0)\) be participant \(i\)’s outcomes under treatment and control. Under random assignment and consistency, the observed outcome is \(Y_i=A_iY_i(1)+(1-A_i)Y_i(0)\) for a binary endpoint. Consistency means that the observed outcome equals the potential outcome corresponding to the treatment actually assigned. A DGM may specify the marginal distributions of the potential outcomes, or specify conditional observed-outcome distributions under each assignment when only the randomized observed data are needed. If a target depends on the joint distribution of both potential outcomes for the same person, that joint distribution must be specified; its two marginal distributions alone do not determine it.
Worked Example: A Continuous Endpoint With a Baseline Stratum
Consider a hypothetical trial with 150 participants: 90 in a low-risk baseline stratum (\(X=0\)) and 60 in a high-risk stratum (\(X=1\)). Randomization is stratified, assigning two participants to treatment for every one assigned to control within each stratum. The endpoint is a continuous symptom score at week 8, measured in points; lower scores are better. Suppose the outcome model is normal with variance \(100\) points squared in each arm and stratum. The control means are 50 points for \(X=0\) and 58 points for \(X=1\); treatment lowers each stratum-specific mean by 5 points.
The resulting conditional distributions are
There are 60 treatment and 30 control participants in the low-risk stratum, and 40 treatment and 20 control participants in the high-risk stratum. Thus each arm has 60% low-risk and 40% high-risk participants. The weighted control mean is \(0.60(50)+0.40(58)=53.2\) points, and the weighted treatment mean is \(0.60(45)+0.40(53)=48.2\) points. The treatment-minus-control difference is \(48.2-53.2=-5.0\) points, matching the stratum-specific effect in this scenario. If the stratum proportions differed across arms or in the target population, the marginal comparison and its interpretation would require corresponding weights. The model also assumes normal outcomes, a common variance, and no missing outcomes; each should be judged against the intended calibration scenario.
Follow-Up Is Part of the Mechanism
A trial dataset contains only outcomes that are observed. Let \(R_i=1\) indicate that the endpoint is observed and \(R_i=0\) indicate that it is missing. Specify \(P(R_i=1\mid Y_i,A_i,X_i)\), or an appropriate conditional model, rather than silently assuming \(R_i=1\) for everyone. The dependence structure matters: if outcome observation varies with the unobserved outcome, the observed outcome distribution can differ substantially from the underlying outcome distribution.
Worked Example: Outcome-Dependent Observation of a Binary Endpoint
In a hypothetical single-arm illustration, suppose the true week-12 response probability is 0.40. A responder’s outcome is observed with probability 0.90, while a nonresponder’s outcome is observed with probability 0.60. Assume these observation probabilities apply to each participant and that observation is conditionally independent across participants given response status.
The probability that a participant is observed is found by adding the probabilities of the two mutually exclusive routes to observation: responding and being observed, or not responding and being observed.
Thus 0.72, or 72%, of participants are observed in expectation. The probability of being both a responder and observed is \(0.90(0.40)=0.36\). Therefore the response proportion among observed participants is
Half of the observed participants are responders, even though the underlying response probability is 0.40. The observed proportion is not the same quantity as the underlying response probability because observation depends on response status. This example shows why missingness assumptions belong in the DGM, not merely in a later analysis description.
A Practical Specification and Its Diagnostics
Name the population, endpoint, time point, treatment contrast, and population weights relevant to the target quantity.
Give sample size, covariate distribution or fixed stratum counts, allocation method, and any constraints on assignments.
State outcome type, conditional distributions, parameter values, and dependence assumptions across participants or time.
Specify follow-up, missingness, censoring, or intercurrent-event rules that affect the data available to the planned analysis.
Calculate simple expectations, verify totals and probabilities, and ensure the generated quantities represent the target population and endpoint.
A specification is suspect if it cannot reproduce basic design facts, such as planned arm sizes, or if its implied outcome rates, event counts, or missingness differ from the stated scenario. Check that all probabilities lie between 0 and 1, variances are nonnegative, and category probabilities sum to 1. Check dimensions and units as well: a variance for a score measured in points is in points squared, while a treatment difference is in points. For time-to-event outcomes, a mechanism that only specifies an event probability at one fixed time cannot by itself represent event times and censoring throughout follow-up.
A useful diagnostic is to derive expectations from the proposed mechanism before using it. For example, calculate expected arm sizes, outcome rates by stratum, and observation proportions. These checks can expose a reversed treatment coding, an incorrect allocation rule, or confusion between an underlying outcome probability and an observed-data proportion. They do not prove the mechanism is realistic, but they test whether its consequences match its stated assumptions.
Alternatives, Assumptions, and Limitations
The DGM is not the same thing as a scenario matrix. A scenario matrix organizes alternative settings, such as different treatment effects or control response rates; the DGM gives a probability law for generating data within a setting. Nor is the DGM the analysis model: the analysis model is applied to generated data, while the DGM states the assumed truth for calibration. Keeping these roles separate makes model misspecification assessable rather than hidden.
A simple parametric mechanism, such as independent Bernoulli outcomes with fixed probabilities, is transparent and easy to inspect. A richer mechanism can include prognostic covariates, treatment-effect variation, repeated outcomes, or informative dropout. Greater detail is valuable only when it represents features relevant to the trial procedure or its target; unnecessary complexity can make assumptions hard to communicate and results hard to diagnose. Empirical or external data can inform parameter choices, but do not remove uncertainty about whether the mechanism is appropriate.
No single DGM covers every plausible truth. Calibration conclusions are conditional on the specified mechanisms and parameter values. In particular, an analysis model that performs well under a matching DGM may behave differently under nonnormal outcomes, effect heterogeneity, or outcome-dependent missingness. The appropriate response is to define a set of scientifically meaningful scenarios, not to treat one convenient model as established fact.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A trial assigns exactly 40 of 100 participants to treatment. Are the assignment indicators independent Bernoulli variables with probability 0.40?
- In a binary-outcome trial, \(P(Y=1)=0.25\), \(P(R=1\mid Y=1)=0.80\), and \(P(R=1\mid Y=0)=0.90\). Find \(P(R=1)\) and the response proportion among observed participants.
- Why can specifying the marginal distributions of \(Y(1)\) and \(Y(0)\) be insufficient for a target that depends on their within-person joint distribution?
- A continuous endpoint is measured in points and has standard deviation 4 points. What is its variance, including units?
- Name two checks that can reveal an internally inconsistent DGM before it is used to assess a trial procedure.
Solutions. (1) No. With exactly 40 assigned, the assignment indicators are dependent because the total is fixed. (2) \(P(R=1)=0.25(0.80)+0.75(0.90)=0.875\). The joint probability of response and observation is \(0.25(0.80)=0.20\), so \(P(Y=1\mid R=1)=0.20/0.875\approx0.2286\). (3) Marginal distributions give the behavior of each potential outcome separately but not their dependence within a participant; quantities involving both outcomes can therefore differ across joint distributions with the same marginals. (4) The variance is \(4^2=16\) points squared. (5) Examples include checking that arm totals match the allocation rule, probabilities are in \([0,1]\), category probabilities sum to 1, and implied outcome or missingness rates agree with the scenario.