Alternative Configurations for Treatment Effects

Why Vary Treatment-Effect Configurations?

A simulation evaluates a trial design under specified conditions. Those conditions should not be summarized by one alternative in which every treatment has the same effect: a MAMS trial may face no effective treatment, one effective treatment among several, or several treatments with different effect sizes. These are distinct data-generating situations, and a design can behave differently in each.

This tutorial focuses on specifying those alternative treatment-effect patterns. The aim is not to calculate power or error rates, but to make the simulated alternatives explicit, interpretable, and plausible. The treatment-effect configuration is only one part of a scenario: the outcome distribution, nuisance parameters, allocation, and trial rules must also be specified. The earlier material on translating a MAMS protocol into simulation logic and generating outcomes provides the context for those other parts.

Fix the effect direction before constructing scenarios. In the examples below, a positive \(\delta_j\) always means a benefit. For a continuous outcome, we orient the scale so that larger values are better and define \(\delta_j=\mu_j-\mu_0\), where \(\mu_j\) is the mean in arm \(j\). For a binary undesirable event, we define \(\delta_j=p_0-p_j\), where \(p_j\) is the event probability in arm \(j\). Thus, positive effects have a consistent interpretation even though their formulas differ by outcome type.

An effect can be nonzero but too small to matter clinically. If a clinically worthwhile effect is specified, denote its threshold on the chosen benefit scale by \(\delta_{\min}>0\). This threshold helps distinguish a merely nonzero effect from one considered worthwhile; it does not change the data-generating model.

Specify a Scenario Before Simulating It

For a continuous endpoint, one simple model assumes independent normally distributed outcomes within each arm, with a common variance and an effect that is constant over stages:

Here \(Y_{ijs}\) is the outcome for participant \(i\) in experimental arm \(j\) at stage \(s\). Control outcomes follow \(N(\mu_0,\sigma^2)\). The model assumes the same \(\delta_j\) and \(\sigma^2\) at every stage. A simulation may use other distributions or stage-specific effects, but those departures should be stated as intentional assumptions rather than introduced accidentally.

For a binary undesirable event, an analogous model is

The specified values must satisfy \(0\leq p_j\leq 1\) for every arm. If effects are instead defined on a log-odds scale, specify that scale and convert the effects to event probabilities through the model before simulating outcomes. An effect size is not meaningful without its scale.

A practical scenario set varies three features separately where possible: the number of arms with effects, the magnitude of those effects, and their pattern across arms. This makes it easier to understand what a result represents. For example, a common-effect alternative and a one-effective-arm alternative differ in the number of treatments that benefit, even if one arm has the same effect size in both.

Worked Configurations

Justifying a Compact Scenario Set

A scenario set does not need to include every possible effect vector. It should, however, cover the features that matter for the intended design evaluation. A compact set might include a common moderate effect, a single effective arm, a mixture of zero and nonzero effects, and unequal effects across the experimental arms. The choice depends on the trial question and on the plausible range of effects.

The logic can be stated directly. Let \(\mathcal{C}\) be the set of configurations selected for simulation. If the design’s behavior is to be examined under \(m\) distinct features—such as number of nonzero effects, effect magnitude, and effect ordering—then \(\mathcal{C}\) should include at least one configuration representing each feature considered important. Otherwise, conclusions from the simulation do not address that feature. This is a coverage argument, not a guarantee that the selected scenarios encompass every possible treatment-effect pattern.

For example, suppose a design is intended to be considered when there may be one, two, or all experimental treatments with benefit. A scenario set containing only \((5,5,5)\) cannot show how the design behaves when just one treatment benefits. Adding a vector such as \((0,0,5)\) addresses that missing pattern. If unequal benefits are also plausible, a vector such as \((2,5,8)\) addresses a different feature. The vectors should be interpreted alongside the outcome model, not as standalone descriptions.

Keep nuisance assumptions fixed across scenarios when the goal is to isolate treatment-effect patterns. Conversely, if the aim is to study a joint condition—such as a smaller effect together with greater outcome variability—state that explicitly. Changing effect size, variance, allocation, and outcome distribution simultaneously makes it difficult to attribute differences between scenarios to any one feature.

Diagnostics, Alternatives, and Limitations

Before running a large simulation, inspect the scenario table or parameter vectors. The following checks catch common construction errors:

• Direction check: confirm that positive effects consistently represent benefit. For an undesirable event, verify that benefit corresponds to a lower event probability.
• Scale check: distinguish points, percentage points, relative risks, and standardized effects. Do not compare their numerical values as though they were the same scale.
• Range check: ensure every binary probability is between \(0\) and \(1\), and that all means and variances are plausible for the stated model.
• Pattern check: count the nonzero effects and confirm that arm labels match the intended pattern. An accidental reordering can turn a one-effective-arm scenario into a different configuration.
• Stage check: decide whether effects are constant across stages. If they change over time, specify how and why; otherwise, do not allow a coding rule to change them inadvertently.

A common mistake is to call every experimental arm “active” and then assign all of them a positive effect. “Experimental arm” identifies the arm’s role in the trial; it does not assert that its true effect is nonzero. The correction is to describe the full vector, including any zero effects, and to state separately how many arms benefit under that scenario.

Another mistake is to select a single convenient alternative and treat it as representative of all alternatives. A common-effect configuration is easy to interpret, but it cannot stand in for a one-effective-arm or unequal-effect configuration. A scenario grid is more informative when its dimensions are explicit and its entries represent distinct patterns.

The main alternative to a small, deliberately chosen set is a broad grid or a probability distribution over effect vectors. A grid can examine several combinations systematically but may require many simulations and can include implausible combinations if its limits are poorly chosen. A distribution over effects can represent uncertainty about plausible treatment effects, but it answers a different question and depends on the chosen distribution. Neither approach removes the need to define the effect scale, assumptions, and plausible range.

Finally, scenario construction cannot establish that the chosen effects are clinically plausible; that judgment requires relevant clinical knowledge. Nor does a finite collection of vectors prove how a design will behave for every possible configuration. The purpose is to make the conditions being examined transparent and broad enough for the stated design question.