From treatment effects to decision targets
A multi-arm trial can answer more than one kind of question. A clinician may ask whether a particular experimental treatment is beneficial compared with control. A programme may instead ask whether any arm is worth advancing, whether all arms meet a target, or whether at least a specified number do. These questions concern different events, even when they use the same trial data and treatment-effect estimates. The first step is to define what effect is being targeted; the next is to state precisely which event would support the intended claim.
Consider \(K\) experimental arms and one common control. For a clear starting point, suppose the outcome is measured at a prespecified time, larger values are better, and the target population is defined before analysis. Let \(Y(j)\) denote the outcome a person in that population would have under assignment to arm \(j\), and let \(Y(0)\) denote the outcome under assignment to control. The arm-specific treatment-effect estimand is the population mean contrast \(\theta_j=E[Y(j)]-E[Y(0)]\), for \(j=1,\ldots,K\). Positive values therefore favour the experimental arm; the units are the outcome's units.
This definition relies on a well-defined treatment strategy for each arm, a common target population and outcome measurement, and a meaningful comparison under assignment to each strategy. A causal interpretation also requires the usual conditions connecting potential outcomes to observed data, including consistency and no interference between participants, together with valid treatment assignment and appropriate handling of missing outcomes. These conditions do not follow just from writing a contrast. If the outcome is a risk for which lower values are better, the contrast can instead be oriented as \(E[Y(0)]-E[Y(j)]\), so positive values still represent benefit. The scale, population, outcome, time point, and direction must all be stated rather than left implicit.
An estimand is not the same as an estimate: an estimate is calculated from observed data to provide information about the target. Nor is either one a decision rule. A decision rule specifies what action follows from evidence about a stated event. Bayesian analysis expresses uncertainty about the event through its posterior probability given the data, written \(P(A\mid D)\) for event \(A\) and data \(D\). The event itself must be defined before that probability has a useful interpretation.
Individual-arm and set-level events
For arm \(j\), a basic superiority claim can be expressed as \(\theta_j>0\). If a positive effect smaller than a clinically important amount would not justify action, use a prespecified margin \(\delta>0\) and define the meaningful-benefit event as \(\theta_j>\delta\). The value and units of \(\delta\) should be justified for the outcome and clinical context; it is not chosen after seeing which arm looks best.
The event for one arm is not the same as an event about the set of arms. For a common margin \(\delta\), several possible set-level targets are:
- At least one arm: \(\max_{1\leq j\leq K}\theta_j>\delta\).
- Every arm: \(\min_{1\leq j\leq K}\theta_j>\delta\).
- At least \(r\) arms: \(\sum_{j=1}^K I(\theta_j>\delta)\geq r\), where \(I(\cdot)\) equals 1 when its condition holds and 0 otherwise.
These targets answer different decision questions. The first supports a claim that there is at least one qualifying option; it does not say that every arm works. The second is a demanding claim about the entire set. The third describes a required number of qualifying arms. The count event is useful when a programme's decision depends on having more than one viable option. None of these definitions alone selects a posterior-probability threshold or dictates an action; those are separate design choices.
Worked Example: Specifying an Arm-Specific Estimand
Suppose a hypothetical trial compares two treatments, A and B, with a common control. The outcome is a symptom-improvement score at 12 weeks, measured in points, with larger scores better. The target is the mean effect in the trial-eligible population. The prespecified minimum clinically important mean improvement is 3 points. State the estimands and the meaningful-benefit events.
For each arm, define \(Y(j)\) as the 12-week score under assignment to that arm, and \(Y(0)\) as the score under assignment to control. The estimands are \(\theta_A=E[Y(A)]-E[Y(0)]\) and \(\theta_B=E[Y(B)]-E[Y(0)]\), both measured in points in the trial-eligible population. Since larger scores indicate greater improvement, positive contrasts favour treatment.
Using \(\delta=3\) points, the arm-specific events are \(\theta_A>3\) points and \(\theta_B>3\) points. The event that at least one arm meets the margin is \(\max(\theta_A,\theta_B)>3\) points; the event that both do is \(\min(\theta_A,\theta_B)>3\) points. The outcome, time, population, direction, and margin are specified, so these claims are interpretable. The definitions do not establish that the trial's data or analysis satisfy the assumptions needed to learn the effects.
Why set-level probabilities require joint information
An arm-specific posterior probability, such as \(P(\theta_A>\delta\mid D)\), summarizes uncertainty about one arm's event. A set-level probability, such as \(P(\text{at least one arm exceeds }\delta\mid D)\), concerns the joint configuration of all arm effects. It generally cannot be calculated from the arm-specific probabilities alone because the effects may be dependent. In a shared-control trial, estimates and posterior effects can be related through their common comparator; the joint distribution, rather than a collection of isolated margins, is what determines set events.
For two arm events \(A\) and \(B\), the identity \(P(A\cup B\mid D)=P(A\mid D)+P(B\mid D)-P(A\cap B\mid D)\) shows the issue directly: the marginal probabilities do not specify the overlap \(P(A\cap B\mid D)\). For example, if each arm event has posterior probability 0.50, the probability that at least one occurs is 0.50 if the events always occur together, but it is 1.00 if exactly one always occurs. Both cases have the same marginals. Thus, combining arm probabilities as if they were independent is not justified unless independence is an assumption supported by the model and design.
When joint posterior draws are available, a set-event probability can be estimated by checking the event in each draw and averaging the resulting indicators. For \(M\) draws \(\boldsymbol{\theta}^{(m)}=(\theta_1^{(m)},\ldots,\theta_K^{(m)})\), the estimate for event \(A\) is \(\widehat{P}(A\mid D)=M^{-1}\sum_{m=1}^{M} I(\boldsymbol{\theta}^{(m)}\in A)\). Each indicator is 1 if that draw satisfies the complete event and 0 otherwise. This is a Monte Carlo estimate of the posterior probability: when the draws represent the joint posterior, the average approximates the posterior expectation of the event indicator, which is \(P(A\mid D)\). The draws must preserve the relevant joint dependence.
Worked Example: Classifying Individual and Set Claims
In a hypothetical posterior draw for three arms, suppose the benefit-oriented effects are \((\theta_A,\theta_B,\theta_C)=(4,2,5)\) points, with a meaningful-benefit margin of 3 points. Determine which individual arms qualify, and evaluate three set-level events for this draw.
Arm A qualifies because \(4>3\); arm B does not because \(2\leq3\); arm C qualifies because \(5>3\). Thus the number of qualifying arms is \(I(4>3)+I(2>3)+I(5>3)=1+0+1=2\).
The “at least one” event is true because two arms exceed the margin. The “every arm” event is false because B does not exceed it. The “at least two arms” event is true because the count is 2. This one draw classifies events; it is not itself a posterior probability. A probability requires evaluating the event across the joint posterior distribution, not merely inspecting a single plausible configuration.
Worked Example: Estimating Marginal and Set-Level Probabilities From Draws
Suppose five hypothetical joint posterior draws for three benefit-oriented treatment effects, in points, are shown below. Use the margin \(\delta=3\) points and estimate each arm's meaningful-benefit probability and the probability that at least two arms meet the margin.
| Draw | Arm A | Arm B | Arm C | Number above 3 |
|---|---|---|---|---|
| 1 | 5 | 2 | 1 | 1 |
| 2 | 4 | 7 | 1 | 2 |
| 3 | 2 | 6 | 5 | 2 |
| 4 | 8 | 1 | 3.2 | 2 |
| 5 | 1 | 2 | 4 | 1 |
For each draw, count an arm only when its effect is strictly greater than 3 points. The counts are 1, 2, 2, 2, and 1: draw 2 has both A and B above 3, and draw 4 has both A and C above 3. Therefore, the estimated probability that at least two arms exceed the margin is \(\widehat{P}(\text{at least two}\mid D)=3/5=0.60\).
Across the draws, A exceeds 3 in draws 1, 2, and 4, so its estimated marginal probability is \(3/5=0.60\). B exceeds 3 in draws 2 and 3, giving \(2/5=0.40\). C exceeds 3 in draws 3, 4, and 5, giving \(3/5=0.60\). The probabilities are only rough Monte Carlo estimates based on five draws, not precise summaries. The example also shows why checking each draw's full vector matters: the set event depends on how arm effects occur together.
Justification, checks, and alternatives
The indicator calculation follows from a basic probability identity. For any event \(A\), its indicator \(I(\boldsymbol{\theta}\in A)\) is 1 when the draw belongs to \(A\), and 0 otherwise. Its posterior expectation is therefore \(E[I(\boldsymbol{\theta}\in A)\mid D]=1\cdot P(A\mid D)+0\cdot P(A^c\mid D)=P(A\mid D)\). Averaging the indicators over representative joint posterior draws approximates this expectation. If draws are correlated, as can occur in simulation output, the same average remains the natural estimate, but its Monte Carlo precision depends on the effective information in the simulation rather than simply the number of stored draws.
The principal alternative is to report only arm-specific probabilities. That can be appropriate when the decision is explicitly about a named arm, but it does not answer a whole-set question. Conversely, a single set-level probability can hide which arm drives the result, so reporting it alone may be inadequate for choosing among treatments. A useful report aligns the summaries with the decision: specify the individual estimands, the set event of interest, and any arm-level information needed to interpret the set result. These are different summaries, not competing estimates of one target.
A Bayesian posterior probability is not a frequentist familywise error rate and should not be described as the probability that a null hypothesis is true. It is a probability, conditional on the data and the specified Bayesian model, that the parameter vector belongs to a defined event. A decision threshold and the consequences of acting on that probability require their own justification. The event definition does not guarantee that the decision will have a desired frequentist error rate.
Before interpreting a result, check the following:
- Direction and scale: verify that positive values mean benefit and that the margin uses the same units as the estimand.
- Target: confirm that population, outcome, time point, and treatment strategies match the intended claim.
- Boundary: distinguish \(>\delta\) from \(\geq\delta\). For a continuous posterior the boundary often has probability zero, but that should not be assumed for every model or estimand.
- Joint draws: ensure that set-event calculations use draws of the full effect vector, not separately generated marginals that discard dependence.
- Monte Carlo stability: repeat or extend the simulation and check that the estimated event probability is not changing materially; a few draws, as in the illustration, are not adequate for a final analysis.
A frequent mistake is to see that two arms each have a marginal posterior probability of 0.60 and conclude that the probability both succeed is \(0.60\times0.60=0.36\). That multiplication is valid only under posterior independence. Without that assumption, the joint probability must be obtained from the joint distribution; multiplying marginal probabilities can misstate the set-level target. Another common error is to call “at least one arm exceeds the margin” an arm-specific claim. It is a set-level claim because its truth depends on the vector of effects and permits different qualifying arms in different posterior draws.
Check Your Understanding
Answer each question, then compare with the solutions below.
- For a higher-is-better outcome measured in points, write the arm-specific mean treatment-effect estimand against a common control and state what a positive value means.
- With three arms and a 2-point benefit margin, write the event that at least two arms exceed the margin.
- Two arm events each have posterior probability 0.50. Can their “at least one” probability be found by adding the marginal probabilities? Explain.
- Across 20 joint posterior draws, 13 satisfy a specified set-level event. What is the Monte Carlo estimate of its posterior probability, and what does it mean?
Solutions. (1) For arm \(j\), \(\theta_j=E[Y(j)]-E[Y(0)]\), in points in the stated population and at the stated time. A positive value means a higher mean outcome under the experimental arm. (2) \(\sum_{j=1}^{3}I(\theta_j>2)\geq2\), with effects and margin in points. (3) No. Adding gives 1.00 before accounting for overlap; the union probability is \(0.50+0.50-P(A\cap B\mid D)\), so the joint probability is needed. (4) \(13/20=0.65\). It estimates the posterior probability, given the data and model, that the parameter vector satisfies that set-level event; it is not a probability that a null hypothesis is true.