Tutorials › Multi-Arm Multi-Stage (MAMS) Designs › Posterior Decisions After Data-Dependent Arm Selection

Home  /  Tutorials  /  Bayesian Multi-Arm Approaches

Bayesian Multi-Arm Approaches · Tutorial 413 of 499

Posterior Decisions After Data-Dependent Arm Selection

Determine when arm selection changes a posterior success probability, and calculate decisions using the likelihood for the data and selection information actually observed.

Advanced 12 min read

What You'll Learn

  • Define a posterior success event for an arm and distinguish it from the rule that selects an arm.
  • Explain why a fully observed, deterministic selection rule adds no separate likelihood factor.
  • Construct a likelihood that includes selection information when outcomes are missing or only selection status is recorded.
  • Update Beta-binomial posteriors using both interim and subsequent outcomes.
  • Diagnose double-counting, omitted-data, and prior-dependence errors in post-selection decisions.

Why the Selection History Matters

In a multi-arm trial, interim outcomes may determine which arm continues, which arm is examined more closely, or which arms enter a final decision. A posterior success probability is a probability about an arm’s unknown response parameter, conditional on observed information. The central question is therefore not simply whether selection occurred, but what information about the selection and the outcomes is in the analysis.

If all the data used to select an arm are available, and selection followed a prespecified rule based only on those data, the selection label is already determined by the data. It does not need to be counted as a second, independent piece of evidence. If some selection-driving outcomes were not recorded or are omitted from the analysis, however, the fact that an arm was selected can itself carry information and must be represented in the likelihood.

Definition: Let \(\theta_i\) be the response probability for arm \(i\), and let \(\theta_*\) be the prespecified response threshold defining success. The posterior success probability for arm \(i\), given observed information \(O\), is \(P(\theta_i>\theta_*\mid O)\). A posterior decision compares this probability with a prespecified decision threshold \(\gamma\). The information \(O\) must specify both the outcomes used and any selection information not already determined by those outcomes.

This is a Bayesian conditioning question, not an automatic correction for having looked at multiple arms. The earlier material on Bayesian multiplicity and posterior decision thresholds explains why a posterior rule need not be turned into a frequentist error-rate adjustment. Here the issue is narrower: defining the correct conditioning information after a data-dependent selection.

Likelihoods With and Without Selection Information

Suppose \(D\) denotes all outcomes observed by the interim decision, and a deterministic rule chooses arm \(S=g(D)\). For example, \(g\) might choose the arm with the largest observed response rate, with a prespecified tie rule. If the full \(D\) is retained, then observing \(S\) provides no extra information once \(D\) is known: \(S\) is already fixed by \(D\). The posterior is

$$ p(\boldsymbol{\theta}\mid D,S=g(D)) \propto p(D\mid\boldsymbol{\theta})p(\boldsymbol{\theta}). $$

The indicator for the rule, \(I\{S=g(D)\}\), equals one for the observed data and is constant as a function of \(\boldsymbol{\theta}\). It therefore cancels from the posterior normalization. Multiplying the likelihood for \(D\) by a second “probability of being selected” based on the same \(D\) would count the selection evidence twice.

A more general notation makes the missing-data case clear. Write \(D_{\mathrm{obs}}\) for retained outcomes and \(D_{\mathrm{mis}}\) for outcomes that influenced selection but were not retained. If selection is deterministic given the complete data, then

$$ p(D_{\mathrm{obs}},S=s\mid\boldsymbol{\theta}) = \sum_{d_{\mathrm{mis}}} p(D_{\mathrm{obs}},d_{\mathrm{mis}}\mid\boldsymbol{\theta}) I\{g(D_{\mathrm{obs}},d_{\mathrm{mis}})=s\}, $$

with an integral replacing the sum for continuous missing data. The selection indicator restricts the possible missing outcomes. In general, the resulting likelihood depends on \(\boldsymbol{\theta}\), so selection status cannot be discarded.

These statements assume that the model for the outcomes and the selection rule are correctly specified. A randomized selection rule can also be handled: include its known selection probability in the likelihood. If the randomization probability depends only on observed history and known design quantities, it is constant in the unknown outcome parameters conditional on that history. By contrast, selection probabilities that depend on unobserved outcomes or unknown parameters require explicit modelling.

Worked Example: Selecting the Arm With the Higher Observed Rate

Worked Example: Selecting the Arm With the Higher Observed Rate

Suppose two hypothetical arms have independent priors \(\theta_A,\theta_B\sim\operatorname{Beta}(1,1)\). Their binary interim outcomes are conditionally independent given their response probabilities, and each arm has 20 participants. Arm A has 12 responses and 8 nonresponses; arm B has 8 responses and 12 nonresponses. A prespecified rule selects the arm with the higher observed response rate. Calculate the posterior success probability for the selected arm when success means \(\theta_A>0.5\).

The observed rates are \(12/20=0.600\) for A and \(8/20=0.400\) for B, so A is selected. Under the Beta-binomial model, a \(\operatorname{Beta}(a,b)\) prior updated with \(y\) responses among \(n\) participants gives a \(\operatorname{Beta}(a+y,b+n-y)\) posterior. Thus,

$$ \theta_A\mid D\sim\operatorname{Beta}(1+12,1+8) =\operatorname{Beta}(13,9). $$

For positive integers \(a,b\), the beta-binomial identity gives \(P(\operatorname{Beta}(a,b)>x)=P\{\operatorname{Binomial}(a+b-1,x)\leq a-1\}\). Therefore,

$$ P(\theta_A>0.5\mid D) = \sum_{j=0}^{12}{21\choose j}(0.5)^j(0.5)^{21-j} \approx 0.8083. $$

The posterior probability that A’s response probability exceeds 0.5 is about 0.8083. If the prespecified decision threshold is \(\gamma=0.80\), this probability just exceeds the threshold. This calculation uses all 40 interim outcomes. Because the selected label follows deterministically from those outcomes, it is not an additional likelihood factor.

When Selection Status Adds Information

If only the selection status is available, the posterior need not equal the prior, even if the selected arm’s individual outcome counts are unavailable. In that case, the likelihood contribution is the probability of the observed selection event under a proposed parameter value. The following calculation illustrates the distinction in a simple binary-outcome setting.

Worked Example: Selection Recorded but Interim Counts Missing

For one hypothetical arm, let \(\theta\sim\operatorname{Beta}(1,1)\). Two interim binary outcomes were collected, but the counts were not retained. The arm was selected to continue if it had at least one response. Calculate the posterior probability that \(\theta>0.5\), given only that the arm was selected.

Conditional on \(\theta\), the probability of selection is the probability of at least one response in two trials:

$$ P(S=1\mid\theta)=1-(1-\theta)^2=2\theta-\theta^2. $$

The posterior density is proportional to this selection likelihood times the uniform prior density. Its normalizing constant is

$$ \int_0^1(2\theta-\theta^2)\,d\theta = \left[\theta^2-\frac{\theta^3}{3}\right]_0^1 =\frac{2}{3}. $$

The unnormalized posterior probability mass above 0.5 is

$$ \int_{0.5}^1(2\theta-\theta^2)\,d\theta = \frac{2}{3}-\left(\frac{1}{4}-\frac{1}{24}\right) =\frac{11}{24}. $$

Consequently,

$$ P(\theta>0.5\mid S=1) = \frac{11/24}{2/3} =\frac{11}{16} =0.6875. $$

The prior probability was 0.5, whereas selection alone raises the posterior probability to 0.6875. The selection rule conveys evidence because the outcomes that triggered it were not otherwise observed. If the two outcomes had been retained, their actual values—not an additional duplicate selection factor—would form the outcome likelihood.

Updating After Selection and Continued Observation

A selected arm may contribute further outcomes after the interim. Under a correctly specified model, those outcomes update the posterior along with the interim outcomes. If the full interim data are retained, selection by a known rule does not erase or discount them. The likelihood is built from the observed outcomes at both stages.

Worked Example: Combining Interim and Subsequent Outcomes

Suppose two hypothetical arms each have 20 interim participants. Arm A has 10 responses and 10 nonresponses; arm B has 9 responses and 11 nonresponses. The rule selects the arm with the higher interim response rate, so A continues. A then has 10 additional participants, with 7 responses and 3 nonresponses. Assume a \(\operatorname{Beta}(1,1)\) prior for A’s response probability and conditionally independent binary outcomes with a constant response probability within A. Calculate A’s posterior probability of success for the threshold \(\theta_A>0.5\).

The interim data select A because \(10/20=0.500\) exceeds \(9/20=0.450\). A’s complete observed outcomes are 17 responses and 13 nonresponses among 30 participants. Therefore,

$$ \theta_A\mid D\sim\operatorname{Beta}(1+17,1+13) =\operatorname{Beta}(18,14). $$

Using the beta-binomial identity,

$$ P(\theta_A>0.5\mid D) = P\{\operatorname{Binomial}(31,0.5)\leq17\} = \sum_{j=0}^{17}{31\choose j}(0.5)^{31} \approx 0.7634. $$

For example, the sum can be evaluated as \(0.5+{31\choose16}/2^{31}+{31\choose17}/2^{31}\), which is approximately \(0.7634\). At a decision threshold of \(\gamma=0.80\), A does not meet the posterior success rule. Being selected to continue is not itself a declaration of success; the posterior decision is made from the specified success event and decision threshold.

A Short Justification for the Full-Data Rule

The cancellation of a deterministic selection rule follows directly from Bayes’ formula. Suppose the likelihood is written for both the complete data and the observed selection:

$$ p(\boldsymbol{\theta}\mid D,S=s) \propto p(D,S=s\mid\boldsymbol{\theta})p(\boldsymbol{\theta}) = p(D\mid\boldsymbol{\theta})I\{g(D)=s\}p(\boldsymbol{\theta}). $$

For the realized data, \(s=g(D)\), so \(I\{g(D)=s\}=1\) for every value of \(\boldsymbol{\theta}\). The factor has no parameter dependence and disappears when the posterior is normalized. If \(s\ne g(D)\), the observed data and recorded selection are inconsistent with the stated rule; that is a data or implementation problem, not a reason to add a selection adjustment.

The argument changes when only part of \(D\) is available. Then the indicator cannot be treated as a known constant across all possible complete data: some missing outcomes would have produced the observed selection and others would not. Summing or integrating over those possibilities gives a selection likelihood that may vary with the unknown parameters, as in the selection-only example.

Diagnostics, Comparisons, and Limitations

Before calculating a post-selection posterior probability, verify the information set and the model. In particular, check:

  • Which outcomes were used to select? Reconcile the selection rule with the interim dataset and verify any tie-breaking or randomization rule.
  • Are those outcomes in the analysis? If they are, selection is usually already encoded in the full data. If not, determine what selection status or summaries were retained and represent the missing outcomes accordingly.
  • Does the likelihood match the design? The Beta-binomial examples assume binary outcomes, a common response probability within the arm, and conditionally independent outcomes. Changing outcome definitions or allowing response probabilities to vary over time requires a different outcome model.
  • Does the prior connect arms? With independent arm-specific priors, unobserved competing-arm data can sometimes integrate out without changing a selected arm’s posterior. Under a shared or hierarchical prior, those data can inform shared parameters and hence the selected arm. Do not assume that omitted competitor data are irrelevant without checking the model.
  • Are data truly missing or merely not used? If recorded selection-driving data exist, using only a selected-arm summary can discard information. Analyze the available outcomes with the stated likelihood whenever that is the intended analysis.

A useful comparison is between three analyses. A full-data Bayesian analysis conditions on all observed outcomes and applies the prior and likelihood to those outcomes; a deterministic selection label adds nothing further. A selection-only analysis conditions on the selection event when the triggering outcomes are unavailable. A frequentist selective analysis may instead condition on a selection event to study properties of a test or estimator. These are different inferential targets; a post-selection Bayesian posterior is not automatically a frequentist selection-adjusted significance calculation.

The principal limitation is that the posterior is only as reliable as its prior, outcome model, and account of what was observed. Selection does not fix a misspecified likelihood, recover outcomes that were never collected, or make an unrecorded adaptive rule ignorable. When the selection process cannot be reconstructed, report that limitation rather than silently treating the retained data as if they were selected independently of outcomes.

Common Mistake

Common mistake: “Arm A was selected, so I should multiply the likelihood for A’s observed outcomes by the probability that A would be selected.” If the full outcomes determining selection are already in the likelihood and selection is a deterministic function of them, this counts the same evidence twice. Use the full observed-data likelihood once. Include a selection-event likelihood when selection conveys information not already captured by the retained data—for example, when the outcomes that triggered selection are missing.
Key takeaway: A posterior decision after arm selection is defined by its conditioning information. State which outcomes were observed, which rule selected the arm, and whether selection adds information beyond the retained outcomes before calculating the posterior success probability.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

  1. All interim outcomes are retained, and the selected arm is the one with the larger observed response rate. Does the selection label require an extra likelihood factor?
  2. A uniform prior is used for \(\theta\), and an arm is selected if at least one of two binary interim outcomes is a response. The outcomes are missing. What is \(P(S=1\mid\theta)\)?
  3. With \(\theta\sim\operatorname{Beta}(1,1)\), the arm has 6 responses among 10 participants. What is its posterior distribution?
  4. Why can selection status matter when the outcomes that triggered selection were not retained?

Solutions. (1) No. If selection is deterministic from the fully observed outcomes, its indicator is constant given those outcomes and adds no parameter-dependent factor. (2) \(P(S=1\mid\theta)=1-(1-\theta)^2=2\theta-\theta^2\). (3) The posterior is \(\operatorname{Beta}(1+6,1+4)=\operatorname{Beta}(7,5)\). (4) The selection event restricts which missing outcomes could have occurred; its probability can depend on the unknown response parameter and therefore contributes likelihood information.

Previous

Next

Back to Tutorials
Clinical Biostats