Posterior Sampling With Nonconjugate Response Models

Why Draws Are Useful When Updates Are Not Closed-Form

For some response models, the posterior distribution can be written in a convenient family and updated by a short formula. For others, the likelihood and prior do not produce a posterior with a closed-form update. A Bayesian model can still be used in the latter case: numerical algorithms can generate draws that approximate the posterior, and those draws can estimate the probability that each treatment has the largest modeled response.

This is the link to Thompson sampling. Given the trial history, the allocation probability for an arm is its posterior probability of being best under the response summary specified by the model. With a nonconjugate model, this probability is generally estimated from posterior draws rather than calculated from a closed-form posterior expression. The calculation is only as meaningful as the response model and the definition of “best” used in it. Here, assume a specified scalar response summary where larger values are preferred; the draws do not themselves determine whether that summary is clinically appropriate.

Definition: Let $H$ denote the observed trial history and let $\theta$ denote the model parameters. For treatment $k$, let $m_k(\theta)$ be its modeled response summary. If the posterior probability of a tie for the largest $m_k(\theta)$ is zero, the Thompson-sampling allocation probability is $$ \pi_k(H)=P\!\left(m_k(\theta)>m_j(\theta)\text{ for every }j\ne k\mid H\right). $$ If ties have positive posterior probability, assign each tied draw among its tied arms using a specified rule—for example, divide its weight equally. The resulting allocation probabilities then sum to 1.

A posterior draw must represent uncertainty about the model parameters conditional on the trial history, not merely a fitted point estimate. For MCMC, draws are generated by a Markov chain whose stationary distribution is the target posterior. After any warm-up period, retained draws are used to approximate posterior quantities. Other valid posterior-sampling methods can be used in the same way, provided their draws represent the target posterior and their numerical error is assessed.

From Posterior Draws to Allocation Probabilities

Suppose $M$ posterior draws $\theta^{(1)},\ldots,\theta^{(M)}$ are available. For each draw, calculate every arm’s response summary and identify the draw’s winner. Define $w_k^{(m)}$ as arm $k$’s share of the winner weight on draw $m$: it is 1 for a unique winner and 0 for a nonwinner; for a tie, the specified tie rule assigns weights across the tied arms that sum to 1. Then the draw-based estimate is

\widehat{\pi}_k=\frac{1}{M}\sum_{m=1}^{M}w_k^{(m)}.

For example, if there are no ties, $w_k^{(m)}$ is the indicator that arm $k$ is the unique winner on draw $m$. On every draw the winner weights sum to 1, so

\sum_k\widehat{\pi}_k =\frac{1}{M}\sum_{m=1}^{M}\sum_k w_k^{(m)} =\frac{1}{M}\sum_{m=1}^{M}1 =1.

This identity depends on the weights covering all posterior draws and summing to 1 on each one. In particular, counting a tied arm as a winner for every arm tied at the maximum does not, in general, produce probabilities that sum to 1. A tie rule must be specified if posterior ties have positive probability.

Generate posterior draws.
Use a sampler targeting the posterior given the current trial history. Retain draws only after the chosen warm-up procedure.

Calculate arm summaries.
For each draw, evaluate the same modeled response summary for every treatment arm.

Assign winner weights.
Give weight 1 to a unique winner, or distribute the draw’s weight across tied winners using the specified rule.

Average the weights.
The average weight for each arm estimates its posterior probability of being best and can be used as its assignment probability.

Worked Example: Binary Responses With a Logistic Model

Consider a hypothetical three-arm trial with a binary response, where response probability is the modeled summary and larger is preferred. A logistic model could have separate arm effects and a prior on its coefficients. Because the resulting posterior need not have a closed-form update, suppose an MCMC algorithm has produced eight retained posterior draws. The table gives the modeled response probabilities for the arms on each draw.

Draw	Control	Arm A	Arm B	Unique winner
1	0.40	0.55	0.48	A
2	0.45	0.50	0.58	B
3	0.52	0.48	0.46	Control
4	0.38	0.61	0.64	B
5	0.44	0.53	0.51	A
6	0.57	0.54	0.50	Control
7	0.41	0.47	0.60	B
8	0.49	0.56	0.53	A

There are no ties in these displayed draws. Arm A wins 3 draws, Arm B wins 3, and Control wins 2. Thus the estimates are

\widehat{\pi}_{A}=\frac{3}{8}=0.375,\qquad \widehat{\pi}_{B}=\frac{3}{8}=0.375,\qquad \widehat{\pi}_{\mathrm{Control}}=\frac{2}{8}=0.250.

They sum to $0.375+0.375+0.250=1.000$. In a sampling step using these estimates, the trial would assign the next participant to A, B, or Control with the corresponding probabilities. These are estimated probabilities, not exact values: eight draws are far too few to expect high numerical precision.

Model-Based Summaries Can Depend on Covariates

A draw’s winner need not be determined by a single treatment coefficient. If treatment effects vary with patient characteristics, compare the arms using the response summary appropriate to the modeled target population. For example, a standardized response probability averages predictions across a specified covariate distribution. The same standardization must be applied to every arm on every draw; comparing coefficients alone can give a different answer from comparing the arm-level summaries.

Worked Example: Standardizing Logistic-Model Predictions

Suppose a hypothetical two-arm trial uses a logistic response model with binary covariate $x$, where $x=0$ denotes a lower-risk group and $x=1$ a higher-risk group. On one posterior draw, let the control linear predictor be $-1+0.8x$, and let the treatment add $0.7-0.5x$. The modeled response probability is $\operatorname{expit}(\eta)=1/(1+\exp(-\eta))$. For a target population with equal weights for the two covariate groups, calculate each arm’s standardized probability.

For control, the two linear predictors are $-1$ and $-1+0.8=-0.2$. The corresponding probabilities are $\operatorname{expit}(-1)=0.269$ and $\operatorname{expit}(-0.2)=0.450$. Thus its standardized probability is $(0.269+0.450)/2=0.360$, to three significant figures.

For treatment, the linear predictors are $-1+0.7=-0.3$ when $x=0$, and $-1+0.8+0.7-0.5=0$ when $x=1$. The probabilities are $\operatorname{expit}(-0.3)=0.426$ and $\operatorname{expit}(0)=0.500$. Its standardized probability is $(0.426+0.500)/2=0.463$. Treatment is the winner on this draw because $0.463>0.360$.

Now suppose six posterior draws give the following standardized probabilities. The first row is the calculation just completed.

Draw	Control	Treatment	Winner
1	0.360	0.463	Treatment
2	0.420	0.390	Control
3	0.310	0.375	Treatment
4	0.500	0.455	Control
5	0.390	0.410	Treatment
6	0.350	0.401	Treatment

Treatment wins 4 of 6 draws, so its estimated allocation probability is $4/6=0.667$; Control’s estimate is $2/6=0.333$. The calculation uses the stipulated equal-weight target population. A different target covariate distribution could produce different standardized probabilities and therefore different winners.

Why the Draw Average Estimates the Desired Probability

Let $W_k(\theta)$ be the winner weight for arm $k$, including the tie rule if needed. By definition, the posterior expectation of this weight is the allocation probability:

E\!\left[W_k(\theta)\mid H\right]=\pi_k(H).

If the draws were independent samples from the posterior, the sample average of $W_k$ would be an ordinary Monte Carlo estimator of this expectation. MCMC draws are generally dependent, but if the chain is run under conditions that give it the target posterior as its stationary distribution and it explores that distribution adequately, an ergodic average still converges to the posterior expectation as the number of draws increases. This justifies averaging winner weights; it does not guarantee that a finite run has small error.

The dependence affects precision. For a binary winner indicator with posterior mean $\pi_k$, independent draws would give approximate Monte Carlo standard error $\sqrt{\pi_k(1-\pi_k)/M}$. For correlated MCMC draws, the effective sample size is generally less than $M$. A common approximation is

M_{\mathrm{eff}}\approx \frac{M}{1+2\sum_{\ell=1}^{\infty}\rho_\ell}, \qquad SE_{\mathrm{MC}}(\widehat{\pi}_k)\approx \sqrt{\frac{\widehat{\pi}_k(1-\widehat{\pi}_k)}{M_{\mathrm{eff}}}},

where $\rho_\ell$ is the lag-$\ell$ autocorrelation of the winner-weight sequence. This is a numerical error assessment for the estimated probability, not a confidence interval for a clinical treatment effect.

Worked Example: Accounting for MCMC Dependence

Suppose a hypothetical analysis retains $M=1{,}200$ draws. Arm A is the unique winner on 756 draws, giving $\widehat{\pi}_A=756/1{,}200=0.630$. The MCMC diagnostics estimate an effective sample size of $M_{\mathrm{eff}}=300$ for the winner indicator.

Using the effective sample size, the approximate Monte Carlo standard error is

SE_{\mathrm{MC}}\approx \sqrt{\frac{0.630(1-0.630)}{300}} =\sqrt{0.000777} \approx 0.0279.

If the 1,200 draws were incorrectly treated as independent, the calculation would instead be $\sqrt{0.630(0.370)/1{,}200}\approx0.0139$, understating the estimated error by about a factor of two. The allocation estimate is still 0.630, but the precision assessment must reflect dependence in the chain.

Diagnostics, Alternatives, and Limitations

Check both the posterior computation and the model-based comparisons. For MCMC, inspect trace plots and convergence summaries across multiple chains, and check effective sample sizes for the winner weights as well as for important model parameters. A chain may estimate a parameter’s mean adequately while rarely changing which arm wins; the allocation probability can then remain imprecise. If chains remain in different regions or explore a multimodal posterior unevenly, their winner counts may disagree substantially.

Also check the response-model implementation: treatment coding, response coding, covariate transformations, and the construction of each arm’s response summary. A numerical sampler can converge to the posterior of a wrongly specified or incorrectly coded model. Posterior predictive checks can help reveal whether the model reproduces important features of the observed outcomes, but they do not prove that the model is correct. Missing outcomes and departures from the assumed response distribution also require appropriate modeling; generating more draws does not repair those problems.

When a conjugate model is available, its analytic posterior update and any available analytic allocation calculation can avoid MCMC error. In nonconjugate settings, posterior sampling is often more flexible than relying on a point estimate. A plug-in greedy rule chooses the arm with the largest fitted response at a single estimate; it does not average over posterior uncertainty and is not Thompson sampling. Other numerical approaches may also approximate posterior quantities, but each has its own approximation error and diagnostics.

The draw-based procedure has practical limits. Rare winners may require many effective draws to estimate accurately. Computational cost can be substantial if the model is refitted or resampled at every allocation step. Finally, a posterior probability that an arm has the largest modeled response is not by itself a statement about the size or clinical importance of a treatment difference. The probability concerns the model’s chosen response summary and its posterior uncertainty.

Common pitfall: Counting a tied draw as a winner for every arm at the maximum and then claiming the resulting probabilities sum to 1. If two arms tie on every draw, this rule gives each an estimated probability of 1. Instead, use a tie rule that divides each draw’s total weight across its tied winners, or use a specified random tie-break; the weights must sum to 1 on every draw.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In 500 posterior draws with no ties, Arm C is the unique winner 185 times. What is its estimated allocation probability?
On a draw, three arms have the same largest modeled response. Under equal division of tie weight, what winner weight does each receive on that draw?
An MCMC run retains many draws, but the winner indicator has strong positive autocorrelation. What happens to the effective sample size and the Monte Carlo error relative to an independent-draw calculation?
Why can selecting the arm with the largest response at a posterior mean parameter fail to give the Thompson-sampling allocation probabilities?

Solutions. (1) $\widehat{\pi}_C=185/500=0.370$. (2) Each receives $1/3$, so the three winner weights sum to 1. (3) Positive autocorrelation reduces effective sample size relative to the number of retained draws, increasing Monte Carlo error relative to the independent-draw calculation. (4) The plug-in rule compares arms at one parameter estimate, whereas Thompson sampling uses the posterior probability that each arm is best across parameter uncertainty; the two need not agree.