From Posterior Draws to a Benefit Probability
A clinical question about treatment benefit often concerns a probability, not just a posterior mean or interval. For example: given the observed trial data, how much posterior probability is there that treatment reduces the event risk by more than five percentage points? Posterior simulation answers this by evaluating the clinical criterion on each draw. The result is a proportion of draws, with numerical uncertainty because only finitely many draws are available.
Suppose the endpoint is an adverse event, and let \(p_T\) and \(p_C\) denote the event risks under treatment and control for the target population. Define the absolute risk reduction as \(B=p_C-p_T\), so positive values favor treatment. For a specified threshold \(\delta>0\), the target quantity is \(q=P(B>\delta\mid D)\), where \(D\) denotes the observed trial data. This is a posterior probability conditional on the model, prior, and data. It is not the proportion of trial participants who benefited, nor a frequentist probability that the treatment effect is fixed and exceeds the threshold.
The draws must be joint posterior draws: \(p_T^{(m)}\) and \(p_C^{(m)}\) in a pair come from the same iteration of the posterior simulation. Pairing preserves the posterior dependence between the arm-specific risks. If risks are obtained by standardizing a regression model, calculate both risks within each draw using the same target population and that draw’s model parameters, then form the difference. The target population and threshold should be clearly specified; changing either changes the probability being estimated.
The inequality is part of the estimand. If the question is whether the reduction is at least \(\delta\), use \(B^{(m)}\geq\delta\) instead. The distinction matters when the posterior has point mass at the threshold, as can happen with discrete models or rounded quantities. For a continuous posterior distribution, exact equality typically has probability zero, but the intended criterion should still be written precisely.
Why the Draw Proportion Estimates the Probability
Assume first that the posterior draws are independent and identically distributed. Each \(I_m\) is then a Bernoulli random variable with probability \(q\) of equalling 1. Consequently, \(E(I_m\mid D)=q\), and the average of the indicators estimates \(q\). Its sampling variability here is Monte Carlo variability: it describes how the estimate varies across repeated simulations from the same posterior, not how uncertain the posterior itself is about treatment benefit.
The second equality follows by adding the variances of \(M\) independent indicators and dividing by \(M^2\). Since \(q\) is unknown, estimate the Monte Carlo standard error (MCSE) by substituting \(\hat{q}\):
An approximate 95% Monte Carlo interval is \(\hat{q}\pm1.96\widehat{SE}_{MC}(\hat{q})\). It describes numerical precision in estimating the posterior probability with the simulation. It is not a 95% credible interval for \(B\), and it does not express posterior uncertainty about the treatment effect. The normal approximation is most useful when there are enough effectively independent indicators and \(\hat{q}\) is not extremely close to 0 or 1. Endpoints outside the interval from 0 to 1, or very few successes or failures, signal that this approximation is unsuitable.
Worked Example: Independent Draws and a Five-Percentage-Point Reduction
Suppose a hypothetical trial analysis produces 520 independent joint posterior draws for an adverse-event risk reduction \(B=p_C-p_T\). In 372 draws, \(B\) exceeds \(\delta=0.05\), a five-percentage-point reduction. The proportion is \(372/520=0.7154\), so the estimated posterior probability is \(\hat{q}\approx0.715\) when reported to three decimal places.
Using that rounded estimate in the MCSE formula gives \(\widehat{SE}_{MC}(\hat{q})\approx\sqrt{0.715(0.285)/520}=\sqrt{0.0003919}\approx0.0198\). Thus the approximate 95% Monte Carlo interval is \(0.715\pm1.96(0.0198)\), or about 0.676 to 0.754.
The interpretation is that the simulated estimate of the posterior probability is about 71.5%, with an approximate Monte Carlo margin of error of 3.9 percentage points. This interval concerns simulation precision only. It does not mean there is a 95% probability that the posterior probability \(q\) lies between 0.676 and 0.754.
Autocorrelation and Effective Information
Posterior draws from MCMC are generally not independent. The indicator sequence \(I_1,\ldots,I_M\) can be autocorrelated even when the underlying parameter traces appear to move smoothly through the posterior. Therefore, the independent-draw formula with \(M\) in the denominator can understate Monte Carlo error.
For a stationary indicator sequence with autocorrelation \(\rho_k\) at lag \(k\), the variance of its average is approximately \(\operatorname{Var}(\hat{q}\mid D)=q(1-q)M^{-1}(1+2\sum_{k=1}^{\infty}\rho_k)\). This motivates an effective sample size for the indicator, \(M_{\mathrm{eff},I}=M/(1+2\sum_{k=1}^{\infty}\rho_k)\). It is the effective sample size of the criterion indicators, not necessarily the effective sample size of either risk parameter. In practice, software estimates the relevant autocorrelation or long-run variance from the sampled indicator sequence. Batch means or spectral variance estimators can also estimate the MCSE directly. These estimates themselves can be unstable when there are too few batches or too little effective information.
This approximation relies on a sufficiently well-behaved, stationary simulation with enough effective information to estimate the long-run variance. It does not repair a chain that has failed to explore the posterior. Assess convergence and mixing using the standard diagnostics for posterior simulation, then examine the derived indicator sequence as well. An indicator can be poorly mixed if the chain spends long stretches on one side of the threshold, even when individual parameter diagnostics are less alarming. Multiple chains can help reveal such problems, but agreement across chains is not a guarantee that all posterior regions have been found.
Worked Example: Adjusting for Autocorrelation in the Benefit Indicator
In a hypothetical analysis, suppose the estimated posterior probability of exceeding the threshold is \(\hat{q}=0.730\). Although the analysis retained many MCMC iterations, the estimated effective sample size for the indicator sequence is only \(M_{\mathrm{eff},I}=200\). Use that effective sample size, rather than the raw number of iterations, for a rough MCSE calculation.
The estimated MCSE is \(\sqrt{0.730(1-0.730)/200}=\sqrt{0.1971/200}=\sqrt{0.0009855}\approx0.0314\). A rough 95% Monte Carlo interval is \(0.730\pm1.96(0.0314)\). Since \(1.96(0.0314)=0.061544\), the endpoints are approximately 0.668 to 0.792.
The interval is relatively wide: simulation error alone is around 6.2 percentage points in either direction at this approximate level. The appropriate response is to report the limited numerical precision and, if greater precision is needed, improve the effective information through longer or better-mixing simulation. Simply retaining more highly correlated iterations may add little.
Applying the Criterion Draw by Draw
The calculation is simple, but correct construction of each draw-level effect is essential. The following sequence makes the estimand and computation explicit.
State the endpoint, target population, effect orientation, and threshold. For an adverse event, \(B=p_C-p_T\) makes a positive reduction favor treatment.
For each iteration \(m\), retain risks from the same joint posterior draw and the same target-population definition.
Compute \(B^{(m)}=p_C^{(m)}-p_T^{(m)}\), then set \(I_m=1\) if \(B^{(m)}>\delta\), and 0 otherwise.
Average the indicators to estimate \(q\), then estimate MCSE using independent-draw assumptions only when justified; otherwise account for autocorrelation in the indicator sequence.
Worked Example: Why the Pairing of Risks Matters
Consider eight hypothetical joint draws for an adverse-event endpoint, with threshold \(\delta=0.05\). The table gives each draw’s control risk, treatment risk, and resulting reduction. For example, in draw 1 the reduction is \(0.20-0.18=0.02\).
| Draw \(m\) | \(p_C^{(m)}\) | \(p_T^{(m)}\) | \(B^{(m)}=p_C^{(m)}-p_T^{(m)}\) | \(I_m\) |
|---|---|---|---|---|
| 1 | 0.20 | 0.18 | 0.02 | 0 |
| 2 | 0.22 | 0.15 | 0.07 | 1 |
| 3 | 0.19 | 0.09 | 0.10 | 1 |
| 4 | 0.25 | 0.19 | 0.06 | 1 |
| 5 | 0.17 | 0.11 | 0.06 | 1 |
| 6 | 0.24 | 0.16 | 0.08 | 1 |
| 7 | 0.21 | 0.18 | 0.03 | 0 |
| 8 | 0.20 | 0.11 | 0.09 | 1 |
Six of the eight draw-level reductions exceed 0.05, so \(\hat{q}=(0+1+1+1+1+1+0+1)/8=6/8=0.75\). If these were independent draws, the plug-in MCSE would be \(\sqrt{0.75(0.25)/8}=\sqrt{0.0234375}\approx0.1531\). This is very imprecise, and a normal Monte Carlo interval would extend above 1; that is a warning not to rely on the normal approximation with only eight draws. The example illustrates the calculation, not an adequate simulation size.
Pairing is consequential. Combining treatment risks from one set of draws with control risks from an independently reordered set can erase or distort their posterior dependence, changing the distribution of \(B\) and the estimated probability. Compute the difference within each original joint draw; do not estimate this probability from separate summaries of the marginal risks.
Diagnostics, Alternatives, and Limitations
Before interpreting \(\hat{q}\), check that the simulation represents the intended joint posterior and that the benefit indicator is sufficiently well sampled. Useful warning signs include very low effective sample size for \(I_m\), long runs of zeros or ones, substantial disagreement in indicator averages across chains, and an MCSE large relative to the precision needed for the scientific report. If nearly all draws satisfy the criterion, the estimate may be close to 1 but still have poor relative precision for the small probability of failure; inspect both successes and failures rather than relying on the estimate alone.
An exact posterior calculation or numerical integration can replace simulation when it is available and computationally tractable. For a complicated posterior, draw-based indicator averaging is direct and does not require a special distribution for the treatment effect. Estimating the posterior distribution of \(B\) and evaluating its tail probability is another route, but it must preserve the same joint draws and can introduce additional approximation if a smooth distribution is fitted. For the direct indicator method, no smoothing is needed: each draw is classified by the stated clinical criterion.
The method has limits. It estimates the probability implied by the chosen model, prior, endpoint definition, target population, and threshold; it does not establish that those choices are appropriate. Monte Carlo precision cannot compensate for model misspecification, biased trial data, poor posterior exploration, or an unclear treatment-effect orientation. Also, a small estimated probability is not proof that benefit is impossible, just as a large one does not by itself establish a clinical or regulatory decision. Interpretation should keep the posterior quantity and its simulation precision distinct.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- For an adverse-event endpoint, define a risk reduction that is positive when treatment is beneficial. If the meaningful threshold is 0.04, write the draw-level indicator for a strict exceedance criterion.
- In 800 independent posterior draws, 624 satisfy the criterion. Calculate \(\hat{q}\) and its estimated MCSE.
- An MCMC estimate is \(\hat{q}=0.60\), based on 4,000 retained iterations, but the indicator effective sample size is 250. Calculate the appropriate approximate MCSE using that effective sample size.
- Why should treatment and control risks be paired by iteration when calculating a posterior probability for their difference?
Solutions. (1) Set \(B^{(m)}=p_C^{(m)}-p_T^{(m)}\), and \(I_m=\mathbf{1}\{B^{(m)}>0.04\}\). (2) \(\hat{q}=624/800=0.780\), and \(\widehat{SE}_{MC}\approx\sqrt{0.780(0.220)/800}=\sqrt{0.0002145}\approx0.0146\). (3) Use the indicator effective sample size: \(\widehat{SE}_{MC}\approx\sqrt{0.60(0.40)/250}=\sqrt{0.00096}\approx0.0310\); using 4,000 as if independent would understate the error. (4) Risks from the same iteration form a joint posterior draw and preserve dependence. Reordering or independently combining marginal draws changes the distribution of the difference, so it can change the probability that the difference exceeds the threshold.