Binary Outcomes In Operating-Characteristic Simulations

Why simulate binary outcomes?

A binary endpoint records whether each participant experiences an event or response by a specified assessment time. Examples include response versus no response, or survival versus death by a fixed date. In a randomized trial, the event probability may differ between treatment groups. Operating-characteristic simulation asks how a prespecified analysis and decision rule perform when those probabilities take particular values.

The key is to distinguish the simulated data-generating scenario from the decision made after analysing one simulated trial. A scenario specifies, for example, a 30% response probability under control and a 45% response probability under treatment. A single simulated trial gives one possible set of binary outcomes under those probabilities. Repeating the process produces an estimate of how often the fixed decision rule would declare the trial successful in that scenario.

Definition: In a fixed-allocation, two-arm binary-outcome simulation, each control outcome is generated as a Bernoulli random variable with response probability \(p_C\), and each treatment outcome as a Bernoulli random variable with response probability \(p_T\). A scenario is a specified pair \((p_C,p_T)\), together with the trial design and analysis rule. The operating characteristic of interest may be the probability that the rule makes a specified decision under that scenario.

Specify the data-generating model first

For a simple parallel-group trial with \(n_C\) control participants and \(n_T\) treatment participants, let \(Y_{Ci}\) and \(Y_{Tj}\) denote their binary outcomes, coded 1 for response and 0 otherwise. The basic model assumes that, within each arm, outcomes are independent Bernoulli draws with a common arm-specific probability:

Y_{Ci}\sim \operatorname{Bernoulli}(p_C),\qquad Y_{Tj}\sim \operatorname{Bernoulli}(p_T).

The simple model also assumes that outcomes in the two arms are independent conditional on the scenario, allocation is fixed at the stated group sizes, and every participant has a defined outcome measured in the same way. It represents the trial as specified: it does not automatically account for clustering, changing response probabilities across sites or participants, or other departures from these assumptions. Such features require a data-generating model that represents them.

The response probabilities are scenario inputs, not estimates obtained from the simulated trial. A simulation may examine a clinically plausible control response probability alongside several treatment probabilities, including no improvement, a small improvement, and a clinically important improvement. If a larger value means a worse outcome, the direction of the contrast and the success rule must instead be defined to reflect clinical benefit.

Under the model, the numbers of responses in each arm have binomial distributions. Conditional on \(p_C\) and \(p_T\), the two response counts are independent:

X_C=\sum_{i=1}^{n_C}Y_{Ci}\sim\operatorname{Binomial}(n_C,p_C), \qquad X_T=\sum_{j=1}^{n_T}Y_{Tj}\sim\operatorname{Binomial}(n_T,p_T).

This count representation is useful for validating a generator and, in small designs, calculating decision probabilities exactly. It does not change the need to define the participant-level outcome or the analysis applied to each simulated trial.

Worked Example: Generating a small hypothetical trial

Consider a hypothetical fixed-allocation trial with five participants per arm, control response probability \(p_C=0.20\), and treatment response probability \(p_T=0.60\). To illustrate Bernoulli generation, suppose each outcome is generated by comparing a uniform draw \(U\) with its arm's response probability: the outcome is 1 if \(U<p\), and 0 otherwise. The hypothetical draws are:

Control: \(0.10, 0.70, 0.90, 0.30, 0.80\). Treatment: \(0.40, 0.20, 0.80, 0.10, 0.60\).

For control, only \(0.10<0.20\), so the outcomes are \(1,0,0,0,0\) and \(X_C=1\). For treatment, \(0.40<0.60\), \(0.20<0.60\), and \(0.10<0.60\), so the outcomes are \(1,1,0,1,1\) and \(X_T=4\). The simulated response proportions are \(1/5=0.20\) and \(4/5=0.80\). These are one possible realization, not the scenario probabilities themselves.

The expected response counts under the scenario are \(5(0.20)=1\) in control and \(5(0.60)=3\) in treatment. A simulated treatment count of four is therefore possible even though the specified treatment response probability is 0.60. A single generated trial need not reproduce its input probabilities exactly.

From generated outcomes to an operating characteristic

Before generating trials, define the trial-level analysis and decision rule. For example, a Bayesian rule might declare success when the posterior probability that the treatment response probability exceeds the control response probability is greater than a fixed cutoff. The rule must state the model, prior distributions, threshold, and direction of benefit. Once specified, it is held fixed across simulated trials and scenarios. The simulation evaluates that rule; changing it partway through would evaluate a different procedure.

Write \(D_r=1\) when the decision rule declares success in simulated trial \(r\), and \(D_r=0\) otherwise. For \(R\) independent simulated trials under the same scenario, the estimated decision rate is the proportion of trials declared successful:

\widehat{\pi}=\frac{1}{R}\sum_{r=1}^{R}D_r.

If the true probability of a success decision under this scenario is \(\pi\), then \(D_r\) is Bernoulli with probability \(\pi\), and \(\widehat{\pi}\) estimates \(\pi\). Conditional independence of the simulation replicates gives the Monte Carlo standard error:

SE(\widehat{\pi})=\sqrt{\frac{\pi(1-\pi)}{R}} \quad\text{and, estimated from the simulation,}\quad \widehat{SE}(\widehat{\pi})=\sqrt{\frac{\widehat{\pi}(1-\widehat{\pi})}{R}}.

This standard error describes simulation noise in estimating a decision rate. It is not the standard error of a treatment effect in one clinical trial, and it does not describe uncertainty in the scenario probabilities. If the decision rate is near 0 or 1, the plug-in estimate can be zero even with a finite number of replicates; that does not prove the true rate is exactly 0 or 1.

Worked Example: A decision rate from five simulated trials

Return to the hypothetical scenario \(p_C=0.20\), \(p_T=0.60\), with one participant per arm for a deliberately small calculation. Suppose the analysis uses independent \(\operatorname{Beta}(1,1)\) priors for \(p_C\) and \(p_T\), and declares success when \(P(p_T>p_C\mid\text{data})>0.80\). With one participant in each arm, observing a treatment response and a control nonresponse gives posterior distributions \(\operatorname{Beta}(2,1)\) and \(\operatorname{Beta}(1,2)\), respectively. The posterior probability of benefit is:

P(p_T>p_C\mid X_T=1,X_C=0) =\int_0^1 2x\{1-(1-x)^2\}\,dx =\int_0^1(4x^2-2x^3)\,dx =\frac{4}{3}-\frac{1}{2} =\frac{5}{6}\approx0.8333.

Thus this outcome pattern meets the 0.80 threshold. If both participants respond, or neither responds, the posterior distributions are identical and the probability is 0.50. If only the control participant responds, symmetry gives \(P(p_T>p_C\mid\text{data})=1/6\). The rule therefore declares success exactly when the treatment participant responds and the control participant does not.

The probability of that pattern under the specified scenario is \(p_T(1-p_C)=0.60(0.80)=0.48\). Now suppose five independent simulated trials produce the following outcome pairs, listed as (control, treatment): (1,1), (0,1), (0,0), (0,1), (0,0). The rule declares success in trials 2 and 4, so \(\widehat{\pi}=2/5=0.400\). This estimate differs from 0.48 because five replicates leave substantial Monte Carlo variation. This tiny design demonstrates the calculation, not a clinically suitable trial size.

Justification: exact decision probability

The simulation estimate has a direct probability interpretation. Let \(d(x_C,x_T)\) equal 1 if the fixed analysis declares success for counts \(x_C,x_T\), and 0 otherwise. The joint probability of those counts under the independent binomial model is the product of their binomial probabilities. Summing that probability over all count pairs for which the rule declares success gives the exact decision probability:

\pi =\sum_{x_C=0}^{n_C}\sum_{x_T=0}^{n_T} d(x_C,x_T) {n_C\choose x_C}p_C^{x_C}(1-p_C)^{n_C-x_C} {n_T\choose x_T}p_T^{x_T}(1-p_T)^{n_T-x_T}.

This follows by partitioning the event “the rule declares success” into the mutually exclusive possible count pairs and adding their probabilities. In the one-participant example, \(d(x_C,x_T)=1\) only for \((x_C,x_T)=(0,1)\), so the sum reduces to \(p_T(1-p_C)=0.48\). For more complex analyses or larger designs, exact enumeration may be inconvenient; repeated simulation estimates the same probability by averaging the decision indicators.

Worked Example: Quantifying Monte Carlo uncertainty

Suppose 1,000 independent simulated trials under one specified binary-outcome scenario yield 620 success decisions. The estimated decision rate is:

\widehat{\pi}=\frac{620}{1000}=0.620.

Its estimated Monte Carlo standard error is:

\widehat{SE}(\widehat{\pi}) =\sqrt{\frac{0.620(1-0.620)}{1000}} =\sqrt{0.0002356} \approx0.01535.

A rough normal-approximation interval for the simulation error is \(0.620\pm1.96(0.01535)\), or approximately \(0.590\) to \(0.650\). This is an interval for the underlying decision rate suggested by the Monte Carlo experiment, not a clinical interval for \(p_T-p_C\). If more precision is needed, increase \(R\); increasing the number of simulated participants per trial changes the trial design and is not a substitute for increasing the number of simulation replicates.

Scenario choice, diagnostics, and alternatives

A decision rate is meaningful only relative to its scenario. Choose response probabilities that answer a clinical question: for example, the rate under no treatment benefit, under a minimally important benefit, and under a larger benefit. The response probabilities should be plausible for the stated population, endpoint definition, and follow-up period. A rate estimated at one pair of probabilities does not automatically describe performance at another pair.

Several checks can reveal errors before a large simulation run:

Check the generator: across many generated outcomes in one arm, the average should be close to its specified response probability. Large discrepancies may indicate a coding error, swapped arm probabilities, or an incorrect outcome threshold.
Check the decision direction: under a rule intended to favour higher response, a scenario with substantially lower treatment response should not routinely produce the same success rate as a beneficial scenario. If it does, inspect the treatment and control labels and the inequality in the rule.
Check the replicate unit: one decision indicator must correspond to one complete simulated trial, not to one participant. Counting individual responses as decisions estimates the wrong quantity.
Check Monte Carlo precision: report \(R\) and the estimated standard error. A rate based on few replicates can be too noisy to distinguish nearby scenarios.
Check the model and analysis match: the generation process and the fitted trial analysis need not be identical, but any difference should be intentional. A mismatch can affect decision rates and should not be hidden by the simulation code.

Exact enumeration is a useful alternative when the count space is small and the analysis decision can be evaluated for every count pair. It avoids Monte Carlo error, as the derivation shows. Monte Carlo simulation is more flexible when the design or analysis is too complex for practical enumeration, but its estimates have simulation uncertainty. Both approaches depend on correctly specifying the scenario, trial design, and decision rule. Neither can make an unrealistic response probability or an inappropriate model clinically informative.

Common mistake

A frequent error is to compare the average simulated treatment response proportion with the input treatment probability and conclude that the simulation is wrong because the numbers differ. The input is the probability governing each outcome; the observed proportion in a finite simulated trial is random. For example, with five participants and response probability 0.60, the expected count is three, but counts other than three occur with positive probability. The correction is to check calibration across many generated outcomes, and separately assess how variable the trial-level decision rate is across simulation replicates.

Key takeaway: Specify the binary-outcome scenario, generate complete trial datasets under its assumptions, apply one prespecified decision rule per dataset, and estimate the proportion of simulated trials meeting that rule. Keep Monte Carlo uncertainty separate from uncertainty within an individual trial.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions below.

In a scenario with 40 participants per arm, \(p_C=0.25\), and \(p_T=0.40\), what are the expected response counts in the two arms?
A simulation has 800 independent trial replicates, of which 360 meet the success rule. Calculate the estimated decision rate and its estimated Monte Carlo standard error.
Why does a simulated response proportion of 0.80 in an arm with specified response probability 0.60 not, by itself, indicate a generator error?
For a small design, what is the advantage of exact enumeration over Monte Carlo estimation?

Solutions. (1) The expected counts are \(40(0.25)=10\) in control and \(40(0.40)=16\) in treatment. (2) The estimated rate is \(360/800=0.450\). Its estimated standard error is \(\sqrt{0.450(0.550)/800}=\sqrt{0.000309375}\approx0.01759\). (3) A finite sample proportion varies randomly around the specified probability; one trial need not reproduce that probability. Check the generator across many outcomes and inspect its coding. (4) Exact enumeration sums the probabilities of all count pairs that meet the rule, avoiding Monte Carlo error, although it may be impractical for large or complex designs.