Continuous Outcomes In Operating-Characteristic Simulations

Why Simulate Continuous Outcomes?

A trial’s operating characteristics depend on both its decision rule and the outcomes that the trial could produce. For a continuous endpoint, the simulated data must reflect the assumed treatment and control means, outcome variances, sample sizes, and treatment effect. Repeating the trial many times under a specified scenario shows how often the decision rule would declare benefit and how the chosen estimator would perform.

The aim is not to simulate one plausible data set and treat it as representative. It is to estimate repeated-trial quantities under stated assumptions. A decision rate is a probability over hypothetical repetitions of the entire trial; an estimate from a finite simulation has Monte Carlo error.

Specify the Data-Generating and Analysis Models

Consider a two-arm parallel trial with \(n_T\) participants assigned to treatment and \(n_C\) assigned to control. Let \(Y_{Ti}\) and \(Y_{Cj}\) be the continuous outcomes. For a basic simulation, suppose outcomes are independent within and between arms and normally distributed:

Here \(\delta\) is the treatment-minus-control effect in the outcome’s units; positive values indicate a higher mean in the treatment arm and favor treatment only when higher outcomes are beneficial. For outcomes where lower values are better, reverse the sign convention or define benefit using \(\delta<0\). The normal distributions and independence assumption must be reasonable for the intended setting. The simple setup also assumes complete outcome measurement, fixed sample sizes, and no additional clustering or covariate adjustment. Those conditions are design choices, not consequences of simulation.

A convenient replicate-level summary is the difference in sample means, \(D=\bar{Y}_T-\bar{Y}_C\). Under the model above,

Thus one can simulate all individual outcomes and calculate \(D\), or draw \(D\) directly from this distribution when the analysis depends only on the difference in means and the model assumptions apply. Direct simulation is faster; simulating individual outcomes also allows checks of data-generation code and analyses that use more than the arm means.

To illustrate a Bayesian decision rule, suppose the analysis uses the sampling model \(D\mid\delta\sim N(\delta,V)\) and prior \(\delta\sim N(m_0,\tau^2)\), with \(V\) treated as known. A common rule declares benefit when \(P(\delta>0\mid D)>\gamma\), for a prespecified cutoff \(\gamma\). This analysis model and prior must be set before examining simulation results. The focus here is how to evaluate the rule under repeated continuous-outcome trials, not how to choose a prior or cutoff.

From a Replicate to a Decision

The following derivation gives a useful check on both the analysis and the simulation. With \(m_0=0\), the posterior mean is \(m=kD\), where \(k=\tau^2/(\tau^2+V)\), and \(v=Vk\). Since \(k>0\), the posterior-probability rule is equivalent to a threshold on the observed difference:

where \(z_\gamma=\Phi^{-1}(\gamma)\). The last step uses the fact that \(\Phi\) is increasing. This threshold is a property of the stated model, prior, and decision rule. For any data-generating value \(\delta\), the decision rate can then be checked using \(D\sim N(\delta,V)\). In more complicated simulations, the same decision rate is estimated by applying the rule to each simulated replicate.

Estimate Decision Rates Across Replicates

For each scenario, generate \(M\) independent trial replicates. In replicate \(r\), apply the same analysis and rule, and let \(I_r=1\) if the rule declares benefit and \(I_r=0\) otherwise. The estimated decision rate is \(\hat{q}=M^{-1}\sum_{r=1}^{M}I_r\). If the replicate decisions are independent Bernoulli variables with common probability \(q\), the Monte Carlo standard error is approximately \(\sqrt{\hat{q}(1-\hat{q})/M}\). This measures simulation noise, not uncertainty about the clinical effect or the appropriateness of the assumed model.

Evaluate Estimator Performance

Decision rates do not describe how accurately an effect is estimated. For each replicate, record the prespecified estimator \(\hat{\delta}_r\), such as the observed difference \(D_r\) or a posterior summary. Across replicates generated at true effect \(\delta\), empirical bias is the mean of \(\hat{\delta}_r-\delta\); empirical variance describes the spread of the estimates around their replicate mean. Root mean squared error (RMSE) is the square root of the average squared error. It combines variability and bias:

For an unbiased estimator, its theoretical MSE equals its variance. More generally, mean squared error equals variance plus squared bias when variance and bias are defined for the estimator’s sampling distribution. Report the estimator and units: an RMSE of 2 outcome units is not a decision rate and does not mean every trial estimate is within 2 units of the truth.

Diagnostics, Alternatives, and Limitations

A sound simulation begins with checks that the generator produces the intended distributions. Across generated replicates, check that the arm means are close to their specified values, the spread of arm outcomes is consistent with the specified variances, and the distribution of \(D\) is consistent with mean \(\delta\) and variance \(V\). Check the treatment-minus-control sign convention with a simple trial where the treatment mean is larger. These checks can reveal swapped arms, use of a standard deviation where a variance is required, or incorrect sample-size divisors.

Compare estimated decision rates with Monte Carlo standard errors and, when available, analytic probabilities. Increase \(M\) if the simulation error is too large for the precision needed; do not treat a small Monte Carlo standard error as evidence that the data-generating assumptions are realistic. Also inspect code for differences between the simulated data model and the analysis model. A mismatch may be deliberate, but then the operating characteristics describe that mismatch and should be reported as such.

When a normal sampling distribution and a simple decision threshold are available, an analytic probability calculation is a useful alternative or cross-check. Simulation is more flexible when the replicate-level analysis is complex, but adds Monte Carlo uncertainty and implementation risk. In either approach, specify the scenario and decision rule; results for one effect and variance combination do not automatically apply to another. A normal, independent, complete-data model may be inadequate for skewed outcomes, outliers, clustering, or other departures. The simulation only answers the question encoded in its assumptions.

• Wrong: “A simulated decision rate of 0.469 is close to a theoretical rate of 0.0467 because the Monte Carlo standard error is about 0.002.” Correction: With \(M=10{,}000\), a rate of 0.469 has standard error about 0.00499 and is not close to 0.0467. A rate of 0.0469 has standard error about 0.002114.
• Wrong: “The posterior probability of benefit in one replicate is the trial’s power.” Correction: It is a probability conditional on that replicate’s observed data and the Bayesian model. Power or a corresponding decision rate is the probability of the rule declaring benefit over repeated trials under a specified true effect.
• Wrong: “A small RMSE means the estimator has no bias.” Correction: RMSE incorporates both bias and variability; assess them separately as well as together.