Subgroup-Specific Operating Characteristics

Why Examine Operating Characteristics by Subgroup?

An overall operating characteristic can look acceptable while the analysis performs poorly for a prespecified group of patients. For example, a decision rule might have a reasonable overall false-positive rate but a substantially higher rate in a smaller subgroup. Likewise, an estimator may be nearly unbiased overall while tending to underestimate the target in one subgroup and overestimate it in another. Reporting subgroup-specific performance can expose these patterns.

The key is to evaluate performance conditional on subgroup membership, using a clear subgroup definition, a specified target, and the same simulation and analysis procedures that define the trial’s operating characteristics. These calculations assess how a procedure behaves within groups. By themselves, differences between subgroup summaries do not establish that treatment effects differ between groups.

Set Up the Subgroup Evaluation

Let \(r=1,\ldots,R\) index simulated trial replicates, and let \(g\) index a subgroup defined before examining trial outcomes. For each replicate, generate trial data under a specified scenario, apply the planned analysis, and record the subgroup-specific output. The subgroup definition, data-generating assumptions, prior, analysis model, decision rule, interval construction, and target estimand should all be fixed before summarizing results.

For a binary decision, let \(D_{rg}=1\) if replicate \(r\) meets the prespecified decision criterion for subgroup \(g\), and \(D_{rg}=0\) otherwise. The estimated decision rate is the fraction of simulated replicates that meet that criterion. If a favorable decision is being assessed when the subgroup target is truly null, this rate estimates the subgroup-specific false-positive rate under that scenario. It is not automatically a frequentist significance level: its value depends on the data-generating scenario, the prior, the analysis, and the decision rule.

The Monte Carlo standard error (MCSE) describes simulation noise in the estimated rate, assuming independent replicates generated under the same scenario. It is not a standard error for patient-level outcomes, nor does it account for uncertainty in the assumed data-generating scenario. If simulation replicates are dependent, the independent-replicate formula is not appropriate.

For an estimator \(\hat{\theta}_{rg}\) of a subgroup target \(\theta_g\), the estimated bias is the average estimation error across replicates. A positive value indicates average overestimation relative to the target; a negative value indicates average underestimation. If \([L_{rg},U_{rg}]\) is a reported interval for \(\theta_g\), estimated coverage is the fraction of replicates in which the interval contains the target. These are repeated-sampling properties, even when the intervals are Bayesian credible intervals.

Here \(I(\cdot)\) is 1 when its condition is true and 0 otherwise. The target \(\theta_g\) must be the estimand that the subgroup estimator and interval are meant to address. A coverage calculation against the wrong target can be numerically exact but scientifically meaningless.

How to Calculate and Interpret the Summaries

The rate formula follows directly from treating \(D_{rg}\) as a Bernoulli outcome across simulation replicates. If its probability is \(\pi_g\), the sample mean of \(R\) independent indicators has variance \(\pi_g(1-\pi_g)/R\). Replacing the unknown \(\pi_g\) with its estimate gives the displayed MCSE. The same reasoning applies to coverage indicators, with the estimated coverage replacing \(\pi_g\). Thus, an observed rate close to a target can still be imprecise when \(R\) is small.

For estimated bias, a simulation-based MCSE can be computed as the sample standard deviation of the replicate estimation errors divided by \(\sqrt{R}\). It describes how precisely the simulation estimates the mean error; it does not turn a particular estimated bias into proof that the analysis is biased in practice. The conclusion is conditional on the chosen data-generating assumptions.

Diagnostics, Comparisons, and Limitations

A subgroup summary is useful only if the simulation supplies enough information to estimate it. Inspect subgroup sample sizes and event counts across replicates, not just the number of simulated trials. A nominally large \(R\) does not prevent unstable subgroup estimates if the subgroup is rare, outcomes are sparse, or many replicates have little or no usable subgroup information. Check for failed model fits, undefined estimates, and intervals that cannot be computed. State how such replicates were handled; silently dropping them can distort the reported rates, bias, and coverage.

Subgroup results should be compared on the same definitions and under the same scenario. A decision rate calculated using a subgroup-specific posterior threshold is not directly comparable with one calculated from a single overall trial decision unless the decision events are explicitly defined to be comparable. If a trial has only one overall go/no-go decision, that event is shared by all patients in that trial; assigning it separately to each subgroup does not create distinct subgroup-specific decision rates. To estimate subgroup-specific rates, define a subgroup-specific analysis output or decision rule and apply it consistently in every replicate.

A pooled summary can conceal differences, but not every overall operating characteristic is a simple weighted average of subgroup-specific quantities. For example, an overall estimator’s bias depends on how its target and estimator combine subgroup information. An overall trial-level decision rate is the probability of one defined trial-level event, not necessarily a prevalence-weighted average of separate subgroup decisions. State the overall estimand and decision event before making such a comparison.

A useful alternative is to report subgroup-specific results alongside the overall operating characteristics and, when relevant, a worst-case summary. The earlier material on Averaging Operating Characteristics Across Scenarios concerns averaging across scenarios; subgroup stratification instead conditions summaries on patient group within a scenario. Neither replaces the other. The earlier material on Worst-Case Operating Characteristics in Bayesian Trials provides a different summary across a prespecified set of conditions; a worst-case value can complement subgroup results but can hide which subgroup drives it.

Subgroup-specific operating characteristics also have limits. They are conditional on the subgroup definitions, trial design, analysis model, prior, scenario grid, and outcome assumptions used in the simulations. Narrow subgroups can yield unstable performance, while examining many subgroups increases the chance of highlighting random-looking differences. Prespecification and transparent reporting help distinguish planned diagnostics from exploratory analyses. Most importantly, a difference in subgroup-specific bias, coverage, or decision rate is a difference in operating performance for the simulated procedure; it is not, on its own, evidence that the clinical treatment effect differs across subgroups.

Common Pitfalls

• Calling a posterior threshold a false-positive guarantee. A posterior probability threshold of 0.95 does not ensure a 5% false-positive rate. Estimate that rate by simulating under a null scenario and applying the complete decision rule.
• Confusing credible probability with coverage. A 95% credible interval does not necessarily contain the fixed target in 95% of repeated trials. Calculate coverage across simulated replicates.
• Comparing unstable subgroup estimates as if they were precise. Report subgroup sample sizes, replicate counts, and MCSEs, and inspect missing or failed fits.
• Interpreting subgroup performance differences as treatment-effect differences. Bias, coverage, or decision-rate differences diagnose procedure performance under specified assumptions; they do not establish effect modification.
• Assuming an overall rate is the weighted average of subgroup rates. That identity depends on how the overall event and weights are defined. Do not apply it automatically to trial-level decisions.