Why Evaluate Performance Within Subgroups?
A design can perform well on average and still perform poorly for a clinically important group. For example, a sampling schedule might estimate clearance precisely for most patients but provide weak information for patients with low renal function. If a simulation reports only one overall bias or RMSE, that weakness can be obscured by the larger, better-performing group.
Subgroup evaluation makes this possibility visible by calculating performance measures separately for groups defined in advance, such as renal-function categories, age bands, or concomitant-medication status. The purpose is not to search many divisions until one looks unfavorable. It is to assess, under the simulated scenarios, whether the candidate design provides adequate information for the groups that matter to the intended use.
Define the Subgroup and Its Target First
Let \(d\) denote a candidate design, \(g\) a prespecified subgroup, and \(r=1,\ldots,R\) the simulation replicate. For a scalar target parameter, write \(\theta_g\) for the true target in subgroup \(g\) and \(\hat{\theta}_{dgr}\) for its estimate under design \(d\) in replicate \(r\). If the scientific target is common across subgroups, the same \(\theta\) can be used for each \(g\); if the target varies by subgroup, define the appropriate \(\theta_g\) explicitly.
This distinction matters. An estimate of a common population clearance parameter and an estimate of clearance for a particular subgroup are not automatically the same target. A subgroup-specific bias is meaningful only relative to a clearly specified subgroup-specific target. The subgroup definition, target, performance metric, and handling of fitting failures should therefore be established before summarizing results.
For example, a subgroup might be patients whose simulated baseline eGFR is below a prespecified threshold. Each replicate should retain the subgroup membership produced by its simulated covariates. If membership can vary across replicates, report that variation: a group containing very few patients in some replicates may be difficult to estimate even if its average size appears adequate.
Subgroup Performance Measures
For replicates in which the estimation procedure succeeds, define the estimation error as \(e_{dgr}=\hat{\theta}_{dgr}-\theta_g\). The subgroup bias is the average error, while subgroup RMSE reflects both systematic error and replicate-to-replicate variability:
Here, \(\mathcal{S}_{dg}\) is the set of replicates in which the fit succeeds for the subgroup-specific analysis, and \(R_{dg}^{\mathrm{fit}}\) is the number of those replicates. Bias retains the units of the parameter; RMSE has those same units and is nonnegative. If the parameter is positive and a relative measure is useful, relative bias can be reported as \(\widehat{\operatorname{Bias}}_{dg}/\theta_g\), usually expressed as a percentage. State the denominator and units or percentage convention.
For a nominal \(100(1-\alpha)\%\) confidence interval, subgroup coverage is the proportion of eligible successful replicates whose interval contains the target. The fitting-failure rate should be reported separately as the proportion of attempted replicates that fail under the stated failure definition. Coverage calculated only among successful fits answers a conditional question; it does not count failed fits as covered.
Why an Overall Summary Can Conceal a Weak Group
A useful check is to see how a combined statistic is constructed. If subgroup errors are pooled with equal replicate counts, pooled RMSE is the square root of the average squared error—not the average of subgroup RMSEs. With unequal numbers of eligible observations or replicates, the relevant weights also differ. In contrast, an average absolute error can be calculated directly by pooling absolute errors. These summaries can tell different stories, so one overall number cannot replace subgroup results.
For equal numbers of subgroup-replicate results, the pooled RMSE follows directly from the definition:
where each subgroup contributes the same number of errors. The square appears because RMSE is based on squared errors. If subgroup counts differ, each subgroup's mean squared error must instead be weighted by its number of included errors divided by the total number of errors. This calculation explains why a moderately sized group can contribute little to a headline summary while still having unacceptable performance for its intended use.
Worked Example: Equal Overall Absolute Error, Unequal Subgroup Performance
Suppose two hypothetical designs estimate a parameter in two equally represented subgroups. There are four successful simulation replicates in each subgroup. Design A has errors of \(-0.10,\ 0.10,\ -0.10,\ 0.10\) units in subgroup 1 and \(-0.80,\ 0.80,\ -0.80,\ 0.80\) units in subgroup 2. Design B has errors of \(-0.45,\ 0.45,\ -0.45,\ 0.45\) units in each subgroup.
For Design A, subgroup 1 has zero mean error and RMSE \( \sqrt{(0.10^2+0.10^2+0.10^2+0.10^2)/4}=0.10 \) units. Subgroup 2 also has zero mean error, but its RMSE is \( \sqrt{(0.80^2+0.80^2+0.80^2+0.80^2)/4}=0.80 \) units. For Design B, each subgroup has zero mean error and RMSE \( \sqrt{(0.45^2+0.45^2+0.45^2+0.45^2)/4}=0.45 \) units.
The pooled mean absolute error for Design A is \( (4(0.10)+4(0.80))/8=0.45 \) units. For Design B it is \( (8(0.45))/8=0.45 \) units. Thus this overall summary ties the designs, although Design A has substantially worse RMSE in subgroup 2. The pooled RMSE does distinguish them: for A it is \( \sqrt{(0.10^2+0.80^2)/2}=\sqrt{0.325}=0.570 \) units; for B it is \( \sqrt{(0.45^2+0.45^2)/2}=0.450 \) units. The example shows why reporting subgroup measures alongside a combined summary matters.
Justification for Subgroup RMSE
RMSE has a direct decomposition that helps explain what it adds beyond bias. For a fixed design and subgroup, let \(e\) denote the estimation error over successful simulation replicates. Its mean is the simulation bias, and its variance is the average squared deviation from that mean, using the probability-distribution definition. Since \(e=(e-E[e])+E[e]\), expanding the square and taking expectations gives:
The cross-product has expectation zero because \(E[e-E[e]]=0\). Taking square roots yields the population relationship \( \operatorname{RMSE}=\sqrt{\operatorname{Var}(e)+\operatorname{Bias}(e)^2} \). Thus a design can have little average bias but still have large RMSE because its estimates vary widely across replicates. Conversely, a stable but systematically biased estimator can also have elevated RMSE. This identity concerns the simulation distribution of errors; the estimated quantities use the corresponding replicate averages.
Worked Example: Subgroup Coverage and Monte Carlo Uncertainty
In a hypothetical simulation with 500 replicates, a candidate design produces successful fits in two prespecified groups. In the common group, 470 of 500 confidence intervals contain the subgroup target; in a smaller clinically important group, 400 of 500 contain its target. Assume each replicate was generated independently and the interval level was nominally 95%.
The estimated coverages are \(470/500=0.940\), or 94.0%, in the common group and \(400/500=0.800\), or 80.0%, in the smaller group. The latter is a substantial shortfall from the nominal level in these simulated conditions. It does not establish why coverage is low; subgroup-specific bias, uncertainty estimation, or other features of the analysis may contribute and should be investigated.
For an estimated coverage proportion \(\hat{c}\) based on \(R\) independent replicates, its Monte Carlo standard error is estimated by \( \sqrt{\hat{c}(1-\hat{c})/R} \). The common-group value is \( \sqrt{0.940(0.060)/500}=0.0106 \), or about 1.06 percentage points. The smaller-group value is \( \sqrt{0.800(0.200)/500}=0.0179 \), or about 1.79 percentage points. Monte Carlo uncertainty is larger for the smaller estimated coverage here, but it does not erase the observed 14-percentage-point difference. These standard errors quantify simulation noise in the estimated coverages, not uncertainty about whether the scientific target is correct.
If, purely for illustration, an overall coverage is defined using weights of 0.80 for the common group and 0.20 for the smaller group, it is \(0.80(0.940)+0.20(0.800)=0.912\), or 91.2%. That combined result is not the unweighted average and does not change the smaller group's 80.0% coverage. The weights must reflect the intended summary; they should not be assumed to define subgroup adequacy.
Failures, Sparse Groups, and Conditional Results
A metric computed only on successful fits can paint an incomplete picture if failures are concentrated in a subgroup. For each design and subgroup, show how many replicates were attempted, how many succeeded, and how many contributed to each conditional estimation metric. Explain the failure rule consistently. Do not silently omit failed fits from the report, and do not treat an undefined estimate as a zero-error estimate.
Worked Example: Low Subgroup Fit Rate and Conditional RMSE
Consider a hypothetical clearance study with a true subgroup target of 5 L/h for patients with low eGFR and 7 L/h for patients with higher eGFR. In 200 attempted replicates, the low-eGFR analysis has 176 successful fits and the higher-eGFR analysis has 198. Among successful fits, suppose the low-eGFR errors have sum 52.8 L/h and sum of squared errors 212.96 \((\mathrm{L/h})^2\); the corresponding higher-eGFR sums are 9.90 L/h and 31.68 \((\mathrm{L/h})^2\).
The low-eGFR failure rate is \( (200-176)/200=24/200=0.120 \), or 12.0%. Its conditional bias is \(52.8/176=0.300\) L/h, and its conditional RMSE is \( \sqrt{212.96/176}=\sqrt{1.210}=1.10 \) L/h. Relative bias is \(0.300/5=0.0600\), or 6.00%.
The higher-eGFR failure rate is \( (200-198)/200=2/200=0.0100 \), or 1.00%. Its conditional bias is \(9.90/198=0.0500\) L/h, and its conditional RMSE is \( \sqrt{31.68/198}=\sqrt{0.160}=0.400 \) L/h. Relative bias is \(0.0500/7=0.00714\), or 0.714%.
The estimation summaries are conditional on successful fits, and the failure rates are reported separately. Together they indicate poorer performance in the low-eGFR group under these simulated conditions. They do not describe the hypothetical results of failed fits as if those fits had produced valid estimates.
Diagnostics and Appropriate Comparisons
Subgroup summaries should be read together with the distribution of results across replicates. A mean bias near zero can arise when large positive and negative errors cancel; inspect the replicate-level errors or performance distributions as well. Check whether subgroup counts are consistently adequate, whether failures cluster in a particular group, and whether an apparent difference is large relative to Monte Carlo uncertainty. When design results are paired across replicates, uncertainty in their difference should be assessed using the paired comparison, rather than treating the two sets of results as independent.
There are several distinct ways to examine groups, and they answer different questions:
| Approach | What it answers | Important caution |
|---|---|---|
| Separate subgroup summaries | How does each design perform for each prespecified group? | Small groups may yield noisy summaries or frequent failures. |
| One combined summary | How does performance look under a specified aggregation? | A favorable average can mask poor performance in one group. |
| Subgroup-by-design contrast | Does the difference between candidate designs vary across groups? | Assess uncertainty in the contrast and keep the comparison paired when applicable. |
A formal model containing subgroup interactions may be useful for some scientific questions, but it is not a substitute for clearly reporting the design’s subgroup-specific simulation performance. Likewise, a subgroup difference in simulated performance is not by itself evidence of a causal effect of the subgroup characteristic.
Common Mistakes and Limitations
- Choosing groups after seeing results. This encourages selective reporting and exaggerates the apparent importance of unusual findings. Define clinically or scientifically motivated groups in advance.
- Reporting only an overall average. This can conceal poor precision, bias, coverage, or fit rates in a smaller group. Show subgroup-specific results alongside any combined measure.
- Using the wrong target. Compare each estimate with the target it is intended to estimate in that subgroup, and state that target explicitly.
- Ignoring failures. Conditional bias and RMSE do not capture how often the analysis fails. Report attempted and successful replicate counts and the failure rate.
- Overinterpreting small differences. Subgroup estimates have Monte Carlo uncertainty, and sparse groups may be especially noisy. A difference in point summaries alone does not establish that one design is reliably better.
Subgroup results are conditional on the covariate distributions, subgroup definitions, parameter settings, and analysis procedure used in the simulation. They may not generalize to patient groups or conditions that were not represented. Subgrouping continuous covariates also trades interpretability against information: narrow groups may reveal local weakness but contain too few observations for stable estimation. Report subgroup sizes and avoid interpreting arbitrary cut points as biological boundaries.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A subgroup has replicate estimation errors \(-0.2,\ 0.2,\ -0.2,\ 0.2\) mg/L. Calculate the bias and RMSE.
- A design has 360 covered intervals among 400 successful subgroup fits and 20 failures among 420 attempted fits. Calculate conditional coverage and the failure rate.
- Why can a subgroup RMSE not be replaced by its subgroup bias?
- What should be reported if a subgroup’s estimates are summarized only among successful fits?
Solutions. (1) The errors sum to zero, so bias is \(0/4=0\) mg/L. RMSE is \( \sqrt{(4(0.2^2))/4}=0.200 \) mg/L. (2) Conditional coverage is \(360/400=0.900\), or 90.0%; the failure rate is \(20/420=0.0476\), or 4.76%. The denominators differ because coverage is calculated among successful fits while failures are calculated among attempts. (3) Bias measures the signed average error, so positive and negative errors can cancel; RMSE also reflects error variability and does not cancel errors by sign. (4) Report the attempted and successful counts, the failure rate, and that the estimation summaries are conditional on successful fits.