Subgroup Performance in External Validation

Why Subgroup Performance Needs Separate Attention

An external model can perform acceptably on average while making systematically different predictions for clinically important groups. For example, prediction errors could be centered near zero in one renal-function category and consistently positive in another. A pooled summary can hide that pattern, particularly when the groups have very different sample sizes. The reverse problem also occurs: a noisy estimate in a small group can look different by chance even when the underlying performance is similar.

Subgroup evaluation therefore asks two questions at once: how do prediction errors differ across groups, and how certain are we about those differences? The distinction matters in population pharmacokinetics (PK), where repeated observations from one participant are not independent observations, and where an external validation sample may include unequal numbers of participants from clinically relevant categories.

The outcome and prediction must refer to the same quantity, unit, time point, and prediction target. For a concentration outcome, for example, errors might be reported in mg/L. If the evaluation uses log-scale errors or a different error definition, that choice should be stated: bias and precision depend on the scale. Population predictions and conditional predictions answer different questions, so a subgroup comparison must use the same prediction type in every subgroup.

Define clinical subgroups before examining their errors whenever possible. A subgroup should be interpretable and determined from information available for the intended evaluation, rather than selected because it displays an unusual result. The validation model remains locked: these comparisons evaluate its predictions and do not re-estimate its parameters.

Estimating Bias and Precision Within Groups

For a subgroup containing \(n_g\) independent participants, let its errors be \(e_{1g}, \ldots, e_{n_g g}\). The estimated bias is the subgroup mean error. A common summary of precision is the sample standard deviation of errors; a second useful summary is the root mean squared error (RMSE). Standard deviation describes scatter around the subgroup mean, while RMSE measures typical error relative to zero and therefore reflects both scatter and bias.

A lower \(s_g\) means less observed spread around that subgroup’s average error. A lower RMSE means smaller errors relative to perfect prediction, which requires errors to be both centered near zero and not too dispersed. Neither quantity alone establishes that differences between groups are real. Uncertainty in the estimated bias or precision is a separate issue, and is especially important when \(n_g\) is small.

Why RMSE Combines Bias and Scatter

The relationship between RMSE, bias, and precision makes clear why these summaries should not be treated as interchangeable. Let the observed subgroup errors be fixed and define their variance using divisor \(n_g\), rather than the \(n_g-1\) divisor used in the sample standard deviation. The mean squared error can be expanded by writing each error as its deviation from the sample mean plus the sample mean itself.

The cross-product term is zero because the deviations from the sample mean sum to zero. Since \(\sum_i(e_{ig}-\bar{e}_g)^2=(n_g-1)s_g^2\), taking square roots gives \(\mathrm{RMSE}_g=\sqrt{\bar{e}_g^2+((n_g-1)/n_g)s_g^2}\). The factor \((n_g-1)/n_g\) appears because the sample standard deviation uses divisor \(n_g-1\), whereas RMSE uses divisor \(n_g\). This identity applies to the observed sample summaries; it does not imply that the sample estimates equal population quantities.

Unequal Subgroup Sizes and the Overall Summary

The pooled mean error weights each participant equally, so larger subgroups contribute more. It is not generally the same as the unweighted average of subgroup means. Neither summary is inherently wrong: they answer different questions. The pooled summary describes the validation sample’s participant mix. An equally weighted subgroup summary gives each defined group the same influence. If a target population has specified subgroup proportions, a weighted summary using those proportions may be more relevant, provided the subgroup estimates are suitable for that target.

Quantifying Uncertainty and Reading Sparse Groups

A visible difference in point estimates is not sufficient evidence of a genuine subgroup difference. Estimate uncertainty for each subgroup and, preferably, for the contrast itself. For two independent groups, a simple contrast in bias is \(\bar{e}_B-\bar{e}_A\). Its standard error can be estimated using the two sample variances and sample sizes. This approach assumes independent participants across groups; if subgroup membership overlaps or observations are paired, the dependence must be reflected in the contrast’s uncertainty calculation.

For many PK validation datasets, a subject-level bootstrap is a useful way to estimate uncertainty in subgroup metrics and their contrasts without relying solely on a normal approximation for the metric. The resampling unit should be the independent participant: if a participant contributes multiple concentrations or occasions, resample that participant and retain all their records together. Resampling individual concentration records would incorrectly treat within-participant observations as independent and can understate uncertainty.

The bootstrap scheme should match the estimand. Resampling participants within each subgroup keeps subgroup sample sizes fixed and estimates uncertainty conditional on those subgroup sizes. Resampling participants from the full validation sample reflects the observed mixture and permits subgroup counts to vary across replicates. In either case, recompute the same bias, precision, or subgroup contrast in each replicate. A bootstrap interval cannot compensate for a subgroup with very few independent participants: resamples repeatedly reuse the same limited information. Inspect the bootstrap distribution for instability, skewness, or degenerate replicates, and interpret it with that limitation in view.

Comparisons, Diagnostics, and Limits

Direct subgroup summaries are transparent and show performance in the units clinicians recognize. A regression model with subgroup indicators and interactions can estimate contrasts and adjust for prespecified factors, but its interpretation depends on the model form and assumptions; it can also obscure a small subgroup’s sparse information. A pooled overall metric is simpler, but cannot answer whether one subgroup has a different bias or error spread. These approaches are complementary, not substitutes for defining the target comparison clearly.

Before interpreting an apparent difference, check whether the pattern is stable and whether the comparison is fair. Look at subgroup sample sizes, participant-level error plots, outliers, and the spread of errors—not just the means. Check that outcomes, prediction types, units, and evaluation times are comparable across groups. A subgroup’s precision estimate can be distorted by a single extreme error, especially at small \(n_g\). Consider whether differences in sampling frequency or the number of records per participant could give some participants disproportionate influence; use participant-level summaries or an analysis that explicitly respects the repeated-measures structure when appropriate.

Also distinguish subgroup-specific performance from the uncertainty of the subgroup estimate. A wide confidence interval reflects limited information about the metric; it does not itself show that predictions are imprecise for each individual. Conversely, a narrow interval for mean bias says little about the spread of individual errors. Report both point estimates and uncertainty, with units and subgroup sample sizes. If many subgroups or metrics are examined, some striking contrasts can arise by chance; prioritize clinically motivated comparisons and treat exploratory findings as such.

Subgroup comparisons describe performance in the groups and validation data actually evaluated. They cannot, by themselves, establish performance in groups with little information or justify extrapolation beyond the observed data. They are also not causal comparisons: a difference in errors does not show that subgroup membership caused the model’s performance pattern. Interpret the results as evidence about predictive behavior, with the scope limited by the validation design and available observations.