Evaluating Subgroup Predictions Without Recalibration

Why Evaluate Predictions Within Subgroups?

A PBPK model can appear adequate overall while systematically overpredicting for one group and underpredicting for another. This can matter when the model will be used for decisions in a defined population, such as people with reduced renal function. An overall summary combines information across participants; it does not by itself show whether predictions are appropriate within clinically relevant strata.

Subgroup evaluation asks whether predictions made by the already calibrated model agree sufficiently with held-out observations in each prespecified subgroup. The defining constraint is that the validation outcomes are used to assess the predictions, not to alter the parameter set that generated them. A subgroup discrepancy may motivate investigation, but changing subgroup-specific parameters to remove that discrepancy is recalibration, not independent validation.

Define the Evaluation Before Looking at Outcomes

A useful subgroup analysis begins with a clearly defined target. For example: “Compare the model-predicted and observed plasma concentration at 8 hours after the first dose among participants whose baseline estimated glomerular filtration rate is below 60 mL/min/1.73 m².” This specifies the population rule, endpoint, and sampling condition. If the scientific question concerns a different endpoint or time, it should be stated separately rather than selected after inspecting which comparison looks most favorable.

The model may use available participant characteristics, such as body weight or renal function, to produce individualized predictions. That is not refitting: the prediction is still generated by the frozen model and its prespecified covariate relationships. Refitting occurs when validation outcomes are used to estimate or alter parameters, covariate effects, or other prediction rules. The distinction is about how the prediction is generated, not whether predictions differ across participants.

For a simple evaluation, let \(O_i\) be the observed endpoint and \(P_i\) the corresponding prediction for participant \(i\). Define the signed discrepancy as \(D_i=O_i-P_i\). A positive value means the observation exceeds the prediction; a negative value means the prediction exceeds the observation. For a subgroup \(g\) with \(n_g\) independent participants, the mean discrepancy is \(\bar{D}_g=\sum_{i\in g}D_i/n_g\). Its units are the same as the endpoint’s units.

Under these assumptions, the sample standard deviation \(s_D\) describes variation among participant-level discrepancies, and \(SE=s_D/\sqrt{n_g}\) estimates the standard error of their mean. A two-sided \(100(1-\alpha)\%\) t confidence interval for the subgroup mean discrepancy is \(\bar{D}_g\pm t_{1-\alpha/2,n_g-1}SE\). For small samples, this interval relies strongly on the distributional assumption; it does not replace inspection of the individual data or a careful account of study design.

Worked Subgroup Comparisons

Justification: Why Pooling Can Conceal a Subgroup Pattern

The overall mean discrepancy is a weighted average of subgroup mean discrepancies. If subgroup \(g\) contains \(n_g\) participants and has mean discrepancy \(\bar{D}_g\), then summing all participant discrepancies gives \(\sum_g\sum_{i\in g}D_i=\sum_g n_g\bar{D}_g\). Dividing by the total sample size \(N=\sum_g n_g\) yields:

This is a weighted average, not a guarantee that each subgroup mean is close to zero. If two equally sized groups have discrepancies \(+2\) and \(-2\) ng/mL, their weighted average is zero. With unequal group sizes, the larger group contributes more to the overall result, so a substantial discrepancy in a smaller subgroup may have little effect on the pooled mean. Reporting subgroup results alongside an overall summary is therefore necessary when subgroup performance is part of the prediction question.

Diagnostics, Alternatives, and Limitations

A subgroup analysis can fail to answer its intended question even when its arithmetic is correct. Check that the subgroup assignment uses the prespecified variable and threshold, that every comparison uses matched prediction-observation pairs, and that the model version and parameters are the frozen ones. Confirm that the subgroup has enough independent participants to support the intended precision. If several measurements come from the same participant, a participant-level summary or a method that accounts for within-participant dependence is needed; counting those measurements as independent makes uncertainty appear too small.

Prespecification matters because trying many cut points or subgroup definitions and emphasizing only the most favorable result makes the reported subgroup look more convincing than the evidence warrants. Small strata also produce unstable estimates and wide intervals. A nonzero estimated discrepancy may reflect sampling variability, study conduct, measurement error, or model limitations; subgroup comparison alone does not identify its cause. Conversely, an interval that includes zero does not prove that predictions are adequate, particularly if it is wide or if the relevant difference is clinically important but not detectable with the available sample.

The main alternative is an overall evaluation without subgroup summaries. It is simpler and can answer an overall target-population question, but may mask opposing subgroup patterns. Recalibrating parameters within each subgroup is a different analysis: it may describe how well an adjusted model can fit those outcomes, but it does not evaluate the original frozen model independently on those same outcomes. A subgroup-specific model can be assessed as a new prediction model only with data not used to make or select that adjustment. A pooled model that already includes a prespecified covariate relationship is also distinct from subgroup recalibration: it can generate different predictions by covariate value while keeping its calibrated parameter set fixed.

No single subgroup mean discrepancy establishes that the PBPK model is generally reliable in that subgroup. The conclusion applies to the defined validation population, endpoint, condition, and data source. Sparse observations, restricted covariate ranges, protocol deviations, and uncertainty in measurements can limit generalization. A clinically meaningful acceptance rule should be specified for the question at hand; the examples above illustrate estimation and interpretation, not a universal pass-or-fail threshold.