Why Summarize Diagnostics by Subgroup?
An overall prediction-error summary can conceal important differences between parts of a pharmacometric data set. For example, a model may have small errors at low concentrations but larger errors at high concentrations, or errors may differ between study occasions. A subgroup table makes those differences easier to inspect by placing observation counts and error metrics side by side for clearly defined groups.
A table is useful only when its rows are comparable. Each row needs a stated subgroup definition, a clear count of the observations used in its calculations, and metrics computed from the same kind of prediction error. A count is not decoration: it tells the reader how much data supports a row and helps distinguish a stable-looking summary from one based on very few records.
This sign convention makes positive errors predictions above the observations and negative errors predictions below them. Keep the same convention throughout a table and state it in a caption or note. If errors are instead defined as observation minus prediction, the signs of signed-error summaries reverse, although absolute-error and squared-error summaries do not.
Metrics and Denominators
Three common summaries answer different questions. The mean error describes the average signed difference and is often called prediction bias. The mean absolute error describes the typical magnitude without cancellation between positive and negative errors. The root mean squared error also measures error magnitude, but gives relatively greater influence to larger errors because errors are squared before averaging.
All three metrics use the same evaluated records and the same denominator \(n_g\). If the observations are concentrations measured in mg/L, each error metric is also in mg/L; the count is a number of observations, not a number of subjects. With repeated measurements, multiple records may come from one subject, so an observation count should not be described as a subject count.
The subgroup definition also needs to be reproducible. It may use a recorded category, such as study or occasion, or a prespecified interval of a covariate, such as a concentration range. State the category labels and, for numeric intervals, the cut points and boundary convention. Assign each eligible observation to one intended row; overlapping or ambiguous groups make counts and comparisons difficult to interpret.
Worked Example: Comparing Two Concentration Groups
Suppose a hypothetical diagnostic data set contains five observed concentrations, all in mg/L. The first three records are in a prespecified low-concentration group and the final two are in a high-concentration group. For the low group, the observed values are 10, 12, and 8, and predictions are 11, 10, and 10. For the high group, observations are 20 and 22, and predictions are 18 and 24. Use prediction minus observation as the error.
For the low group, the errors are \(11-10=1\), \(10-12=-2\), and \(10-8=2\) mg/L. Thus \(n_g=3\). The mean error is \((1-2+2)/3=0.3333\) mg/L, the MAE is \((|1|+|-2|+|2|)/3=5/3=1.667\) mg/L, and the RMSE is \(\sqrt{(1^2+(-2)^2+2^2)/3}=\sqrt{3}=1.732\) mg/L.
For the high group, the errors are \(18-20=-2\) and \(24-22=2\) mg/L, so \(n_g=2\). The mean error is \((-2+2)/2=0\) mg/L, the MAE is \((2+2)/2=2.000\) mg/L, and the RMSE is \(\sqrt{(4+4)/2}=\sqrt{4}=2.000\) mg/L.
| Concentration group | Evaluated observations | Mean error (mg/L) | MAE (mg/L) | RMSE (mg/L) |
|---|---|---|---|---|
| Low | 3 | 0.3333 | 1.667 | 1.732 |
| High | 2 | 0.000 | 2.000 | 2.000 |
The high group has a zero mean error because its two errors cancel, not because both predictions are exact. Its MAE and RMSE show that the errors are still 2 mg/L in magnitude. The small counts also matter: these rows describe the five hypothetical records, not a precise estimate of performance in every future observation.
A Reproducible Table-Building Sequence
A reliable workflow fixes the definitions before calculating summaries. This prevents a table from silently changing its meaning when a report is regenerated or when additional records are included.
State what one record represents, which prediction is being evaluated, and the error sign convention. Do not mix different prediction types within a metric column unless the table explicitly separates them.
Name the grouping variable and its categories or interval cut points. Record the rule for boundary values so every assignment can be reproduced.
For each group, count records that contribute to the metrics. If group membership includes records that cannot contribute to a metric, do not present that larger number as the metric denominator.
Use the same formulas and units for every row. Include the evaluated observation count and enough precision to make the summaries interpretable without implying unwarranted accuracy.
As a basic check, recompute at least one row directly from its errors and verify that the count agrees with the number of terms in each sum. For every subgroup, MAE and RMSE must be nonnegative. The absolute value of mean error cannot exceed MAE, because the magnitude of a sum cannot exceed the sum of magnitudes. These checks can reveal coding mistakes or inconsistent denominators.
Worked Example: Reporting a Group with an Unusable Prediction
In a hypothetical study, four records belong to the morning-occasion subgroup. Three have both an observation and a usable prediction, producing errors of 2, -1, and 3 mg/L. The fourth record has an observation but no usable prediction for the diagnostic calculation. Report the evaluated count and metrics for the three usable errors.
The subgroup contains four records, but only three contribute to the metrics, so the evaluated observation count is \(n_g=3\). The mean error is \((2-1+3)/3=4/3=1.333\) mg/L. The MAE is \((2+1+3)/3=2.000\) mg/L. The RMSE is \(\sqrt{(2^2+(-1)^2+3^2)/3}=\sqrt{14/3}=2.160\) mg/L.
The table’s count for these calculations should therefore be 3, not 4. If the report also displays the number of records assigned to the subgroup before checking whether a prediction is usable, label that quantity separately. Keeping the two counts distinct prevents a reader from assuming that all four records contributed to the error summaries.
Why Pooled Results Are Not Simple Averages of Rows
Subgroup summaries describe each group separately. If an overall result is also reported, calculate it from all eligible observations or combine subgroup summaries using the appropriate subgroup counts. Simply averaging the subgroup metrics gives each group equal weight, regardless of how many observations it contains.
For example, the pooled mean error is the count-weighted mean of subgroup mean errors. Pooled MAE is likewise the count-weighted mean of subgroup MAEs. For RMSE, it is the squared subgroup RMSEs that are averaged with count weights; take the square root only after that averaging. These identities follow directly by separating each sum across disjoint subgroups.
To justify the RMSE identity, within group \(g\), the sum of squared errors is \(n_g\text{RMSE}_g^2\), by the definition of RMSE. Summing these squared-error totals over the groups gives the total squared error across all records. Dividing by the total count and taking the square root gives the pooled RMSE. The same partition-of-sums argument gives the mean-error and MAE identities.
Worked Example: Pooled Error Versus Averaging Subgroup Rows
Consider two hypothetical subgroups with errors in mg/L. Group A has two errors, 1 and 3, so its mean error and MAE are both 2 mg/L, and its RMSE is \(\sqrt{(1+9)/2}=\sqrt{5}=2.236\) mg/L. Group B has four errors, all equal to -1, so its mean error is -1 mg/L, its MAE is 1 mg/L, and its RMSE is 1 mg/L.
Across all six observations, the sum of signed errors is \(1+3-1-1-1-1=0\), so pooled mean error is \(0/6=0\) mg/L. Pooled MAE is \((1+3+1+1+1+1)/6=8/6=1.333\) mg/L. The pooled RMSE is \(\sqrt{(1+9+1+1+1+1)/6}=\sqrt{14/6}=1.528\) mg/L.
An unweighted average of the subgroup mean errors would be \((2+(-1))/2=0.500\) mg/L, not the pooled mean error. An unweighted average of subgroup MAEs would be \((2+1)/2=1.500\) mg/L, not the pooled MAE. Averaging the subgroup RMSEs would give \((2.236+1)/2=1.618\) mg/L, also not the pooled RMSE. The larger group contributes four of the six observations, so the pooled result must reflect that imbalance.
Interpreting Comparisons and Recognizing Problems
A subgroup table is descriptive. A difference in observed metrics does not by itself establish that subgroup membership caused the difference, nor does it demonstrate that a model will perform similarly on future data. In pharmacometric data, records from the same subject may be related. Consequently, a subgroup observation count does not indicate how many independent subjects support the pattern; uncertainty assessments must respect the data structure.
Common alternatives answer different questions. Mean error is useful for direction and average bias, but opposite-signed errors can cancel. MAE is easier to interpret as an average error magnitude and is less dominated by a large error than RMSE. RMSE emphasizes larger errors, which may be useful when large discrepancies are especially consequential, but it can be sensitive to outlying records. A median absolute error can be added when a robust summary is desired, provided its definition and denominator are stated. None of these summaries replaces inspection of diagnostic plots, which can reveal patterns that a single number hides.
- Very small groups: A metric based on only one or two observations can change substantially when a record changes. Show the count, avoid ranking such rows as if they were precise, and consider whether the grouping is too fine for useful comparison.
- Unequal or changing denominators: If metrics in a row use different records, comparisons across columns may be misleading. State the evaluated count for each metric when denominators differ.
- Unclear subgroup boundaries: Overlapping bins or undocumented cut points can duplicate or misclassify records. Specify intervals and boundary rules, then check that assignments match them.
- Misreading a zero mean error: Zero can result from cancellation. Inspect MAE or RMSE alongside signed error before describing a group’s prediction accuracy.
- Incompatible scales: Absolute error metrics depend on the measurement units and scale. Do not compare their numerical sizes across outcomes with different units as if they were directly equivalent.
A frequent mistake is to report the number of records assigned to a subgroup as the metric denominator even though some assigned records did not contribute an observed-predicted pair. The correction is to define the evaluated set explicitly and report the count of records used in the sums. Another common mistake is to take an unweighted average of subgroup RMSEs as the overall RMSE; the pooled formula averages squared RMSEs with count weights before taking the square root.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A subgroup has errors of -2, 0, and 4 mg/L. Calculate its mean error, MAE, RMSE, and evaluated observation count.
- Why can a subgroup have a mean error of zero while still having appreciable prediction error?
- Two groups have 5 and 15 evaluated observations. Why is the unweighted mean of their MAEs generally not the pooled MAE?
- What does the observation count in a subgroup row mean when the data include repeated measurements from subjects?
Solutions. (1) The count is 3. Mean error is \((-2+0+4)/3=0.667\) mg/L; MAE is \((2+0+4)/3=2.000\) mg/L; RMSE is \(\sqrt{(4+0+16)/3}=\sqrt{20/3}=2.582\) mg/L. (2) Positive and negative errors can cancel in the signed sum; MAE and RMSE summarize magnitude without that cancellation. (3) The pooled MAE weights each group by its evaluated count: \((5\text{ MAE}_1+15\text{ MAE}_2)/20\). An unweighted average assigns each group half the weight, which is generally different. (4) It is the number of evaluated records, not necessarily the number of distinct subjects; one subject may contribute multiple observations.