Why Summarize Prediction Error by Subject?
A diagnostic profile can show how predictions track observations within an individual. A compact subject-level summary adds a complementary view: it describes the direction and size of that subject’s prediction errors across their records. This is useful when individuals contribute different numbers of observations. A summary that gives every record equal influence answers a different question from one that gives every subject equal influence.
The calculations begin with matched observed and predicted values on the same records. For this tutorial, a positive error means the observed value exceeds the prediction. State that convention in a report, because reversing the subtraction reverses the interpretation of bias.
Bias is signed and can be close to zero when positive and negative errors cancel. MAE summarizes the average absolute discrepancy, while RMSE gives relatively greater influence to larger errors because it squares them before averaging. These are descriptive summaries of the selected records and predictions; they do not, by themselves, establish why predictions differ from observations.
The errors and all three summaries have the same units as the observed values. For example, if concentrations and predictions are in mg/L, the bias, MAE, and RMSE are also in mg/L. Calculate them on the scale specified for the analysis. A summary calculated after a log transformation describes errors on the log scale, not directly in mg/L.
Calculate Each Subject’s Summaries
The required assumptions are primarily about data construction and interpretation. Each observed value must be paired with the prediction for the same subject and record; the observed and predicted values must use compatible units and scales; and the records included in the calculation must be defined consistently. A subject needs at least one valid pair. These are descriptive calculations, so repeated records within a subject do not need to be independent for the arithmetic to be defined. However, treating records as independent when making inferential claims would require additional justification.
Worked Example: Calculating Error Summaries for One Subject
Suppose a hypothetical subject has three observed concentrations, in mg/L, of 4, 8, and 10, with corresponding predictions of 5, 7, and 8. The signed errors are observed minus predicted: \(-1\), \(1\), and \(2\) mg/L. The pairs are matched by record, and all values are on the same concentration scale.
The subject-level bias is
The positive bias means that, on average over these three records, observations were 0.667 mg/L higher than the predictions. The absolute errors are 1, 1, and 2 mg/L, so
The squared errors are 1, 1, and 4 \((\text{mg/L})^2\). Therefore,
The bias is smaller than the MAE because the signed errors partly cancel. Reporting bias alone would not communicate the typical size of the discrepancies in this example.
For a reproducible report, retain the number of paired records alongside each subject’s summaries. For example, display subject identifier, \(n_i\), \(B_i\), \(MAE_i\), and \(RMSE_i\), with units stated in the column labels or caption. The count provides essential context: a subject summary based on one record has a different evidential basis from one based on many records, even though the formula is the same.
Choosing How to Combine Subjects
A second decision arises when a report gives an overall summary across subjects. An equal-subject summary first calculates a metric for every subject and then averages those subject-level metrics. A record-weighted summary gives each paired record equal weight. When subjects have unequal record counts, these are generally different summaries and answer different questions.
For signed error, the relationship follows directly from the definitions. The pooled record-level bias is the sum of all errors divided by all records. Regrouping that sum by subject gives the weighted average of subject biases:
By contrast, the equal-subject mean bias is \(\frac{1}{S}\sum_{i=1}^{S}B_i\). It describes the average subject’s bias, giving a subject with few records the same influence as a subject with many. Neither weighting scheme is universally preferable; name the target and report the weighting.
Worked Example: Unequal Record Counts Change the Overall Summary
Consider hypothetical concentration data in mg/L for three subjects. Subject A has errors \(1\) and \(-1\); subject B has three errors, each equal to \(1\); subject C has one error equal to \(2\). The record counts are therefore 2, 3, and 1, for \(N=6\) records. The within-subject summaries are:
| Subject | \(n_i\) | Errors (mg/L) | \(B_i\) (mg/L) | \(MAE_i\) (mg/L) | \(RMSE_i\) (mg/L) |
|---|---|---|---|---|---|
| A | 2 | 1, -1 | 0 | 1 | 1 |
| B | 3 | 1, 1, 1 | 1 | 1 | 1 |
| C | 1 | 2 | 2 | 2 | 2 |
The equal-subject mean bias is
The record-weighted bias is
For MAE, the equal-subject mean is \((1+1+2)/3=1.33\) mg/L, whereas the record-weighted MAE is \((2(1)+3(1)+1(2))/6=7/6=1.17\) mg/L. For RMSE, do not take a record-weighted average of the subject RMSE values. Pool the squared errors first:
The equal-subject mean of the three subject RMSE values is \((1+1+2)/3=1.33\) mg/L. These results differ because they summarize the subjects and records with different weights. Subject C’s single error has the same influence as each of A’s and B’s records in the equal-subject summary, but only one-sixth of the influence in the pooled record summary.
A Practical Calculation and Reporting Workflow
A reliable automated summary should make the unit of analysis and weighting explicit. The following sequence keeps the calculation auditable and helps prevent silent changes in which records contribute.
Specify which predictions are being summarized, the observed outcome, units, and whether calculations use the original or transformed scale.
Check that each observed value and prediction refer to the same subject and record. Apply a documented rule for records without either value.
Compute \(e_{ij}=y_{ij}-\hat{y}_{ij}\), then \(B_i\), \(MAE_i\), and \(RMSE_i\) over that subject’s valid pairs.
State whether subjects are equally weighted or records are equally weighted. Include subject counts and record counts where relevant.
Inspect counts, units, error signs, extreme values, and any excluded records before interpreting the summaries.
A compact subject-level table is often more informative than a single overall number. It preserves between-subject differences and makes it possible to see whether a pooled result is driven by a small number of subjects with many records. If an overall number is also shown, label it precisely—for example, “mean subject MAE” or “record-weighted MAE”—rather than simply “MAE.”
Checks, Comparisons, and Failure Modes
The main alternative to equal-subject summaries is pooling records, as shown above. A third useful presentation is to show the subject-level results without collapsing them to one across-subject number. These approaches are complementary: the subject table exposes heterogeneity, the equal-subject summary characterizes an average subject, and the record-weighted summary characterizes an average record in the assembled data.
| Approach | Weighting | Question it describes | Important caution |
|---|---|---|---|
| Subject-level table | No across-subject averaging | How do summaries vary by subject? | Record counts may differ substantially. |
| Equal-subject mean | Each subject has equal weight | What is the average subject-level summary? | A subject with few records has the same weight as one with many. |
| Record-weighted summary | Each paired record has equal weight | What is the average error summary over the assembled records? | Subjects with more records have greater influence. |
For RMSE, the pooled record summary is the square root of a record-weighted average of subject mean squared errors, not a weighted average of subject RMSEs. This follows because each subject’s mean squared error is \(RMSE_i^2\):
Taking the square root only after pooling squared errors is necessary to preserve the definition of RMSE. Averaging the subject RMSE values first computes a different quantity.
Several checks catch common failures:
- Reversed subtraction. If software calculates prediction minus observation while the report says observation minus prediction, the sign interpretation of bias is reversed. Verify the convention with a simple pair: an observation of 12 and prediction of 10 should give an error of \(+2\) under the convention used here.
- Mispaired records or duplicated rows. A prediction attached to the wrong observation, or a record counted twice, changes the errors and their weights. Confirm the matching keys and record counts before aggregation.
- Unstated exclusions. Missing values or records excluded by a data rule can change \(n_i\), \(N\), and the resulting summaries. Report the valid-pair rule and retain counts so exclusions are visible.
- Interpreting zero bias as accurate predictions. Positive and negative errors can cancel. In Worked Example 1, the first two errors cancel, yet their MAE is 1 mg/L. Examine a magnitude-based summary as well as signed bias.
- Comparing unlike scales. Errors in mg/L cannot be directly compared in magnitude with errors in another unit or on a log scale without an explicit scale interpretation. Label units and transformations.
These summaries have limits. Bias can hide large but opposing errors; MAE and RMSE do not indicate direction; and a summary can conceal patterns over time or across conditions within a subject. The number of records also does not, by itself, measure how representative those records are. These are descriptive measures for the chosen observed-predicted pairs, not proof of model adequacy or evidence that a particular mechanism caused the error. Read them alongside the relevant diagnostic displays and the data-processing record.
Worked Example: Checking Cancellation Before Reporting Bias
A hypothetical subject has observed values of 6 and 10 mg/L, with predictions of 8 and 8 mg/L. The signed errors are \(6-8=-2\) and \(10-8=2\) mg/L. The bias is \((-2+2)/2=0\) mg/L, but the MAE is \((2+2)/2=2\) mg/L and the RMSE is \(\sqrt{(4+4)/2}=2\) mg/L. The calculations are valid because both pairs use the same units and the stated error convention. A report that included only zero bias would omit the fact that both predictions differ from their observations by 2 mg/L.
Check Your Understanding
Work through each item, then compare your result with the solutions below.
- A subject has observed values 3 and 7 mg/L and predictions 4 and 6 mg/L. Calculate the signed errors, bias, MAE, and RMSE.
- Two subjects have biases of \(-1\) and \(3\) mg/L, based on 2 and 6 records, respectively. Find the equal-subject mean bias and the record-weighted bias.
- Why can a subject have zero bias but a nonzero MAE? Give a numerical illustration.
- Three subjects have RMSE values 1, 2, and 3 mg/L and record counts 2, 1, and 1. Calculate the pooled record RMSE.
- Name two checks to make before comparing subject-level summaries from an automated report.
Solutions. (1) Errors are \(-1\) and \(1\) mg/L. Bias is \((-1+1)/2=0\) mg/L, MAE is \((1+1)/2=1\) mg/L, and RMSE is \(\sqrt{(1+1)/2}=1\) mg/L. (2) The equal-subject mean is \((-1+3)/2=1\) mg/L. The record-weighted bias is \(\{2(-1)+6(3)\}/8=16/8=2\) mg/L. (3) Errors of \(-2\) and \(2\) mg/L sum to zero, so bias is zero, while MAE is \((2+2)/2=2\) mg/L. (4) Pool squared errors using the subject RMSEs: \(RMSE_{\text{records}}=\sqrt{\{2(1^2)+1(2^2)+1(3^2)\}/4}=\sqrt{15/4}=1.94\) mg/L. (5) Appropriate checks include confirming that observations and predictions are correctly paired, that units and scales match, that the error sign convention is stated, and that record counts and exclusions are consistent.