Interpreting Prediction Error Patterns

Why Prediction-Error Patterns Matter

An overall prediction-error summary can look acceptable while a model consistently misses in a clinically important part of its intended use. For example, errors may be small at low concentrations but show systematic underprediction at high concentrations. Or average error may be near zero because positive and negative errors cancel, even though predictions are poor under a particular dosing history or measurement condition. Interpreting the pattern of errors helps locate such failures and guides what to investigate next.

The goal is not to infer a cause from a plot alone. A pattern can signal a model limitation, a data-processing problem, a mismatch between prediction time and sample time, or differences in measurement conditions. The first task is to describe where errors occur, in what direction, and with what magnitude. The earlier material on calibration problems addresses calibration assessment; here the emphasis is on reading error summaries across clinically relevant subsets without letting an aggregate obscure them.

A residual is often defined as observed minus fitted value, so its sign convention may match this one. The distinction is about how the prediction was obtained: a residual commonly describes a value used to fit the model, whereas a prediction error should be calculated for a prediction evaluated against an observation not used to produce that prediction, such as a held-out or prospective record. In either case, state the convention explicitly. Error patterns from fitted records can appear overly favorable because the same data helped estimate the model.

Summaries That Answer Different Questions

The mean signed error in a group of \(n\) records describes average direction and magnitude of bias on the original concentration scale. The mean absolute error (MAE) describes average error magnitude without allowing positive and negative errors to cancel. The root mean squared error (RMSE) also penalizes large errors more heavily. These summaries are complementary, not interchangeable.

All three summaries have concentration units. A signed mean near zero does not establish that predictions are accurate: positive and negative errors can cancel. Conversely, a positive mean describes average underprediction under the stated convention; it does not imply that every record is underpredicted. A spread summary, such as the sample standard deviation \(s\) of signed errors, or a display of the individual errors, adds information about variability within a group.

Patterns are often examined by plotting signed errors against predicted concentration, observed concentration, time, dose, or a relevant measurement variable. Grouped summaries can then quantify a visible pattern. Use both where possible: plots reveal shapes, outliers, and transitions, while summaries make comparisons more concise. Stratification should be clinically motivated and specified with care; choosing cut points after inspecting many possible groupings can make an apparent pattern unstable.

Reading Patterns Across Dosing Histories

A prediction depends on the information available when it is generated. A recent dose change, a missed dose, an uncertain administration time, or a record that does not reflect the actual dosing history may be relevant to its error. When comparing histories, verify that the prediction and observed concentration are aligned to the same intended time and that the history labels are defined consistently. Otherwise, the group comparison may mix genuine differences in model performance with timing or data-recording errors.

Stratifying by history is informative when the groups have enough observations to support a meaningful summary and when the groups are comparable for the question being asked. If a particular history occurs only at high concentrations, for example, a history-associated pattern could instead reflect concentration range, or both factors could contribute. Examine the underlying values and relevant variables rather than treating a subgroup mean as an isolated explanation.

Measurement Conditions and Error Dispersion

A group can have low average signed error but high variability. Differences in assay, sampling time, sample handling, or measurement precision may matter when those conditions differ across records. A mean signed error describes direction on average, not how tightly individual predictions agree with observations. Compare signed errors with absolute errors or a spread measure, and inspect the individual values for unusual observations.

Measurement noise is part of the observed prediction error: the observation is not necessarily the unknown true concentration. A pattern that appears only under one measurement condition may therefore reflect measurement characteristics, a timing mismatch, or a model limitation. Confirm that units, sample identifiers, assay conventions, and timing fields are consistent before interpreting it as model behavior.

A Practical Interpretation Sequence

A defensible reading of prediction-error patterns begins with data alignment and proceeds from broad summaries to specific subsets. The following sequence keeps the interpretation tied to what the records support.

Deriving the Pooled Mean and Recognizing Cancellation

The overall mean signed error is a size-weighted average of subgroup means. If records are partitioned into groups \(g=1,\ldots,G\), with \(n_g\) records and subgroup mean \(\bar{e}_g\), then \(\sum_g n_g=N\), where \(N\) is the total number of records. Since each subgroup mean is \(\bar{e}_g=(1/n_g)\sum_{i\in g}e_i\), the overall mean is

This identity explains why subgroup means must be weighted by group size and why an overall mean can conceal opposing patterns. For example, if two equally sized groups have means \(+5\) and \(-5\) mg/L, the pooled mean is \((5+(-5))/2=0\) mg/L even though neither group has zero mean error. If group sizes differ, the larger group contributes more to the pooled result. The MAE does not cancel errors of opposite sign, but an overall MAE can still hide which subgroup contributes most of the error.

Assumptions, Diagnostics, and Common Pitfalls

Descriptive summaries do not require errors to be normally distributed. Their interpretation does require valid pairings, a consistent sign convention, common units, and a clear definition of the evaluation records. A subgroup comparison is most useful when the subgroup variable is measured reliably and the group sizes are reported. If several records come from the same patient, those errors may be correlated; counting every record as independent can overstate how much independent information is present. For descriptive reporting, show the record counts and, where relevant, the number of patients. Formal uncertainty calculations would need to account for clustering and the evaluation design.

Diagnostic signs of trouble include errors that change direction across a concentration range, a subgroup with much higher MAE or spread, isolated extreme errors, and apparent patterns that disappear after correcting timing or unit inconsistencies. A single outlier should be checked against source data, but should not be removed simply because it is inconvenient. When records are sparse, show the individual errors and avoid confident generalization from a subgroup average. When many possible stratifications are examined, label exploratory findings as such and seek confirmation in additional appropriate data.

• Mistake: “The overall mean error is nearly zero, so predictions are accurate.” Positive and negative errors can cancel. Compare MAE or RMSE and inspect subgroup patterns.
• Mistake: “A positive error means the model overpredicts.” With \(e_i=C_i-\hat{C}_i\), a positive error means the prediction is below the observation: underprediction.
• Mistake: “A subgroup difference proves the subgroup characteristic caused the error.” The difference is descriptive; other characteristics, measurement conditions, or data problems may explain it.
• Mistake: “Every row is an independent evaluation.” Repeated records from one patient can be related. Report the structure and account for it when making inferential claims.

No single summary is universally best. The signed mean is useful for average direction; MAE gives an interpretable average magnitude; RMSE emphasizes large errors; and plots show how errors vary across the predictor or clinical context. These approaches answer different questions and should generally be combined. Error-pattern analysis is limited by the quality and representativeness of the evaluation records, measurement uncertainty, sparse subgroups, and the possibility that several factors change together. Its strongest conclusion is often a precise description of where performance differs and what should be checked next—not a causal diagnosis.