Error Patterns Across Time and Operations

Why Slice Realized Errors by Time and Process?

An overall performance measure can show that predictions are inaccurate without revealing when or where the errors occur. A rise in average error might follow a workflow change, or it might reflect a shift toward a more difficult acquisition channel. Likewise, a small overall signed error can conceal substantial overprediction in one period and underprediction in another.

Time-based and operational slices make those patterns visible. The aim is to organize predictions with their eventual observed outcomes by a meaningful time marker or process attribute, then compare consistently defined errors. This is descriptive diagnosis: a pattern can point to a question worth investigating, but it does not by itself prove what caused the pattern.

Definition: For observation $i$, let $y_i$ be the realized outcome and $\hat{y}_i$ the prediction made for that observation. Define the signed error as $e_i=y_i-\hat{y}_i$. A positive error means the prediction was too low; a negative error means it was too high. The absolute error is $|e_i|$, and the squared error is $e_i^2$. A time slice groups observations by a defined period; an operational slice groups them by a process feature such as workflow stage, acquisition channel, or handling route.

These definitions use the convention that error equals actual minus predicted. Other settings may define a residual with the opposite sign, so state the convention before interpreting direction. A signed-error average summarizes direction, while mean absolute error (MAE) summarizes typical error size in the outcome’s units. Root mean squared error (RMSE) also measures error size in those units but gives greater influence to large errors.

\overline{e}_G=\frac{1}{n_G}\sum_{i\in G}e_i,\qquad \operatorname{MAE}_G=\frac{1}{n_G}\sum_{i\in G}|e_i|,\qquad \operatorname{RMSE}_G=\sqrt{\frac{1}{n_G}\sum_{i\in G}e_i^2}

Here $G$ is a defined slice and $n_G$ is the number of observations in it. These are descriptive summaries of realized errors in the slice; they do not require a probability distribution assumption. Their interpretation does require meaningful and comparable predictions and outcomes: use the same outcome definition, prediction horizon, units, and error convention across slices. Include only observations whose outcomes are available and sufficiently mature for the chosen horizon.

A Consistent Diagnostic Process

A useful comparison begins with the operational question, not with a search through many arbitrary groupings. If a workflow changed on a known date, compare periods around that date. If cases enter through different channels, compare those channels. Define periods and categories in a way that would be interpretable to the people responsible for the process.

Align predictions and outcomes.
Match each prediction to its realized outcome and record the relevant prediction date, outcome period, and process attributes. Confirm that outcome availability is comparable across groups.

Choose slices that reflect a question.
Examples include calendar periods around a workflow change or an acquisition channel. Specify cut points and category definitions before interpreting the results.

Calculate more than one useful summary.
Review the number of observations, signed error, and an error-size measure such as MAE or RMSE. A direction measure alone can hide cancellation; an error-size measure alone does not show whether predictions tend to be high or low.

Check the pattern and its context.
Look for a repeated change or a concentration of errors in a process slice. Then consider volume, changing case mix, and data-handling differences before suggesting an explanation.

A short derivation explains why an overall MAE must be weighted by slice volume. Suppose a data set is divided into non-overlapping groups $G_1,\ldots,G_k$. Group $j$ has $n_j$ observations and MAE $\operatorname{MAE}_{G_j}$. By the definition of MAE, its sum of absolute errors is $n_j\operatorname{MAE}_{G_j}$. Summing those group totals and dividing by the total number of observations gives:

\operatorname{MAE}_{\mathrm{all}} =\frac{\sum_{j=1}^{k}\sum_{i\in G_j}|e_i|}{\sum_{j=1}^{k}n_j} =\frac{\sum_{j=1}^{k}n_j\operatorname{MAE}_{G_j}}{\sum_{j=1}^{k}n_j}

Thus, the overall MAE is a volume-weighted average of group MAEs, not generally their simple average. The same reasoning applies to mean signed error and mean squared error; take the square root only after aggregating mean squared error to obtain overall RMSE.

Worked Example: Locating a Time-Based Change in Direction

A hypothetical claims team reviews eight matured predictions, four from each of two consecutive weeks. For each claim, the signed error $e_i=y_i-\hat{y}_i$, measured in hundreds of dollars, is recorded. Week 1 errors are $2,-1,1,-2$; Week 2 errors are $3,2,4,3$. Calculate each week’s mean signed error and MAE, then interpret the difference.

Method. Calculate signed and absolute errors within each week, using the same unit and error convention. There are four observations in each group.

Week 1. The signed-error sum is $2+(-1)+1+(-2)=0$, so the mean signed error is $0/4=0$. The absolute errors are $2,1,1,2$, with sum $6$, so $\operatorname{MAE}=6/4=1.5$ hundreds of dollars.

Week 2. The signed-error sum is $3+2+4+3=12$, so the mean signed error is $12/4=3$ hundreds of dollars. The absolute errors are $3,2,4,3$, also summing to $12$, so $\operatorname{MAE}=12/4=3$ hundreds of dollars.

Interpretation and check. Week 1 has no average signed error, but its individual errors are not all zero; its MAE is 1.5. Week 2 errors are all positive, indicating predictions were too low on average, and its MAE is twice Week 1’s. In dollars, the weekly MAEs are $150 and $300. The common mistake is to infer perfect accuracy in Week 1 from its zero signed mean; the nonzero MAE shows why that conclusion is wrong.

Operational Slices and Workload Mix

Operational attributes can identify where errors recur: for example, whether one acquisition channel has larger errors, or whether a particular handling route is associated with systematic underprediction. Keep the interpretation tied to the observed slice. A channel may contain a different mix of cases, and a workflow group may be assigned more complex cases. A difference between groups is not automatically a difference caused by the process itself.

Counts belong beside error measures. An MAE based on a few observations may be less informative than the same MAE based on many observations, and a missing or delayed outcome can make a slice appear artificially favorable if it excludes difficult cases. Also avoid comparing a metric computed over different outcome horizons or with different inclusion rules.

Worked Example: Comparing Two Acquisition Channels

A hypothetical service team records four matured predictions from each of two acquisition channels. Errors, in tens of dollars, are $1,1,-1,1$ for the direct channel and $4,2,3,3$ for the partner channel. Calculate the mean signed error and MAE for each channel and for all eight cases.

Method. Sum signed errors to describe direction and absolute errors to describe size. Then combine channel summaries using their observation counts.

Direct channel. The signed sum is $1+1-1+1=2$, so mean signed error is $2/4=0.5$. Absolute errors sum to $1+1+1+1=4$, so MAE is $4/4=1$.

Partner channel. The signed sum is $4+2+3+3=12$, so mean signed error is $12/4=3$. All errors are positive, and the absolute-error sum is also $12$, giving MAE $12/4=3$.

All cases. The total signed sum is $2+12=14$, so the overall mean signed error is $14/8=1.75$. The total absolute-error sum is $4+12=16$, so overall MAE is $16/8=2$. Checking the weighted calculation gives $(4(1)+4(3))/8=2$ for MAE.

Interpretation and check. The partner channel has larger errors in this small hypothetical sample, and its mean signed error indicates underprediction. Both channels have equal volume, so their simple average MAE happens to equal the overall MAE. The common mistake is to treat that equality as a general rule; it occurs here because the group sizes match. The comparison flags a pattern to investigate, not proof that the channel caused the errors.

Worked Example: Separating a Time Change From a Process-Mix Change

A hypothetical underwriting operation compares errors before and after a routing change. Within the routine route, every absolute error is 1 unit in both periods. Within the specialist route, every absolute error is 5 units in both periods. Before the change, there are 8 routine and 2 specialist cases; after it, there are 2 routine and 8 specialist cases. Assume every listed case has a positive signed error. Calculate the overall MAE and mean signed error in each period, then decide whether either process-specific error size changed.

Method. First calculate each period’s aggregate from the process-specific counts and error sizes. Then compare like process groups across periods rather than treating the aggregate alone as evidence of deterioration.

Before the change. The absolute-error total is $8(1)+2(5)=8+10=18$ over $8+2=10$ cases, so MAE is $18/10=1.8$. Since every signed error is positive, the signed-error total is also 18 and the mean signed error is $18/10=1.8$.

After the change. The absolute-error total is $2(1)+8(5)=2+40=42$ over $2+8=10$ cases, so MAE is $42/10=4.2$. The signed-error total is also 42, giving mean signed error $42/10=4.2$.

Interpretation and check. Aggregate MAE rose from 1.8 to 4.2, but routine-route MAE remained 1 in each period and specialist-route MAE remained 5. The stated data show a changed workload mix, not a change in either route’s error size. The common mistake is to label the aggregate increase as model deterioration without examining process composition. This example does not establish why the mix changed or whether other operational changes occurred.

Common Traps and Responsible Interpretation

Signed errors cancel. A mean signed error near zero can coexist with large individual errors. Pair it with MAE or RMSE when error magnitude matters.
Group averages are not always comparable. Different group sizes require weighting to reconstruct an overall mean. Different case mixes may also affect comparisons even when the arithmetic is correct.
A time boundary is not a causal explanation. An error increase after a workflow change is a temporal association. Other simultaneous changes, changing cases, or outcome availability may contribute.
Unmatured outcomes distort slices. If labels arrive late, recent periods may omit cases whose outcomes have not yet been observed. Compare periods with a consistent maturity rule.
Too many slices invite selective interpretation. Choose operationally meaningful comparisons and report the counts and definitions, rather than highlighting only the largest observed difference.

Time- and process-based error summaries are useful because they translate a broad performance concern into a more specific diagnostic question: which period or route deserves examination? Their value depends on consistent definitions, transparent denominators, and restrained interpretation. An observed pattern is a starting point for investigation, not a substitute for evidence about its cause.

Check Your Understanding

For each problem, use signed error $e_i=y_i-\hat{y}_i$ and show the relevant group counts in your own working.

A hypothetical forecasting team has errors $2,-2,0$ in Period A and $1,1,1$ in Period B. Find each period’s mean signed error and MAE. (Answer: Period A mean signed error $=0$, MAE $=4/3$; Period B mean signed error $=1$, MAE $=1$.)
A service channel has 5 cases with MAE 2, and another has 15 cases with MAE 4. Find the overall MAE. (Answer: $(5(2)+15(4))/20=3.5$.)
In a hypothetical process, Group A has errors $-3,1,2$, and Group B has errors $2,2$. Find the mean signed error and MAE across all five cases. (Answer: Mean signed error $=4/5=0.8$; MAE $=10/5=2$.)
Before a routing change, there are 6 routine cases with MAE 1 and 4 specialist cases with MAE 4. Afterward, there are 2 routine cases with MAE 1 and 8 specialist cases with MAE 4. Find both overall MAEs and state what changed in the given data. (Answer: Before $=(6(1)+4(4))/10=2.2$; after $=(2(1)+8(4))/10=3.4$; the process-specific MAEs are unchanged while the workload mix shifts toward specialist cases.)
For a period with errors $1,-1,3,-3$, find the mean signed error, MAE, and RMSE. (Answer: Mean signed error $=0$; MAE $=2$; RMSE $=\sqrt{5}\approx2.236$.)

What You'll Learn