Success Criteria for Consequential Large Errors

When an Average Error Is Not Enough

A model can perform well on most observations and still make a few predictions that are unacceptably wrong. If a hypothetical insurer forecasts routine claim amounts, a large miss on one unusually costly claim may matter more than several small misses. Similarly, a staffing forecast that is usually close but occasionally far too low can create a serious service shortfall. The evaluation question is not only how wrong is the model on average? It is also how often are errors severe, and how severe are they?

The earlier material on MAE and RMSE distinguishes average absolute error from average squared error. Those summaries are useful, but neither by itself states whether a chosen severity threshold is exceeded often enough to be a concern. A success criterion for consequential errors makes the unacceptable region explicit and evaluates model performance there.

Definition: For observation \(i\), let \(y_i\) be the observed outcome and \(\hat{y}_i\) the prediction. Define the signed prediction error as \(e_i=y_i-\hat{y}_i\), and its magnitude as \(|e_i|\). Given a nonnegative consequential-error threshold \(c\), an observation is a threshold exceedance when \(|e_i|>c\). The empirical exceedance rate in \(n\) evaluated observations is the number of exceedances divided by \(n\).

The absolute error is appropriate when errors in either direction can be harmful. If underprediction is the main concern, however, use the signed error consistently: with \(e_i=y_i-\hat{y}_i\), an underprediction has \(e_i>0\). A one-sided criterion can then count cases with \(e_i>c\). Choosing the direction and threshold requires understanding the consequences in the application; the calculation cannot make that choice on its own.

Describe Both Frequency and Severity

A threshold exceedance rate answers a frequency question: what fraction of evaluated predictions missed by more than \(c\)? A tail-severity measure answers a different question: among those misses, how large were the errors? These are complementary. A model can exceed a threshold frequently but only slightly, or exceed it rarely with very large errors.

\widehat{p}_c=\frac{1}{n}\sum_{i=1}^{n} I(|e_i|>c)

Here \(I(|e_i|>c)\) equals 1 when the condition is true and 0 otherwise. The average exceedance amount, conditional on exceeding the threshold, is

\frac{\sum_{i=1}^{n}|e_i|I(|e_i|>c)} {\sum_{i=1}^{n}I(|e_i|>c)}

This conditional average is defined only when at least one observation exceeds \(c\). It reports the average size of the errors in the exceedance group. Another useful quantity is the average amount above the threshold, obtained by replacing each exceedance magnitude \(|e_i|\) with \(|e_i|-c\). State which version is being reported; “average tail error” alone is ambiguous.

Worked Example: Equal MAE, Different Large-Error Protection

A hypothetical service organization compares two forecasts on five comparable held-out cases. Their absolute errors are: Model A, \(2, 3, 4, 5, 20\) units; Model B, \(6, 6, 7, 7, 8\) units. A miss greater than 10 units is considered consequential. Compare MAE, RMSE, and the threshold exceedance rate.

Method. Calculate each model's average absolute error, root mean squared error, and fraction of cases with absolute error greater than 10. This uses the same five cases and the same error units for both models.

Model A MAE. The absolute errors sum to \(2+3+4+5+20=34\). Therefore, \(\mathrm{MAE}_A=34/5=6.8\) units.

Model B MAE. The errors sum to \(6+6+7+7+8=34\). Therefore, \(\mathrm{MAE}_B=34/5=6.8\) units. MAE alone gives a tie.

Model A RMSE. The squared errors sum to \(2^2+3^2+4^2+5^2+20^2=4+9+16+25+400=454\). Thus, \(\mathrm{RMSE}_A=\sqrt{454/5}=\sqrt{90.8}\approx9.529\) units.

Model B RMSE. The squared errors sum to \(6^2+6^2+7^2+7^2+8^2=36+36+49+49+64=234\). Thus, \(\mathrm{RMSE}_B=\sqrt{234/5}=\sqrt{46.8}\approx6.841\) units.

Threshold exceedance. Model A has one error greater than 10, so its exceedance rate is \(1/5=0.20\). Model B has none, so its rate is \(0/5=0\). The observed data therefore show one consequential miss for A and none for B.

Check. The models have equal total absolute error, 34 units, but A's single 20-unit miss contributes \(400\) to its sum of squared errors. The calculations agree: RMSE and the threshold criterion both distinguish the severe miss, while MAE does not. Common mistake avoided: concluding that equal MAE means the models have the same protection against large errors.

Why Squared Error Reacts to Large Misses

RMSE is a measure of average error magnitude that gives extra influence to large errors. This follows directly from squaring: increasing an error from \(a\) to \(b\), where \(b>a\geq0\), increases its squared-error contribution by \(b^2-a^2=(b-a)(b+a)\). The increase is larger when the starting or ending error is large. Taking the square root at the end restores the units of the outcome, but does not undo the greater influence in the sum.

Justification: For errors \(e_1,\ldots,e_n\), \(\mathrm{RMSE}=\sqrt{(1/n)\sum e_i^2}\). If one error changes in magnitude from \(a\) to \(b\), its contribution to the quantity inside the square root changes by \((b^2-a^2)/n=(b-a)(b+a)/n\). Because \(b+a\) grows with error size, a given increase in an already large error adds more squared error than the same increase in a small error. RMSE therefore penalizes large misses more strongly than MAE, but it is still an average and does not guarantee that no error exceeds a specified limit.

This distinction matters when setting success criteria. If the requirement is “reduce typical error,” MAE or RMSE may be suitable, depending on the consequences of error size. If the requirement is “no more than a specified fraction of forecasts may miss by over 10 units,” calculate that exceedance rate and compare it with the stated limit. A low RMSE is not a substitute for checking that requirement.

Worked Example: Frequency and Severity Beyond a Limit

A hypothetical claims team evaluates absolute reserve-prediction errors, in thousands of dollars, of \(3, 4, 5, 7, 12, 15, 18, 30\). An error greater than 10 thousand dollars is consequential. Calculate the exceedance rate, the average error magnitude among exceedances, and the average amount above the threshold.

Method. Identify all errors greater than 10, then calculate frequency and two clearly distinguished severity summaries.

Exceedance cases. The values \(12, 15, 18, 30\) exceed 10. There are 4 exceedances among 8 cases, so the rate is \(4/8=0.5\), or 50%.

Average magnitude among exceedances. The exceedance errors sum to \(12+15+18+30=75\). Their conditional average magnitude is \(75/4=18.75\) thousand dollars.

Average amount above 10. The excess amounts are \(12-10=2\), \(15-10=5\), \(18-10=8\), and \(30-10=20\), all in thousands of dollars. Their sum is \(2+5+8+20=35\), so the average amount above the threshold is \(35/4=8.75\) thousand dollars.

Check. The conditional average magnitude must equal the threshold plus the average excess: \(10+8.75=18.75\). Common mistake avoided: reporting 8.75 thousand dollars as the average error among exceedances. It is the average excess above the limit, not the full error magnitude.

Quantiles and Maximum Error

A high quantile of absolute error describes a point below which a stated proportion of observed errors fall. It can make the upper part of the error distribution visible without letting one extreme observation determine the whole summary. The maximum error reports the single largest observed miss and is easy to interpret when even one such miss matters. Both have limitations: a quantile can fail to reveal a rare extreme event, while a maximum can change sharply when one new case is added.

Definition: For a finite sample, define the empirical \(q\)-quantile using the inverse empirical cumulative distribution function: sort the \(n\) absolute errors from smallest to largest and select the value at rank \(\lceil nq\rceil\), for \(0<q\leq1\). The maximum absolute error is the largest of the \(n\) absolute errors. Quantile conventions can differ, so name the convention when a precise sample calculation is required.

Worked Example: A High Quantile Can Hide a Rare Miss

Two hypothetical forecasting systems have ten absolute errors each, in service units. System A's sorted errors are \(1,1,2,2,3,3,4,4,5,30\). System B's sorted errors are \(4,4,4,4,5,5,5,5,6,7\). Compare their MAE, their 90th percentile using the stated inverse empirical CDF convention, and their maximum error.

System A MAE. The sum is \(1+1+2+2+3+3+4+4+5+30=55\), giving \(55/10=5.5\) units.

System B MAE. The sum is \(4+4+4+4+5+5+5+5+6+7=49\), giving \(49/10=4.9\) units.

90th percentiles. With \(n=10\) and \(q=0.90\), the selected rank is \(\lceil10(0.90)\rceil=9\). The ninth value for A is 5 units, and the ninth value for B is 6 units. By this quantile summary, A's 90th percentile is lower.

Maximum errors. A's maximum is 30 units; B's maximum is 7 units. The maximum exposes A's isolated severe miss, which its ninth-ranked value does not include.

Check. A has the lower 90th percentile but the higher MAE and a much higher maximum. The metrics answer different questions, so the quantile must not be presented as proof that extreme errors are controlled. Common mistake avoided: assuming a high quantile necessarily captures every rare but consequential observation.

Set Criteria Around the Consequence

A defensible evaluation plan states both the primary average-performance measure and any explicit tail-protection requirement. For example, a team might require a reasonable MAE and also require that no more than a specified fraction of errors exceed a threshold. The threshold should be expressed in units that have operational meaning, and the exceedance rate should be calculated on observations relevant to the intended use.

When direction matters, separate underprediction from overprediction rather than allowing positive and negative errors to cancel. With \(e_i=y_i-\hat{y}_i\), one can report the fraction with \(e_i>c\) for a consequential underprediction threshold, alongside the fraction with \(|e_i|>c\) if either direction is important. State the denominator and whether equality counts: “greater than \(c\)” differs from “at least \(c\).”

Evaluation results are empirical summaries, not guarantees about future cases. A sample with no exceedances does not prove that future exceedances are impossible. The observations should reflect the situations in which predictions will be used, including relevant rare or high-severity cases when available. Where the data contain few such cases, report that limitation rather than interpreting a small observed rate as strong evidence of protection.

Describe the consequence.
Specify which direction of error is harmful and what magnitude becomes consequential, in context and in the outcome's units.

Choose an average measure.
Use MAE or RMSE to summarize overall error, recognizing that RMSE gives greater influence to large misses.

Add a tail criterion.
Report a threshold exceedance rate, a clearly defined severity summary, a high quantile, or a maximum when it addresses the stated concern.

Interpret evidence cautiously.
Give the number of evaluated cases and explain what the observed summaries establish—and what they cannot guarantee about future errors.

The central exam distinction is between optimizing average performance and setting a separate standard for severe misses. RMSE may respond strongly to large errors, but it is not a direct limit on their frequency or size. A precise threshold criterion makes the protection requirement visible and testable.

Check Your Understanding

For each problem, use the stated error definition and threshold carefully.

A forecasting system has absolute errors of \(2, 4, 6, 8, 20\) units. For a consequential threshold of 10 units, calculate the exceedance rate, the average magnitude among exceedances, and the average amount above the threshold. (Answer: exceedance rate \(1/5=0.20\); average exceedance magnitude \(20\) units; average amount above threshold \(10\) units.)
Model A has absolute errors \(1,1,2,2,14\), and Model B has errors \(5,5,5,5,5\). Calculate both MAEs, both RMSEs to three decimals, and each exceedance rate for errors greater than 10. (Answer: MAE A \(=4\), MAE B \(=5\); RMSE A \(=\sqrt{206/5}\approx6.419\), RMSE B \(=5.000\); exceedance rates A \(=0.20\), B \(=0\).)
Six absolute errors are \(3,7,11,12,20,25\) units. For threshold 10, calculate the exceedance rate, average magnitude among exceedances, and average excess above the threshold. (Answer: exceedance rate \(4/6\approx0.6667\); average magnitude \(68/4=17\) units; average excess \(28/4=7\) units.)
Ten sorted absolute errors are \(1,2,3,4,5,6,7,8,9,40\). Using the inverse empirical CDF convention, find the 90th percentile and the maximum. Explain in one sentence why the two summaries differ. (Answer: 90th percentile \(=9\) units; maximum \(=40\) units; the selected ninth-ranked error does not include the largest observation.)
For three cases, actual outcomes are \(12,16,20\), and predictions are \(10,20,14\). Using \(e_i=y_i-\hat{y}_i\), calculate the underprediction rate, MAE, and maximum absolute error. (Answer: errors are \(2,-4,6\); underprediction rate \(=2/3\); MAE \(=(2+4+6)/3=4\); maximum absolute error \(=6\).)

What You'll Learn