Reporting Numeric Prediction Errors

Make Numeric Errors Meaningful

A value such as “RMSE = 4.2” is difficult to use unless a reader knows what was predicted, the unit of measurement, and whether an error of that size matters for the decision. A useful report translates a metric into those terms: for example, “On the validation cases, the model’s root mean squared error was about $4,200 per claim.” That statement still needs context, but it gives a decision-maker a scale they can interpret.

This tutorial focuses on communicating errors for numeric predictions, such as predicted claim amounts or reserve amounts. It assumes that the predictions and validation outcomes have already been obtained. The reporting task is to explain what the errors indicate, compare them fairly with a relevant reference, and avoid claims that the validation results cannot support. This topic is aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Definition: For an observed numeric outcome $y_i$ and its prediction $\hat{y}_i$, define the signed prediction error as $e_i = y_i-\hat{y}_i$. Under this convention, a positive error means the prediction was too low (underprediction), and a negative error means it was too high (overprediction). State the convention in a report; reversing the subtraction reverses the interpretation of the sign.

Several familiar summaries answer different questions. The mean error describes the average signed direction of errors. The mean absolute error (MAE) describes the average magnitude without allowing overpredictions and underpredictions to cancel. The root mean squared error (RMSE) also summarizes magnitude, but gives greater influence to large errors because errors are squared before they are averaged. All three are expressed in the same unit as the numeric outcome.

\overline{e}=\frac{1}{n}\sum_{i=1}^{n}e_i,\qquad \operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|e_i|,\qquad \operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}e_i^2}

These are summaries of the evaluated cases, not guarantees about future errors. In particular, a mean error near zero does not mean the predictions were individually close: positive and negative errors can offset one another. Nor does an MAE of $500 mean that every prediction is within $500 of its outcome. A report should name the evaluation set or period, specify the outcome and unit, and explain the metric in ordinary language.

Why RMSE Reacts to Large Errors

RMSE is often useful when large misses deserve particular attention, but its response to them needs to be made clear. One way to see this is to separate the mean error from variation around that mean. Let $\overline{e}$ be the mean signed error. For each case, write $e_i=(e_i-\overline{e})+\overline{e}$. Since $\sum_{i=1}^{n}(e_i-\overline{e})=0$, expanding the square gives the identity below.

\begin{aligned} \frac{1}{n}\sum_{i=1}^{n}e_i^2 &=\frac{1}{n}\sum_{i=1}^{n} \left((e_i-\overline{e})+\overline{e}\right)^2\\ &=\frac{1}{n}\sum_{i=1}^{n}(e_i-\overline{e})^2 +\frac{2\overline{e}}{n}\sum_{i=1}^{n}(e_i-\overline{e}) +\overline{e}^{\,2}\\ &=\frac{1}{n}\sum_{i=1}^{n}(e_i-\overline{e})^2+\overline{e}^{\,2}. \end{aligned}

Thus, the mean squared error contains both the squared average signed error and the average squared deviation around that signed error. Taking the square root produces RMSE. This identity explains why RMSE can be high when errors have a consistent direction, when they vary substantially, or both. It does not make RMSE an estimate of the error for any particular case.

Worked Example: Explain a Large RMSE in Dollars

A hypothetical insurer evaluates five quarterly claim-amount predictions. The signed errors, in dollars, are $-10,\ 10,\ -10,\ 10,\ 300$. Calculate the mean error, MAE, and RMSE, then state what the results support saying.

Method and direction. Use $e_i=y_i-\hat{y}_i$, so a positive error indicates underprediction. There are $n=5$ evaluated cases.

Mean error. Add the signed errors: $-10+10=0$, then $0-10+10=0$, and finally $0+300=300$. Therefore,

\overline{e}=\frac{300}{5}=\$60.

On average, predictions were $60 below the observed outcomes under this sign convention. The average is small relative to the largest error, so it should not be presented as evidence that individual predictions were close.

MAE. Add the absolute errors: $10+10+10+10+300=340$. Hence,

\operatorname{MAE}=\frac{340}{5}=\$68.

The average absolute difference between prediction and outcome was $68 across these five cases.

RMSE. The squared errors are $100,\ 100,\ 100,\ 100,$ and $90{,}000$. Their sum is $90{,}400$, so the mean squared error is $90{,}400/5=18{,}080$ dollars squared. Taking the square root gives $\sqrt{18{,}080}=\$134.46$, rounded to the nearest cent.

Check and interpretation. The one $300 error contributes $90{,}000$ of the total $90{,}400$ squared dollars. That is why RMSE ($134.46) is much larger than MAE ($68). A careful report could say that MAE was $68 and RMSE was $134.46 on these five cases, with the larger RMSE driven mainly by one $300 underprediction. It should not claim that $134.46 is the error expected for every claim.

Translate the Metric Into a Decision Context

To make an error summary useful, connect it to a decision-relevant quantity or threshold that is defined independently of the observed result. For example, a planning team might state in advance that a deviation of $2,500 per claim is operationally material. The report can then show the error metric in dollars and explain how many validation cases fell within that threshold. The threshold does not change the metric; it provides context for interpreting it.

Be precise about the distinction between an average and a count of cases. “MAE was $2,000” does not tell the reader how many cases had an absolute error below $2,500. If that threshold matters, calculate and report the proportion directly. Even then, the proportion is a result for the evaluated cases, not a promise about future cases.

Worked Example: Relate Errors to an Operating Tolerance

A hypothetical monthly planning team validates four predictions of claim amounts, recording all values in thousands of dollars. The signed errors are $-2,\ 1,\ 3,\ -2$. The team’s pre-established tolerance is an absolute error no greater than $2,500 per claim. Calculate MAE and RMSE, and report how the observed errors compare with that tolerance.

Method. Since the errors are in thousands of dollars, an error of $2$ represents $2,000. The absolute errors are $2,\ 1,\ 3,\ 2$, whose sum is $8$. Therefore, $\operatorname{MAE}=8/4=2$, or $2,000.

For RMSE, the squared errors are $4,\ 1,\ 9,\ 4$. Their sum is $18$, so the mean squared error is $18/4=4.5$ in squared thousands of dollars. Thus, $\operatorname{RMSE}=\sqrt{4.5}=2.1213$ thousand dollars, or $2,121.32 when rounded to the nearest cent.

Compare with tolerance. The absolute errors are $2,000, $1,000, $3,000, and $2,000. Three of the four validation cases had absolute errors at or below $2,500; one did not. A suitable report would say: “MAE was $2,000 and RMSE was $2,121.32 per claim; three of four validation errors were within the $2,500 tolerance.” This describes the evaluated cases without implying that future errors will meet the tolerance at the same rate.

Common mistake avoided. It would be incorrect to describe the RMSE of $2,121.32 as “the average absolute error.” The MAE is the average absolute error here, $2,000. RMSE is a different summary, calculated by squaring, averaging, and taking a square root.

Compare Against a Relevant Baseline

A model’s error is easier to interpret when compared with a credible alternative, such as an existing process or a simple benchmark. Both methods must be evaluated on the same cases, for the same outcome, in the same units, using the same metric. Otherwise, an apparent improvement may reflect a change in evaluation conditions rather than better predictions.

State how the baseline was obtained. For example, a constant prediction might be the average outcome calculated from training data and then applied to every validation case. Do not calculate a benchmark using the validation outcomes if the intended comparison is a prediction that would have been available before those outcomes were known. A percentage reduction in error can summarize a comparison, but it does not replace the original values or explain whether the difference matters operationally.

Worked Example: Compare a Model With a Training-Set Baseline

A hypothetical team predicts quarterly claim amounts, in thousands of dollars. A baseline predicts $12,800 for every validation case; this value was calculated from the training data. On five validation cases, actual amounts are $8,\ 12,\ 16,\ 9,\ 25$, and the model predictions are $7,\ 14,\ 14,\ 10,\ 23$. Compare model and baseline MAE on these same cases, and interpret the result.

Model errors. Using actual minus predicted, the errors are $8-7=1$, $12-14=-2$, $16-14=2$, $9-10=-1$, and $25-23=2$. Their absolute values sum to $1+2+2+1+2=8$, so the model MAE is $8/5=1.6$ thousand dollars, or $1,600.

Baseline errors. Subtract the baseline prediction of $12.8$ from each actual value: $8-12.8=-4.8$, $12-12.8=-0.8$, $16-12.8=3.2$, $9-12.8=-3.8$, and $25-12.8=12.2$. The absolute errors sum to $4.8+0.8+3.2+3.8+12.2=24.8$. Baseline MAE is therefore $24.8/5=4.96$ thousand dollars, or $4,960.

Relative comparison. The reduction in MAE is $4.96-1.6=3.36$ thousand dollars. Relative to the baseline MAE, this is $3.36/4.96=0.677419\ldots$, or a 67.7% reduction when rounded to one decimal place.

Interpretation and check. On these five validation cases, the model’s MAE was $1,600, compared with $4,960 for the training-set baseline, a 67.7% reduction. Both errors were calculated on the same outcomes, so the comparison is consistent. This result supports describing the model as having lower MAE on this evaluation set; it does not establish that it will have lower error in every future period or on every individual claim.

Write the Claim at the Right Strength

A strong numeric-error statement usually includes four elements: the outcome and unit, the metric and value, the evaluation population or period, and a measured interpretation. For instance, “For the 240 validation claims, the model’s MAE was $1,600 per claim” is more informative than “the model was accurate.” If a comparison is material, state the baseline and the same metric’s result for it.

Avoid converting aggregate performance into unsupported statements about individual predictions. An average error does not identify which cases will have large errors, and a small average does not rule out a serious miss. If the report includes a threshold, call it a threshold and show the observed count or proportion meeting it; do not imply that the threshold is a guarantee.

The wording should also match the scale of the decision. A $1,000 error could be immaterial for one application and consequential for another. Give the unit and, when useful, compare the metric with a prespecified tolerance, a typical amount, or a relevant baseline. If no decision-relevant reference is available, report the number plainly rather than inventing a claim about its importance.

Key takeaway: Report numeric prediction errors with a stated sign convention, understandable units, and a relevant comparison. Explain what an average metric summarizes, and keep conclusions about the evaluated cases separate from promises about individual or future predictions.

Check Your Understanding

For each problem, interpret the result in the stated units and avoid treating a validation summary as a guarantee.

A hypothetical repair-cost model has signed errors of $20,\ -20,\ 40,\ -40$ dollars on four validation jobs. Calculate mean error and MAE. What does the mean error conceal? (Answer: Mean error $=\$0$; MAE $=\$30$. The zero mean conceals nonzero errors that cancel in sign.)
For four hypothetical reserve forecasts, errors in thousands of dollars are $1,\ -1,\ 1,\ -5$. Calculate RMSE and explain why it exceeds MAE. Round the RMSE to two decimals in thousands. (Answer: MAE $=2$ thousand dollars; RMSE $=\sqrt{7}=2.65$ thousand dollars. The $5$-thousand-dollar miss receives greater influence under squaring.)
A hypothetical model has MAE of $900 on 80 validation policies. A stated operating tolerance is $1,000, and 52 policies had absolute error no greater than that tolerance. Give two distinct performance statements and one limitation. (Answer: MAE was $900 per policy; $52/80=65\%$ of validation policies were within $1,000. Neither result guarantees that future policies will meet the tolerance.)
A model’s validation MAE is $3,000, while a baseline’s MAE is $4,000 on the same cases. Calculate the relative reduction in MAE and state the proper scope of the claim. (Answer: $(4{,}000-3{,}000)/4{,}000=25\%$. The model’s MAE was 25% lower on those validation cases; this does not establish lower error for every future prediction.)
A report says, “The average error was $15, so every prediction was within $15 of the outcome.” Explain the flaw, and state what extra summary would address the “every prediction” part for a specified validation set. (Answer: A signed mean error of $15 does not bound any individual error. Report the largest absolute error on that validation set to determine whether every evaluated error was within $15; that still would not guarantee future errors.)

What You'll Learn