Choose Criteria for a Numeric Prediction Case

Start With the Consequence of a Prediction Error

For a numeric prediction, a useful success criterion answers more than “How close are the predictions?” It specifies what counts as close, which errors matter most, and how performance will be summarized across cases. A criterion that does not reflect the decision being supported can reward a model that looks accurate on average but performs poorly where errors are costly.

Suppose a hypothetical utility plans daily staffing using forecasts of customer demand. A difference of four customers may be acceptable on an ordinary day, while a miss of ten customers may trigger expensive emergency staffing. In another setting, overestimating a need may be costly, but underestimating it may create a safety risk. Those differences should shape the criterion before candidate models are compared.

Definition: For observed numeric outcome $y_i$ and prediction $\hat{y}_i$ on case $i$, define the signed prediction error as $e_i=y_i-\hat{y}_i$. Thus, $e_i>0$ means underprediction and $e_i<0$ means overprediction. A success criterion is a stated rule for summarizing prediction performance, including the outcome scale, aggregation, and any cost or tolerance requirements.

The sign convention matters whenever the direction of error has consequences. For example, a positive error in a demand forecast means predicted demand was below observed demand. Always state the convention before interpreting signed errors; reversing the subtraction reverses the meaning of the sign.

Match the Error Scale and Summary to the Requirement

A criterion should use a scale that stakeholders can interpret. If the decision is made in dollars, minutes, or units of demand, errors on that original scale are often directly meaningful. Relative errors may be more useful when the same absolute miss has different significance for small and large cases, but they can be unstable or undefined when observed values are near or equal to zero. A transformed scale is appropriate only when the decision and interpretation support it; it should not be chosen merely because it produces a more favorable score.

Two familiar summaries of errors are mean absolute error (MAE) and root mean squared error (RMSE). For $n$ evaluated cases, they are

\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|e_i|, \qquad \operatorname{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}e_i^2}.

Both are expressed in the units of the outcome. MAE averages the magnitude of errors, while RMSE gives more influence to large errors through squaring. Neither is universally preferable: the business consequences determine whether that extra emphasis is useful. A signed average error, $\frac{1}{n}\sum_i e_i$, can indicate a pattern of underprediction or overprediction, but positive and negative errors can cancel. It should not be treated as a measure of typical error by itself.

If a business requirement is stated as a tolerance, evaluate that requirement directly as well. For example, the proportion of predictions within $T$ units is

\frac{1}{n}\sum_{i=1}^{n} I(|e_i|\leq T),

where $I(\cdot)$ equals 1 when the condition is true and 0 otherwise. The boundary convention matters: this expression counts an error exactly equal to $T$ as within tolerance. If the requirement instead concerns errors greater than a threshold, state whether “greater than” is strict.

Justification: RMSE is the square root of the average squared error. Squaring makes each absolute error $a$ contribute $a^2$, so a larger error contributes more than a smaller one in proportion to its square. For $a>b\geq0$, $a^2>b^2$. With $b$ fixed, the difference $a^2-b^2$ increases as $a$ increases; if both errors are scaled by the same positive factor $c$, their squared-error difference becomes $c^2(a^2-b^2)$. However, the difference need not increase when both errors vary in unrelated ways. Taking the square root restores the outcome’s units and does not change which model has the smaller mean squared error.

Work Through the Business Requirement

A practical selection process makes the requirement explicit before settling on a headline score. Begin with the decision, then identify the scale, error direction, and aggregation that match it. If stakeholders name a specific costly event, include a direct measure of that event rather than assuming a general average will represent it.

Name the decision.
State what action uses the prediction and what can go wrong when it is too high or too low.

Choose the scale.
Use outcome units when unit differences have stable meaning; use a relative or transformed scale only when it matches the decision and is well-defined for the cases being evaluated.

Specify the error pattern of concern.
Decide whether typical error, unusually large error, directional error, or threshold violations are consequential.

State the aggregation and guardrails.
Specify how cases are combined and add a direct threshold, cost, or directional measure when a single average would conceal a material risk.

Worked Applications

Worked Example: Demand Forecasts With Costly Large Misses

A hypothetical distribution center compares two forecasts for daily package demand. The observed demands are 100, 120, 140, and 160 packages. Model A predicts 98, 118, 138, and 150; Model B predicts 96, 116, 136, and 156. Underforecasting incurs a base cost of $8 per missing package. In addition, any daily underforecast greater than 5 packages incurs a $100 escalation cost. Which criterion best reflects this requirement, and which model has the lower stated cost?

Method. Define $e_i=y_i-\hat{y}_i$, calculate the two average error summaries, and then apply the stated business cost. All errors are underforecasts, so all $e_i$ values are positive.

For Model A, the errors are $100-98=2$, $120-118=2$, $140-138=2$, and $160-150=10$. Their absolute values sum to $2+2+2+10=16$, so $\operatorname{MAE}_A=16/4=4$ packages. Their squares sum to $4+4+4+100=112$, giving $\operatorname{RMSE}_A=\sqrt{112/4}=\sqrt{28}\approx5.2915$ packages.

For Model B, the errors are $100-96=4$, $120-116=4$, $140-136=4$, and $160-156=4$. Their absolute values sum to $16$, so $\operatorname{MAE}_B=16/4=4$ packages. Their squares sum to $4(4^2)=64$, so $\operatorname{RMSE}_B=\sqrt{64/4}=4$ packages.

Both models have the same total underforecast, $16$ packages, and the same base cost: $16\times \$8=\$128$. Model A has one daily error greater than 5 packages, so its escalation cost is $\$100$; Model B has none. Total stated costs are therefore $\$128+\$100=\$228$ for A and $\$128+\$0=\$128$ for B. The direct cost criterion selects B. RMSE also favors B, while MAE ties them; the threshold count and cost calculation show why the tie in MAE is not enough.

Check. Model A’s one error of 10 packages is the only error exceeding 5; Model B’s four errors are each 4. The calculated escalation costs match those counts. Common mistake avoided: choosing solely by MAE would report a tie and ignore the explicitly costly large miss.

Worked Example: Unequal Costs for Underprediction and Overprediction

A hypothetical equipment service team predicts repair costs for three jobs. Actual costs are $100, $200, and $300. Model A predicts $120, $180, and $280; Model B predicts $80, $220, and $320. Underprediction costs $4 per dollar of error, while overprediction costs $1 per dollar of error. Compare MAE and total asymmetric cost.

Method. Use $e_i=y_i-\hat{y}_i$, so positive errors are underpredictions. For each case, charge $4\max(e_i,0)+\max(-e_i,0)$ dollars. This applies the higher rate only to underprediction and the lower rate only to overprediction.

For Model A, the signed errors are $100-120=-20$, $200-180=20$, and $300-280=20$. The absolute errors sum to $20+20+20=60$, so MAE is $\$60/3=\$20$. The first case is an overprediction costing $20\times \$1=\$20$. The next two are underpredictions costing $20\times \$4=\$80$ each. Total cost is $\$20+\$80+\$80=\$180$.

For Model B, the signed errors are $100-80=20$, $200-220=-20$, and $300-320=-20$. Again, the absolute errors sum to $60$, so MAE is $\$20$. The first case costs $20\times \$4=\$80$, and the two overpredictions cost $20\times \$1=\$20$ each. Total cost is $\$80+\$20+\$20=\$120$.

The models tie on MAE, but Model B has the lower cost under the stated rates. Check. Each model has one error in one direction and two in the other; because underprediction has the higher rate, the direction attached to the single error changes the total. Common mistake avoided: treating equal absolute errors as equal business impact when the costs are asymmetric.

Worked Example: A Tolerance and a Directional Guardrail

A hypothetical clinic uses predicted appointment durations to plan room schedules. For four appointments, actual durations are 30, 60, 90, and 120 minutes; predictions are 25, 70, 110, and 100 minutes. The clinic wants predictions within 15 minutes and is especially concerned about underpredictions greater than 15 minutes, which may leave the next patient waiting. Calculate MAE, RMSE, the proportion within 15 minutes, and the proportion with an underprediction greater than 15 minutes.

Method. Use $e_i=y_i-\hat{y}_i$, and count a prediction as within tolerance when $|e_i|\leq15$. Count the directional event only when $e_i>15$.

The signed errors are $30-25=5$, $60-70=-10$, $90-110=-20$, and $120-100=20$ minutes. The absolute errors are $5,10,20,20$, so MAE is $(5+10+20+20)/4=55/4=13.75$ minutes. Squared errors sum to $25+100+400+400=925$, so RMSE is $\sqrt{925/4}=\sqrt{231.25}\approx15.2069$ minutes.

Only the first two absolute errors are at most 15 minutes. The within-tolerance proportion is $2/4=0.5$, or 50%. Only the fourth error is an underprediction greater than 15 minutes, since $e_4=20$; the third error is an overprediction. Thus the directional proportion is $1/4=0.25$, or 25%.

Check. The signed errors sum to $5-10-20+20=-5$, so the mean signed error is $-5/4=-1.25$ minutes, indicating slight average overprediction under this convention. That average does not replace the directional guardrail: it combines over- and underpredictions. Common mistake avoided: counting an overprediction as a consequential underprediction because its absolute error exceeds the tolerance.

State the Criterion So It Can Be Applied Consistently

A clear success statement identifies the evaluation cases, outcome scale, summary, and any additional business condition. For instance: “On the specified evaluation cases, report MAE in minutes, and require at least 90% of predictions to be within 10 minutes; separately report the proportion of underpredictions exceeding 10 minutes.” This distinguishes a typical-error summary from a tolerance requirement and a directional risk measure.

When costs per unit and error direction are known, a cost-based criterion can directly represent them. A simple per-case asymmetric cost is $c(e)=c_u\max(e,0)+c_o\max(-e,0)$, where $c_u$ is the cost per unit of underprediction and $c_o$ is the cost per unit of overprediction. The formula assumes costs are linear within each direction. If costs change at thresholds or depend on other conditions, the cost rule must state those features explicitly; a linear formula would not represent them.

Business concern	Useful criterion to consider	Important limitation
Typical miss in meaningful outcome units	MAE	Does not emphasize the largest errors as strongly as RMSE
Large misses have disproportionate consequences	RMSE or an explicit large-error cost	RMSE may emphasize large errors without matching their actual financial cost
Over- and underprediction have different consequences	Signed directional rates or an asymmetric cost function	A signed average can hide offsetting errors
A specified accuracy tolerance matters operationally	Proportion within tolerance and relevant exceedance rates	A single average does not show how often the tolerance is breached

A common exam trap is to choose a metric because it is familiar, then justify it after seeing model results. Instead, translate the requirement into a measurable rule first. Also check units and direction: a dollar error is not interchangeable with a percentage error, and the sign of $e_i$ has a meaning only after its definition is stated. These choices are aligned with the SOA Exam PA (Predictive Analytics) syllabus.

Key takeaway: Select numeric prediction criteria by connecting the decision to the error scale, the cost of large misses, and the consequences of each error direction. Use a direct threshold or cost measure when an average alone could conceal a business-critical failure.

Check Your Understanding

For each problem, use $e_i=y_i-\hat{y}_i$ unless the question states otherwise.

A parts planner observes demands of 50, 70, and 90 units and predicts 46, 75, and 80. Calculate MAE and mean signed error, and interpret the sign of the latter. (Answer: Absolute errors are 4, 5, and 10, so MAE is $19/3\approx6.33$ units; signed errors are 4, $-5$, and 10, so the mean signed error is $9/3=3$ units, indicating average underprediction.)
Two models have error magnitudes A: 1, 1, 7 and B: 3, 3, 3. Calculate each MAE and RMSE. Which has lower RMSE? (Answer: Both MAEs are 3; RMSE for A is $\sqrt{51/3}=\sqrt{17}\approx4.12$, and for B is $\sqrt{27/3}=3$, so B has lower RMSE.)
Actual costs are $40, $80, and $120; predictions are $35, $90, and $115. Underprediction costs $5 per dollar of error and overprediction costs $2 per dollar. Find MAE and total asymmetric cost. (Answer: Errors are 5, $-10$, and 5; MAE is $20/3\approx\$6.67$, and total cost is $10\times\$5+10\times\$2=\$70$.)
For four forecasts, signed errors are 10, $-5$, 15, and $-10$ units. What proportion are underpredictions greater than 10 units? Use a strict greater-than condition. (Answer: Only the error of 15 qualifies, so the proportion is $1/4=0.25$, or 25%.)
Actual losses are $100 and $1,000, and predictions are $80 and $1,100. Underprediction costs $3 per dollar and overprediction costs $1 per dollar. Calculate MAE and total asymmetric cost. (Answer: Errors are 20 and $-100$; MAE is $(20+100)/2=\$60$, and total asymmetric cost is $20\times\$3+100\times\$1=\$160$.)

What You'll Learn