Defining Success for Numeric Predictions

What Does a Useful Numeric Prediction Need to Do?

For a numeric prediction, “good” is not a complete success criterion. A forecast can be close on average yet miss an important threshold, or have a small dollar error that is large relative to a small target. In another setting, the same-sized overprediction and underprediction may have very different consequences. Before evaluating a model, specify what the prediction is for and what errors matter in that use.

This tutorial focuses on defining success, not selecting a model or comparing standard summary measures. The practical task is to state, in measurable terms, which prediction outcomes will be acceptable. A useful criterion identifies the prediction target, the scale on which error matters, an acceptable error size, and any difference in the consequences of overprediction and underprediction.

Definition: For an observed numeric outcome $y$ and its prediction $\hat{y}$, define the signed prediction error as $e=\hat{y}-y$. If $e>0$, the prediction is an overprediction; if $e<0$, it is an underprediction; and if $e=0$, the prediction equals the observation. The absolute error is $|e|$, the size of the discrepancy without its direction.

A criterion should be written before using the predictions to assess performance. Otherwise, it is tempting to choose a tolerance after seeing the errors, which can make a model appear successful by moving the standard to fit its results. Also state whether the threshold is inclusive. “Within 10 units” commonly means $|e|\leq 10$; “less than 10 units” means $|e|<10$. The distinction matters for a prediction exactly on the boundary.

Specify the Relevant Scale and Tolerance

A tolerance must use a scale meaningful for the decision. An absolute tolerance is a fixed amount of error, such as no more than 12 thousand dollars. It can suit a setting where a fixed discrepancy has a similar practical impact across the range of outcomes. A relative tolerance measures the discrepancy in relation to the observed value. It can be more useful when a 5-unit error is substantial for a target of 20 but minor for a target of 500.

For a positive observed value $y$, the absolute percentage error is $|\hat{y}-y|/y$. Its denominator is the observed value because the question is how large the discrepancy is relative to what actually occurred. A relative criterion needs special care when actual values are zero or close to zero: division by zero is undefined, and a small denominator can make a modest absolute discrepancy look enormous. In such settings, define an alternative rule, such as a fixed absolute tolerance for zero or near-zero outcomes, rather than applying a percentage rule mechanically.

A single tolerance may also hide important differences across ranges or groups. For example, a fixed 10-unit error could be reasonable for large monthly totals but unacceptable for small ones. The criterion can therefore specify different tolerances for different ranges or operational groups, provided those rules are defined in advance and can be applied consistently.

Worked Example: Applying an Inclusive Absolute Tolerance

A hypothetical transit agency forecasts monthly maintenance spending for five depots. Actual and predicted amounts, in thousands of dollars, are: Depot A, 40 and 35; Depot B, 60 and 67; Depot C, 80 and 70; Depot D, 100 and 112; and Depot E, 150 and 135. The agency defines an acceptable prediction as one with an absolute error of at most 12 thousand dollars. At least four of the five depots must meet the criterion. Determine which pass and whether the requirement is met.

Method. Calculate $e=\hat{y}-y$ for each depot, then check whether $|e|\leq 12$. The quantities are already in thousands of dollars, so the tolerance of 12 is on the same scale.

Depot A: $e=35-40=-5$, so $|e|=5\leq12$. It passes.

Depot B: $e=67-60=7$, so $|e|=7\leq12$. It passes.

Depot C: $e=70-80=-10$, so $|e|=10\leq12$. It passes.

Depot D: $e=112-100=12$, so $|e|=12\leq12$. It passes exactly at the inclusive boundary.

Depot E: $e=135-150=-15$, so $|e|=15>12$. It fails.

Four depots pass, meeting the requirement of at least four. The agency’s criterion is satisfied.

Common mistake. Excluding Depot D because its error equals 12 would wrongly treat “at most 12” as “less than 12.” The stated rule includes equality, so Depot D passes.

Worked Example: Using a Relative Tolerance

A hypothetical food distributor forecasts daily demand for five products. Actual and predicted demand, in cases, are: 50 and 54; 100 and 107; 250 and 230; 400 and 430; and 25 and 28. A product passes if its absolute percentage error is at most 8%, inclusive. At least four products must pass. Which products pass, and is the requirement met?

Method. For each product with positive actual demand, calculate $|\hat{y}-y|/y$, then compare the result with 0.08.

Product 1: The absolute error is $|54-50|=4$ cases. Relative to the actual demand, $4/50=0.08=8\%$. It passes at the inclusive limit.

Product 2: The absolute error is $|107-100|=7$ cases. The relative error is $7/100=0.07=7\%$. It passes.

Product 3: The absolute error is $|230-250|=20$ cases. The relative error is $20/250=0.08=8\%$. It passes at the inclusive limit.

Product 4: The absolute error is $|430-400|=30$ cases. The relative error is $30/400=0.075=7.5\%$. It passes.

Product 5: The absolute error is $|28-25|=3$ cases. The relative error is $3/25=0.12=12\%$. It fails.

Four of five products pass, so the distributor’s requirement is met.

Common mistake. Dividing by the prediction instead of the actual demand would give $3/28$ for Product 5. The stated relative-error rule uses the actual value in the denominator, so the correct calculation is $3/25=12\%$.

Account for the Direction of Error

An absolute tolerance treats an overprediction and an underprediction of the same size equally. That is appropriate only if the consequences are equal for the purpose at hand. If forecasting too low can cause costly shortages while forecasting too high mainly creates modest holding costs, the success rule should reflect that difference.

A simple way to express such a rule is to assign a cost per unit of underprediction and a cost per unit of overprediction. Suppose the costs are linear and additive: each unit of underprediction costs $c_{\text{under}}$, and each unit of overprediction costs $c_{\text{over}}$, where both are nonnegative. With $e=\hat{y}-y$, underprediction occurs when $e<0$, and its size is $-e$. Overprediction occurs when $e>0$, and its size is $e$. Thus the error cost is:

C(e)=c_{\text{under}}\max(-e,0)+c_{\text{over}}\max(e,0).

This expression follows directly from the two possible directions. If $e<0$, then $\max(-e,0)=-e$ and $\max(e,0)=0$, so $C(e)=c_{\text{under}}(-e)$. If $e>0$, then $\max(-e,0)=0$ and $\max(e,0)=e$, so $C(e)=c_{\text{over}}e$. If $e=0$, both terms are zero. The formula therefore charges only for the error direction that occurred, with the stated cost per unit.

This is a useful specification when those per-unit costs are credible over the range being considered. It does not apply unchanged if the consequences are nonlinear, include a threshold, or depend on other conditions. In those cases, the success criterion should state the relevant cost or operational rule directly rather than assume a constant cost per unit.

Worked Example: Choosing Between Forecasts With Unequal Error Costs

A hypothetical utility forecasts a seasonal reserve requirement. Values are measured in units of $10,000. The actual requirements for three seasons are 10, 20, and 30 units. Forecast A is 9, 22, and 29; Forecast B is 10.5, 19, and 31.5. For each unit of underprediction, the utility assigns a cost of $4,000; for each unit of overprediction, it assigns a cost of $1,000. The forecast must also be within 2 units of the actual requirement in every season. The total error cost must not exceed $7,000. Evaluate both forecasts.

Method. First calculate signed errors and check the absolute tolerance in each season. Then calculate direction-specific costs using the stated per-unit costs. Because the monetary costs are defined per unit of $10,000 in reserve, they are applied to the numeric errors in those units.

Forecast A: Its errors are $9-10=-1$, $22-20=2$, and $29-30=-1$. The absolute errors are 1, 2, and 1 units, each at most 2, so the forecast passes the season-by-season tolerance. The first and third errors are underpredictions, costing $1\times\$4,000=\$4,000$ each. The second is an overprediction, costing $2\times\$1,000=\$2,000$. Total cost is $\$4,000+\$2,000+\$4,000=\$10,000$. This exceeds $7,000, so Forecast A fails the total-cost requirement.

Forecast B: Its errors are $10.5-10=0.5$, $19-20=-1$, and $31.5-30=1.5$. The absolute errors are 0.5, 1, and 1.5 units, each at most 2, so it passes the season-by-season tolerance. The first and third errors are overpredictions, costing $0.5\times\$1,000=\$500$ and $1.5\times\$1,000=\$1,500$. The second is an underprediction, costing $1\times\$4,000=\$4,000$. Total cost is $\$500+\$4,000+\$1,500=\$6,000$. This is at most $7,000, so Forecast B meets both requirements.

Forecast B satisfies the defined success criteria; Forecast A does not. The result does not mean that overprediction is always preferable. It follows from the hypothetical costs and thresholds specified in this problem.

Common mistake. Applying the $4,000 underprediction cost to every error would ignore the different cost assigned to overprediction. Forecast A’s overprediction costs $2,000, not $8,000; Forecast B’s overpredictions cost $500 and $1,500, not their amounts multiplied by $4,000.

Turn the Purpose Into a Testable Rule

A practical success criterion should be clear enough that two analysts applying it to the same observations reach the same conclusion. It is useful to state the rule in operational terms, such as “at least 90% of predictions must be within 5 cases,” “no site may exceed a 10% relative error,” or “the total cost of prediction errors must be below the specified budget.” A threshold can be combined with a count or proportion requirement, but both parts must be stated.

Before evaluating predictions, check the following:

Target and units: What quantity is being predicted, and are observed and predicted values expressed in matching units?
Error scale: Does a fixed amount of error matter, or should the discrepancy be judged relative to the observed value?
Boundary: Is the limit inclusive or exclusive?
Direction: Do overprediction and underprediction have different consequences?
Aggregation rule: Must each prediction pass, must a specified proportion pass, or is there a total-cost limit?
Scope: Does the rule behave sensibly for small, large, zero, or unusual observed values?

A criterion should reflect the intended use, rather than whichever rule makes the reported model results look most favorable. If a fixed absolute tolerance, a relative tolerance, and a cost rule answer different practical questions, they should not be presented as interchangeable. The analyst should explain which one governs success and why.

Key takeaway: Define success for numeric predictions before assessing performance. Specify the relevant scale, the size and boundary of acceptable error, how many predictions must qualify, and whether the consequences of overprediction differ from those of underprediction.

Check Your Understanding

For each problem, apply the stated criterion exactly, including its units, denominator, and boundary rule.

A hypothetical repair service predicts weekly parts demand. Actual and predicted demand are 20 and 24, 30 and 25, and 50 and 55 units. A prediction passes when its absolute error is at most 5 units. At least two of the three must pass. Which pass, and is the requirement met? (Answer: All three pass: the absolute errors are 4, 5, and 5; the requirement is met.)
A hypothetical clinic predicts weekly appointment requests. Actual and predicted counts are 50 and 53, 120 and 112, and 200 and 188. A prediction passes if its absolute percentage error, calculated relative to actual requests, is at most 6%. How many pass? (Answer: Two pass: $3/50=6\%$ and $12/200=6\%$; $8/120\approx6.6667\%$ fails.)
A hypothetical warehouse forecasts inventory shortfall in units. For each unit of underprediction the cost is $3, while each unit of overprediction costs $1. Actual shortfall is 40 and 80. Forecast X is 35 and 90; Forecast Y is 45 and 75. Calculate each total cost and identify the lower-cost forecast. (Answer: Forecast X costs $5\times\$3+10\times\$1=\$25$; Forecast Y costs $5\times\$1+5\times\$3=\$20$; Forecast Y has the lower cost.)
A hypothetical water authority predicts daily demand for two districts. Actual and predicted amounts are 70 and 62, and 100 and 112 units. Each prediction must be within 12 units, inclusive, and no underprediction may exceed 6 units in absolute size. How many meet both conditions? (Answer: One of two. The first has absolute error 8 and underpredicts by 8, so it fails the underprediction condition; the second overpredicts by 12, meeting the inclusive tolerance and the underprediction condition.)
A hypothetical supplier predicts positive daily orders of 40, 80, and 160 items as 44, 74, and 172. The requirement is that at least two predictions have an absolute percentage error no greater than 10%, using actual orders as the denominator. Determine whether the requirement is met. (Answer: All three pass: $4/40=10\%$, $6/80=7.5\%$, and $12/160=7.5\%$; the requirement is met.)

What You'll Learn