Why Prediction Results Need Uncertainty Language
A reported prediction or performance score is an estimate based on data and a modelling process. It is not a guarantee about what will happen next. For example, a model may have an estimated area under the ROC curve (AUC) of 0.78 on a validation sample, but a different sample from the same target population would generally produce a somewhat different estimate. Separately, an individual predicted outcome can differ from the eventual outcome even if the model is well calibrated.
Clear reporting begins by identifying which quantity is uncertain. A validation metric may vary because the validation cases are a sample. An estimated event rate may vary because only some eligible cases were observed. An individual outcome may vary even when the model’s expected value is known. These are related, but they are not interchangeable forms of uncertainty.
This is aligned with the SOA Exam PA (Predictive Analytics) syllabus: a decision-maker should be able to understand both the reported result and the limits of what the available evidence supports. Earlier material on validation design and performance metrics helps identify the quantity being reported; the task here is to communicate its uncertainty accurately.
Identify the Quantity Before Describing Its Uncertainty
Before presenting an interval, say what it is an interval for. A confidence interval for a population performance metric concerns uncertainty in estimating that metric from a sample. A prediction interval for a future observation concerns the range of a new outcome under a predictive model. The latter is usually wider because it includes both uncertainty about the model’s expected value and variation in the outcome itself.
| Reported quantity | What may vary | What an interval can describe |
|---|---|---|
| Validation AUC or RMSE | Which validation cases are observed | Sampling variability in the estimated metric, under a stated resampling design |
| Event rate in a population | Which eligible cases are observed | Uncertainty in estimating the population event rate, under a sampling model |
| Expected outcome for a case | Data and fitted model used to estimate the expected value | Uncertainty about the estimated conditional mean |
| Outcome for a future case | Model estimation and the future outcome itself | A prediction interval, under assumptions about both sources of variation |
The scope matters. A resampling interval based on a validation set does not automatically account for changes in the population, changes in data collection, or decisions made after the model was developed. A prediction interval based on a specified outcome distribution does not guarantee coverage if that distribution or the model assumptions are wrong.
Worked Example: Reporting Uncertainty in a Validation AUC
A hypothetical service team evaluates a ranking model on a held-out sample and obtains an AUC of 0.78. The team resamples validation cases 1,000 times, recalculates AUC for each resample, and uses the 2.5th and 97.5th percentiles of the bootstrap results, 0.72 and 0.83, as a 95% percentile interval. A baseline model’s AUC is 0.74. A paired bootstrap of the difference, defined as candidate AUC minus baseline AUC on each resample, has a 95% interval from -0.01 to 0.09. What should the report say?
Method. Use the percentile bootstrap interval to describe resampling variability in the candidate model’s estimated AUC. Use the paired difference interval for the comparison, because each resample evaluates both models on the same resampled cases.
Candidate model. The point estimate is 0.78. The reported bootstrap endpoints are 0.72 and 0.83, so the interval’s width is \(0.83-0.72=0.11\). A suitable statement is: “The validation AUC was 0.78; the 95% bootstrap interval was 0.72 to 0.83, reflecting resampling variability among cases like those in this validation sample.”
Comparison. The point-estimate difference is \(0.78-0.74=0.04\). The paired interval runs from -0.01 to 0.09 and includes zero. The data therefore do not establish a positive AUC difference at the level represented by this interval. The estimate is consistent with a small decrease as well as with an increase.
What not to infer. The interval is not a guarantee that the model’s future AUC will be between 0.72 and 0.83. It also does not show that the candidate model is better simply because its point estimate exceeds the baseline’s. This interpretation assumes the resampling design is suitable for the validation cases and target population.
Common mistake avoided. Reporting “the candidate model improves AUC by 0.04” treats the point-estimate difference as certain. The paired difference interval makes the comparison’s uncertainty visible and avoids treating overlapping or separate model intervals as a substitute for an interval on the difference itself.
How an Event-Rate Interval Is Obtained
A simple interval for a proportion illustrates how sampling uncertainty enters a report. Suppose \(n\) independent observations each have the same probability \(p\) of an event, and let \(\hat{p}\) be the observed event proportion. Each observation can be represented by an indicator that is 1 if the event occurs and 0 otherwise. Its variance is \(p(1-p)\). The variance of the average of \(n\) independent indicators is therefore \(p(1-p)/n\), giving the standard error of the estimated proportion.
Replacing the unknown \(p\) in the standard error with the observed \(\hat{p}\) gives an estimated standard error. When the sample is large enough for a normal approximation to be reasonable, an approximate 95% interval is \(\hat{p}\) plus or minus 1.96 estimated standard errors. This is an approximate method, not a universal interval formula; with small samples or very rare events, the approximation can be poor and a different method may be needed.
Worked Example: Describing Uncertainty in a Segment Event Rate
A hypothetical operations team reviews 100 randomly selected eligible cases from one segment. Twelve cases had the event of interest. Assume the cases are independent and representative of a stable population with a common event probability. Calculate an approximate 95% interval for the segment event rate and state what it supports.
Step 1: Estimate the rate. There are 12 events among 100 cases, so \(\hat{p}=12/100=0.12\), or 12%.
Step 2: Calculate the estimated standard error.
The estimated standard error is about 0.0325, or 3.25 percentage points.
Step 3: Calculate the approximate 95% interval. The margin is \(1.96(0.0325)=0.0637\), approximately 6.37 percentage points. The lower endpoint is \(0.12-0.0637=0.0563\), and the upper endpoint is \(0.12+0.0637=0.1837\). The approximate interval is 5.63% to 18.37%.
Step 4: Interpret it. Under the stated sampling assumptions and normal approximation, this interval describes uncertainty in estimating the population event rate from the 100 cases. It does not mean that 95% of individual cases have event probabilities between these endpoints, nor does it guarantee that the next 100 cases will have a rate within the interval.
Common mistake avoided. Adding and subtracting 1.96 percentage points would confuse the multiplier with the standard error. The estimated standard error is 3.25 percentage points, so the margin is \(1.96 \times 3.25=6.37\) percentage points, not 1.96.
Expected Values and Future Outcomes Are Not the Same
For a numeric prediction, distinguish uncertainty about the model’s expected outcome for a case from variation in the outcome that case may actually produce. For example, a model may estimate an expected annual cost for a case. Even if that expected value were known exactly, an individual’s realized cost could differ because outcomes vary.
A simple model-based prediction interval can illustrate both components. Assume the future outcome for a case is normally distributed around its conditional mean, with residual standard deviation \(\sigma\). Suppose the estimate of that mean has standard error \(s_{\hat{\mu}}\), and assume the estimation error and future residual are independent and approximately normal. Then the variance of the prediction error is the sum of their variances:
The square root gives the prediction standard error. A 95% normal-based prediction interval is the estimated mean plus or minus 1.96 prediction standard errors. This formula depends on the stated normality and independence assumptions and on the estimated standard errors adequately representing their sources of variation. It does not apply unchanged when those assumptions or the prediction setting differ.
Worked Example: Separating Mean Uncertainty From Outcome Variation
A hypothetical model predicts an annual cost of $1,200 for a particular case. Assume the future cost, conditional on the case information, is normally distributed with residual standard deviation \(\sigma=\)$400. The estimated conditional mean has standard error \(s_{\hat{\mu}}=\)$60. Assume these two sources of error are independent and approximately normal. Find an approximate 95% interval for the conditional mean and an approximate 95% prediction interval for one future cost.
Method. A confidence interval for the conditional mean uses the standard error of that estimated mean. A prediction interval for a future outcome also includes the residual variation of an individual outcome.
Conditional mean interval. Its endpoints are \(1200 \pm 1.96(60)\). The margin is \(1.96 \times 60=\)$117.60. Therefore, the approximate interval for the conditional mean is $1,082.40 to $1,317.60.
Prediction standard error. The variance is \(400^2+60^2=160{,}000+3{,}600=163{,}600\). Its square root is approximately $404.475.
Prediction interval. The margin is \(1.96(404.475)\approx\)$792.771. The lower endpoint is \(1200-792.771=\)$407.229, and the upper endpoint is \(1200+792.771=\)$1,992.771. Rounded to the nearest cent at the final step, the approximate prediction interval is $407.23 to $1,992.77.
Interpretation. The conditional mean interval describes uncertainty about the estimated average cost for cases with these characteristics. The wider prediction interval describes the range of one future cost under the assumed model. Neither is a guarantee for the individual case.
Common mistake avoided. Using only the $60 standard error to construct a prediction interval would give the conditional-mean interval, not an interval for a future cost. The prediction interval must include the $400 residual standard deviation as well.
Write Uncertainty So Readers Can Use It
An uncertainty statement is most useful when it names the estimate, the interval, and the evidence behind it. Include enough context for the reader to understand the target population, validation design, or model assumptions that matter to interpretation. Avoid presenting uncertainty as a decorative range with no explanation.
Say whether the interval concerns a validation metric, an event rate, an expected value, or a future outcome.
Identify the validation sample, resampling procedure, or probability model used to obtain the estimate and interval.
Describe the kind of variation the interval represents and avoid implying that it covers sources of uncertainty it did not include.
Explain what the estimate does not establish, such as guaranteed future performance or a certain improvement over a baseline.
Uncertainty also affects comparisons and decisions. If the evidence does not distinguish two models clearly, report that fact rather than presenting a small difference as decisive. If a prediction interval is broad relative to the decision threshold, that breadth may matter more than the point prediction. In either case, explain the implication without claiming more than the available evidence supports.
- Do not call an estimate a guarantee. A strong validation result is evidence about performance under a specified evaluation design, not certainty about every future case.
- Do not confuse interval targets. An interval for a mean or a performance metric is not an interval for an individual outcome.
- Do not treat a method’s confidence level as a promise. A nominal 95% interval is tied to its assumptions and procedure; it does not ensure that this particular interval contains the unknown value.
- Do not omit material scope. If a resampling analysis reflects only variation among validation cases, say so instead of implying that all possible future changes have been captured.
Check Your Understanding
For each problem, identify the uncertainty target and give an appropriately limited interpretation.
- A hypothetical validation set contains 200 cases, of which 40 have an event. Using the approximate normal interval for a proportion, calculate the estimated rate, standard error, and 95% interval. (Answer: Rate 0.20; SE \(\sqrt{0.20(0.80)/200}\approx0.0283\); interval \(0.20\pm1.96(0.0283)\), or approximately 0.1445 to 0.2555.)
- A model has a validation RMSE of $520, and a case-resampling bootstrap gives a 95% interval of $470 to $590. What uncertainty does this interval describe, and does it guarantee future RMSE is in that range? (Answer: It describes resampling variability in the estimated validation RMSE under the stated bootstrap design; no, it does not guarantee future RMSE lies in the range.)
- A model predicts a conditional mean of $800. Assume normal future outcomes with residual standard deviation $150, negligible uncertainty in the estimated mean, and the stated distribution is appropriate. Calculate an approximate 95% prediction interval. (Answer: \(800\pm1.96(150)=800\pm294\), or $506 to $1,094.)
- Two models have AUC estimates of 0.81 and 0.79. A paired bootstrap interval for their AUC difference, first minus second, is -0.02 to 0.06. What should the comparison report avoid claiming? (Answer: It should avoid claiming that the first model is demonstrably better; the interval includes zero and is compatible with a small negative or positive difference.)
- An individual prediction report gives a 95% interval for the conditional mean of a future claim cost but labels it a “95% range for the claim.” Explain the problem. (Answer: The interval concerns uncertainty in the estimated mean, not the variation of one claim; a future-claim prediction interval must also account for outcome variability.)