Evaluating Time-Series Prediction Intervals

Why Evaluate Prediction Intervals?

A time-series forecast is often reported as a range rather than as a single predicted value. That range is useful only if its uncertainty is credible: an interval that is very narrow but often misses the future observation can be misleading, while an extremely wide interval may contain nearly everything but provide little practical information. Evaluating prediction intervals therefore requires looking at coverage and width together.

For a sequence of forecasts made at successive forecast origins, each interval must be compared with the actual observation it was intended to cover. The observations are kept in time order, and the forecast information available at each origin is the information that should have been used to produce that interval. This evaluation is aligned with the SOA Exam SRM (Statistics for Risk Modeling) syllabus.

Definition: For test observation $y_t$, let $[L_t,U_t]$ be its prediction interval, where $L_t$ and $U_t$ are the lower and upper endpoints and $L_t\leq U_t$. The observation is covered if $L_t\leq y_t\leq U_t$. For $n$ time-ordered test observations, empirical coverage is the proportion covered: $$ \widehat{C}=\frac{1}{n}\sum_{t=1}^{n} I(L_t\leq y_t\leq U_t), $$ where $I(\cdot)$ equals 1 when its condition is true and 0 otherwise. The average width is $$ \overline{W}=\frac{1}{n}\sum_{t=1}^{n}(U_t-L_t). $$

The width is measured in the units of the response variable. If forecasts concern monthly claim counts, width is in claims; if they concern a dollar-valued series, width is in dollars. Compare widths meaningfully only when the models use the same response scale and evaluation observations.

Nominal Coverage and What Calibration Means

A prediction interval is often constructed for a stated nominal level, such as 80%, 90%, or 95%. Nominal coverage is the target probability under the model and its assumptions; empirical coverage is the fraction of test observations actually inside the intervals. For example, a 90% interval is not a promise that every particular batch of ten observations will contain exactly nine covered values. It states a probability target, and the realized coverage can differ, especially in a short test period.

A useful justification comes from the conditional distribution of the future observation. Suppose that, at forecast origin $t$, the model’s conditional distribution for $Y_t$, given the information available then, is $F_t$. If the interval endpoints are conditional quantiles at probabilities $\alpha/2$ and $1-\alpha/2$, and the distribution is continuous, then

P(L_t\leq Y_t\leq U_t\mid\mathcal{F}_{t-1}) =F_t(U_t)-F_t(L_t) =(1-\alpha/2)-(\alpha/2) =1-\alpha.

Here $\mathcal{F}_{t-1}$ denotes the information available when the forecast is made. The first equality uses continuity, so the probability at a single endpoint is zero. Under these conditions, the interval has its stated conditional coverage. If the conditional distribution is misspecified, its parameters are poorly estimated, or the process changes, the realized coverage can depart from the nominal level. For discrete outcomes, quantile conventions and probability mass at endpoints mean the exact nominal probability may not be attainable; the interval construction’s endpoint rule must be respected.

An overall coverage value is a summary across the test set. It does not, by itself, establish that coverage is correct at every date or forecast horizon. In particular, a series can have reasonable overall coverage while missing disproportionately during a particular regime or at longer horizons. Keep the chronological order in view and, when relevant, summarize performance separately by horizon.

Worked Example: Coverage and Width for One Test Sequence

A hypothetical model produces five one-step-ahead intervals for a test sequence. The actual observations and intervals are: 20 with $[18,22]$; 25 with $[23,27]$; 28 with $[26,30]$; 31 with $[28,32]$; and 40 with $[35,39]$. Calculate empirical coverage and average width. The stated nominal level is 90%.

Method. Check each actual against its corresponding interval using $L_t\leq y_t\leq U_t$, then calculate the widths and average them.

For the first observation, $18\leq20\leq22$, so it is covered. For the second, $23\leq25\leq27$, so it is covered. For the third, $26\leq28\leq30$, so it is covered. For the fourth, $28\leq31\leq32$, so it is covered. For the fifth, 40 is above the upper endpoint 39, so it is not covered. There are four covered observations out of five:

\widehat{C}=\frac{4}{5}=0.8000=80.00\%.

The five widths are $22-18=4$, $27-23=4$, $30-26=4$, $32-28=4$, and $39-35=4$. Therefore,

\overline{W}=\frac{4+4+4+4+4}{5}=\frac{20}{5}=4.

The test coverage is 80.00%, which is below the 90% nominal target, and the average width is 4 response units. With only five observations, this difference is a descriptive result, not conclusive evidence that the model is miscalibrated. Common mistake avoided: the third actual, 28, is inside its interval $[26,30]$, so it counts as covered; do not confuse an observation merely near an endpoint with one outside it.

Coverage and Width Must Be Read Together

Coverage alone rewards intervals for being wide: an interval expanded far enough can cover almost every observation. Width alone rewards intervals for being narrow: very tight intervals can have poor coverage. A practical assessment reports both and compares coverage with the stated nominal level. The comparison does not create a universal score that settles every decision; it makes the tradeoff explicit.

Observed pattern	Interpretation to investigate
Coverage near nominal, relatively narrow intervals	A potentially useful balance of uncertainty calibration and precision on this test set
Coverage well below nominal, narrow intervals	Intervals may be too optimistic or the predictive distribution may not represent the test data
Coverage well above nominal, very wide intervals	Intervals may be conservative but less informative than necessary
Similar overall coverage but different horizon-specific results	Overall performance may conceal a weakness at particular forecast horizons

“Near nominal” is not a fixed pass/fail rule: its interpretation depends on the size and nature of the test set. A small test sample can produce a coverage proportion far from the target even when the underlying predictive procedure is well calibrated. Conversely, a proportion near nominal in a small sample does not prove that the procedure is reliable. This tutorial’s summaries describe the observed test performance; they are not, on their own, formal uncertainty statements about differences between forecasting methods.

Worked Example: A Narrow Interval That Under-Covers

Two hypothetical forecasting procedures are evaluated at the same eight successive origins. The actual values are 10, 12, 14, 16, 18, 20, 22, and 24. Their 90% prediction intervals are listed in the same order:

Procedure A: $[9,11]$, $[11,13]$, $[11,13]$, $[15,17]$, $[17,19]$, $[17,19]$, $[21,23]$, and $[22,24]$.
Procedure B: $[8,12]$, $[10,14]$, $[12,16]$, $[14,18]$, $[16,20]$, $[18,22]$, $[20,24]$, and $[27,31]$.

Method. Count hits for each procedure, calculate each interval’s width, and compare both summaries with the 90% nominal target.

For Procedure A, the first actual, 10, is covered; 12 is covered; 14 is not in $[11,13]$; 16 is covered; 18 is covered; 20 is not in $[17,19]$; 22 is covered; and 24 is covered. Thus there are six hits:

\widehat{C}_A=\frac{6}{8}=0.7500=75.00\%.

Every A interval has width 2, so its average width is $(8\times2)/8=2$.

For Procedure B, the first seven actuals are covered by their respective intervals: 10 is in $[8,12]$, 12 in $[10,14]$, 14 in $[12,16]$, 16 in $[14,18]$, 18 in $[16,20]$, 20 in $[18,22]$, and 22 in $[20,24]$. The final actual, 24, is not in $[27,31]$. Therefore,

\widehat{C}_B=\frac{7}{8}=0.8750=87.50\%.

Each B interval has width 4 except the final interval, whose width is also $31-27=4$. Thus the average width is $(8\times4)/8=4$. Procedure B has coverage closer to the 90% target in this test sequence, while Procedure A is narrower but covers only 75% of the actuals. The result does not establish that B is generally superior: it summarizes only these eight observations. Common mistake avoided: selecting A solely because its intervals are narrower would ignore that two of its actuals fall outside the intervals.

Evaluate Each Forecast Horizon

The width of a prediction interval often changes with the forecast horizon. For example, uncertainty may accumulate as the forecast extends further from the origin. Combining horizons into a single average can obscure this pattern, so compute coverage and width separately for each horizon when the test design includes multiple horizons. Match every actual with the interval issued for that same origin and horizon.

Worked Example: Comparing One-Step and Three-Step Intervals

A hypothetical time-series model is evaluated at three successive forecast origins, producing one-step and three-step forecasts at each origin. For the one-step horizon, the actuals and intervals are 50 with $[48,52]$, 52 with $[50,54]$, and 55 with $[51,55]$. For the three-step horizon, they are 54 with $[49,53]$, 60 with $[56,64]$, and 63 with $[59,65]$. Calculate coverage and average width by horizon, then calculate the combined summaries. The nominal level is 90%.

Method. Evaluate the three intervals within each horizon first. Use the inclusive endpoint rule: an actual equal to either endpoint is covered.

At one step, 50 is inside $[48,52]$. The actual 52 is the upper endpoint of $[50,54]$, so it is covered. The actual 55 is the upper endpoint of $[51,55]$, so it is also covered. One-step coverage is $3/3=1.0000$, or 100.00%. The widths are $52-48=4$, $54-50=4$, and $55-51=4$, giving average width $(4+4+4)/3=4$.

At three steps, 54 is above the upper endpoint 53 of $[49,53]$, so it is not covered. The actual 60 is inside $[56,64]$, and 63 is inside $[59,65]$, so two of the three are covered:

\widehat{C}_{3\text{-step}}=\frac{2}{3}=0.6667=66.67\%.

The three-step widths are $53-49=4$, $64-56=8$, and $65-59=6$. Their average is $(4+8+6)/3=18/3=6$.

Across both horizons, there are five covered observations among six, so combined coverage is $5/6=0.8333$, or 83.33%. The six widths sum to $4+4+4+4+8+6=30$, giving combined average width $30/6=5$. The aggregate summaries conceal the lower three-step coverage, so the horizon-specific results are important. Common mistake avoided: excluding the one-step actuals that equal interval endpoints would incorrectly discard valid hits; with the stated inclusive convention, all three one-step observations are covered.

Practical Checks and Common Traps

Before interpreting the summaries, verify that each interval is paired with the correct test observation and forecast horizon, and that the forecasts were made using information available at their origins. Then check the following:

Use the stated endpoint convention. For a closed interval $[L_t,U_t]$, equality at either endpoint counts as coverage.
Do not report coverage without a denominator. State the count or proportion of hits and the number of test observations.
Do not compare widths across different response scales. A width in dollars is not directly comparable with one in log dollars.
Do not treat a single observed proportion as a guarantee. Test-set coverage can vary, particularly when the test set is short.
Do not let overall coverage hide horizon-specific failures. Calculate separate summaries when performance may differ by horizon.

The central interpretation is deliberately two-sided: coverage asks whether the stated uncertainty is being respected on the test sequence, while average width describes how much uncertainty the intervals report. A narrow interval is valuable only when its coverage is adequate for the intended use. These summaries support a careful assessment; neither one alone is enough.

Key takeaway: On time-ordered test observations, compute empirical coverage with an explicit endpoint rule and average the interval widths. Compare coverage with the nominal target, interpret width in context, and inspect results by forecast horizon before drawing conclusions.

Check Your Understanding

For each problem, use the stated interval endpoints and keep coverage and width as separate summaries.

A model forecasts five test values. The actuals and intervals are 7 with $[6,8]$, 9 with $[8,10]$, 12 with $[10,13]$, 15 with $[12,14]$, and 14 with $[13,15]$. Find empirical coverage and average width. (Answer: coverage $4/5=0.8000$, or 80.00%; average width $11/5=2.2$ units.)
A 95% interval procedure covers 9 of 10 time-ordered test observations. All ten intervals have width 3. Give the empirical coverage and average width, and compare the coverage with nominal. (Answer: coverage $9/10=0.9000$, or 90.00%; average width 3 units; observed coverage is 5 percentage points below nominal.)
At horizon 1, five intervals have widths 2, 4, 2, 4, and 3, and four observations are covered. At horizon 4, five intervals have widths 8, 8, 10, 6, and 8, and three observations are covered. Calculate coverage and average width for each horizon and in aggregate. (Answer: horizon 1 coverage $4/5=80.00\%$, width 3; horizon 4 coverage $3/5=60.00\%$, width 8; aggregate coverage $7/10=70.00\%$, width $55/10=5.5$ units.)
Two procedures target 90% coverage on the same ten test observations. Procedure A covers 7 and has average width 2; Procedure B covers 9 and has average width 6. Which has coverage closer to nominal, and what tradeoff remains? (Answer: B is closer to nominal, with 90% coverage versus A’s 70%; A is narrower, so coverage and width show a tradeoff rather than a universal winner.)
Four actuals are 4, 7, 9, and 12, with intervals $[3,4]$, $[6,8]$, $[10,12]$, and $[12,14]$, respectively. Using closed intervals, calculate coverage and average width. (Answer: coverage $3/4=0.7500$, or 75.00%; average width $(1+2+2+2)/4=1.75$ units.)

What You'll Learn