Evaluating Volatility and Density Forecasts

Why Forecast Evaluation Requires More Than a Point Error

A volatility forecast and a density forecast make different claims. A volatility forecast predicts the scale of future variation; a density forecast describes a full distribution of possible outcomes. Evaluating either forecast requires outcomes observed after the forecast was issued, a loss or diagnostic suited to the forecast, and care about what the result does and does not establish.

For a one-step-ahead return \(y_t\), let \(h_t\) denote a forecast of its conditional variance, made using information available before \(y_t\) is observed. Let \(RV_t\) denote a realized-variance measure for the same forecast period, used as an observed proxy for latent variance. For a density forecast, let \(f_t(y)\) and \(F_t(y)\) denote the forecast density and cumulative distribution function for \(y_t\), respectively. Evaluation should be out of sample: the forecast for period \(t\) must not use information that became available only at or after \(t\).

Definition: A volatility forecast evaluation compares a forecast variance \(h_t>0\) with a realized-variance measure \(RV_t\) using a stated loss function. A density forecast evaluation assesses the forecast distribution through, for example, the log density value \( \log f_t(y_t) \) at the outcome that occurs, or through calibration diagnostics based on \(F_t(y_t)\).

The distinction between a density and a probability matters. For a continuous distribution, the probability assigned to any one exact outcome is zero. A log score therefore uses the density value at the observed outcome, not the probability of that exact value. Density values can exceed one, depending on the measurement units; their logarithms remain valid scores when forecasts are compared for the same outcome variable and units.

Losses for Volatility Forecasts

A common starting point is squared error, \( (RV_t-h_t)^2 \), or absolute error, \( |RV_t-h_t| \). Both require forecasts and realized variance to be expressed on the same scale. Squared error penalizes large misses more heavily than absolute error. Neither loss is automatically best for every application, so the loss and its rationale should be stated before comparing models.

A widely used alternative for positive variance forecasts is QLIKE. In a normalized form, its period-\(t\) loss is \(RV_t/h_t-\log(RV_t/h_t)-1\). This form requires \(RV_t>0\) and \(h_t>0\). The conventional form \( \log h_t+RV_t/h_t \) gives the same ranking of forecasts for a given \(RV_t\), because it differs from the normalized form only by a term that depends on \(RV_t\), not on \(h_t\).

L_{\mathrm{QLIKE},t} = \frac{RV_t}{h_t} - \log\left(\frac{RV_t}{h_t}\right) - 1

QLIKE is nonnegative: for \(x>0\), \(x-\log x-1\geq 0\), with equality only at \(x=1\). Thus the loss is zero when the forecast equals the realized-variance measure. A lower average loss indicates better performance on the chosen sample and loss function; it does not establish that a forecast is best under every criterion or for every use.

Worked Example: Comparing QLIKE Losses

Suppose two hypothetical models forecast daily variance for three days. The realized-variance proxies are \(0.011\), \(0.009\), and \(0.005\). Model A forecasts \(0.012\), \(0.008\), and \(0.006\); Model B forecasts \(0.013\), \(0.007\), and \(0.006\). All values are in squared-return units and are positive, so the normalized QLIKE formula applies.

For Model A on day 1, the ratio is \(0.011/0.012=0.9167\), so \(L=0.9167-\log(0.9167)-1=0.003678\). On day 2, the ratio is \(0.009/0.008=1.125\), giving \(L=1.125-\log(1.125)-1=0.007217\). On day 3, the ratio is \(0.005/0.006=0.8333\), giving \(L=0.8333-\log(0.8333)-1=0.01566\). The mean loss is \((0.003678+0.007217+0.01566)/3=0.008852\), with the slight difference from \(0.008850\) attributable to rounding the displayed daily losses.

For Model B, the day 1 ratio is \(0.011/0.013=0.8462\), so its loss is \(0.011/0.013-\log(0.011/0.013)-1\approx0.01321\). The day 2 ratio is \(0.009/0.007=1.2857\), giving \(1.2857-\log(1.2857)-1=0.03440\). On day 3, its forecast is \(0.006\), as for Model A, so its loss is \(0.01566\). The mean is approximately \((0.01320793+0.03439987+0.01565509)/3=0.02109\). Model A has the lower average QLIKE loss for these three hypothetical days. This is a descriptive ranking on a very short evaluation sample, not evidence by itself of a persistent advantage.

The calculation also shows why rounding should be delayed until the end. When ratios are close to one, QLIKE losses are small, so premature rounding of ratios or logarithms can noticeably affect the displayed mean. Report enough precision to make the comparison reproducible.

Justification: Let \(g(x)=x-\log x-1\) for \(x>0\). Then \(g'(x)=1-1/x\), which is negative for \(0<x<1\) and positive for \(x>1\). Therefore \(g\) has its minimum at \(x=1\), where \(g(1)=0\). This establishes the nonnegativity and minimum of normalized QLIKE.

Scoring Density Forecasts

A logarithmic score for a continuous density forecast is \( \log f_t(y_t) \). Higher values are better, provided the forecasts describe the same outcome on the same scale. Equivalently, one can use the negative log score, \( -\log f_t(y_t) \), as a loss to minimize. The score rewards assigning relatively high density to outcomes that occur and penalizes assigning them very low density.

The score evaluates the density at the realized outcome, but one observation cannot fully describe the quality of a forecast distribution. Across many forecast periods, mean log scores summarize performance under the logarithmic scoring rule. A log score accounts for the full forecast distribution through its value at the outcome, but a good average score does not guarantee that every aspect of calibration is satisfactory.

Worked Example: Computing a Log Score

For a hypothetical return, the observed value is \(y_t=0.020\). Model A forecasts a normal density with mean \(0\) and standard deviation \(0.020\); Model B forecasts a normal density with mean \(0.010\) and standard deviation \(0.015\). These are hypothetical forecasts on the same return scale. For a normal density with mean \(\mu_t\) and standard deviation \(\sigma_t\), the log score is \( -\log(\sigma_t)-\tfrac12\log(2\pi)-\tfrac12((y_t-\mu_t)/\sigma_t)^2 \).

For Model A, \(z=(0.020-0)/0.020=1\). Its score is \( -\log(0.020)-\tfrac12\log(2\pi)-\tfrac12(1)^2 =3.9120-0.9189-0.5000=2.493 \).

For Model B, \(z=(0.020-0.010)/0.015=0.6667\). Its score is \( -\log(0.015)-\tfrac12\log(2\pi)-\tfrac12(0.6667)^2 =4.1997-0.9189-0.2222=3.059 \).

Model B assigns the higher density value at this particular observed return, so it has the higher log score for this observation. That does not show that Model B has better average density-forecast performance; that comparison requires a set of out-of-sample forecasts and outcomes.

Calibration and Probability Integral Transforms

A forecast distribution is calibrated when its probability statements agree with the frequencies of outcomes over repeated forecasts, conditional on the information set used to make those forecasts. A standard diagnostic for continuous forecasts is the probability integral transform, or PIT. For each period, compute the forecast cumulative probability evaluated at the observed outcome.

Definition: The probability integral transform for a continuous forecast distribution is \(u_t=F_t(y_t)\). If the forecast conditional distributions are correctly specified and the observations are generated according to them, the PIT values are uniformly distributed on \([0,1]\). Under the corresponding conditional specification across time, they should also have no systematic serial dependence.

Uniformity is a calibration condition, not a measure of sharpness. Sharpness describes how concentrated a forecast distribution is; among calibrated forecasts, greater concentration can be useful, but a narrow distribution that misses too often is not well calibrated. PIT histograms can reveal broad patterns: a U-shape often indicates forecasts that are too narrow, while a hump shape can indicate forecasts that are too wide. Skewness may signal a location or asymmetry problem. These patterns are diagnostic clues rather than unique explanations.

Worked Example: Interpreting PIT Values

Suppose a hypothetical normal density forecast has mean \(0.010\) and standard deviation \(0.005\), and the observed return is \(0.015\). The standardized value is \(z=(0.015-0.010)/0.005=1\). Therefore the PIT is \(u_t=\Phi(1)=0.8413\), where \(\Phi\) is the standard normal cumulative distribution function. The forecast assigned about 84.13% probability to returns no larger than the observed return.

Now consider six hypothetical PIT values: \(0.08, 0.22, 0.37, 0.48, 0.76,\) and \(0.94\). Divide the unit interval into three equal-width bins: \([0,1/3)\), \([1/3,2/3)\), and \([2/3,1]\). The counts are 2, 2, and 2. A uniform distribution would have expected count \(6/3=2\) in each bin, so this tiny sample shows no imbalance across these coarse bins. It is not enough to establish calibration: the sample is small, and the PIT values should also be examined for serial dependence and other systematic patterns.

Comparing Predictive Accuracy

To compare two forecasts, evaluate them on the same dates with the same loss function. Let \(L_{A,t}\) and \(L_{B,t}\) be the losses for forecasts A and B on date \(t\), and define the paired loss difference \(d_t=L_{A,t}-L_{B,t}\). A negative mean difference favors A because its loss is lower on average. Pairing matters: it compares the forecasts against the same realized outcomes and removes variation common to both forecasts.

Definition: The null hypothesis of equal predictive accuracy under a chosen loss is \(H_0:E[d_t]=0\). The alternative may be two-sided, \(H_a:E[d_t]\ne 0\), or directional when a specific ordering is justified in advance. A Diebold–Mariano comparison uses the sample mean of \(d_t\) relative to an estimate of its sampling uncertainty. For dependent loss differences, that uncertainty must account for serial correlation.

For a sample of \(T\) paired forecasts, write \(\bar d=T^{-1}\sum_{t=1}^{T}d_t\). If the differences are serially uncorrelated, an estimated standard error is \(s_d/\sqrt{T}\), where \(s_d\) is their sample standard deviation. For serially dependent differences, use a heteroskedasticity-and-autocorrelation-consistent estimate of the long-run variance instead. The resulting test statistic is compared with an appropriate reference distribution, commonly an asymptotic standard normal distribution. The validity of the comparison depends on the sampling assumptions, the loss choice, and the treatment of dependence.

Worked Example: A Paired Loss-Difference Check

Consider four hypothetical dates and losses from the same forecast target and loss function. Suppose Model A's losses are \(1,2,1,3\), while Model B's are \(2,1,2,4\). The paired differences \(d_t=L_{A,t}-L_{B,t}\) are \(-1,1,-1,-1\), and their mean is \(\bar d=(-1+1-1-1)/4=-0.5\). The negative mean favors A descriptively.

For illustration only, assume these differences are independent, normally distributed observations with a common variance. Their deviations from \(-0.5\) are \(-0.5,1.5,-0.5,-0.5\). Thus \(s_d^2=(0.25+2.25+0.25+0.25)/(4-1)=1\), so the estimated standard error is \(1/\sqrt{4}=0.5\). The statistic is \(t=\bar d/(s_d/\sqrt{4})=-0.5/0.5=-1\). With \(df=3\), the two-sided \(t\)-test p-value is approximately \(0.391\). At the 5% level, this result does not provide evidence against equal mean losses under the illustrative independence and normality assumptions. Four dates give very little information; a serious forecast comparison needs a longer evaluation sample and an appropriate treatment of dependence.

Common Mistakes and Practical Checks

Forecast evaluation is easiest to misread when the score, the forecast target, or the evaluation sample is left implicit. Before drawing a conclusion, specify the target and units, make sure forecasts are genuinely out of sample, and check that all models are scored on the same dates.

Calling a density value a probability. A continuous density assigns probability zero to one exact outcome. The log score uses \( \log f_t(y_t) \), the log density value at the observed outcome.
Comparing losses with different scales or targets. A variance loss must compare a variance forecast with a variance measure for the same period. A density score comparison requires the same outcome variable and units.
Inferring calibration from a favorable average score. A good mean log score does not rule out systematic calibration errors. Inspect PIT behavior as well.
Treating a lower sample loss as proof of superiority. The ranking is descriptive until uncertainty in paired loss differences is assessed, with serial dependence handled where needed.
Using a PIT histogram as a complete test. A histogram can reveal marginal nonuniformity but may miss serial dependence or more subtle departures from the forecast model.

Key takeaway: Use realized-variance losses to compare volatility forecasts, log scores to compare density forecasts, and PIT diagnostics to examine calibration. Comparative claims should be based on paired out-of-sample results and should reflect uncertainty and dependence in the evaluation sample.

Check Your Understanding

Answer these questions using the definitions and procedures above.

Why does a continuous density forecast use a density value at the observed outcome rather than the probability of that exact outcome?
For positive \(RV_t\) and \(h_t\), at what ratio does normalized QLIKE attain its minimum, and why?
A PIT value is \(0.90\). What does this say about the forecast cumulative probability at the observed outcome?
Why should two forecasts be compared using losses on the same dates and under the same loss function?
What additional issue must be addressed when paired loss differences are serially correlated?