Posterior Prediction For Econometric Outcomes

Why Prediction Includes More Than Coefficient Uncertainty

A regression model can describe the conditional mean of an outcome without yet describing the uncertainty in a future observation. For a new unit with covariates $\mathbf{x}_0$, the conditional mean is $\mathbf{x}_0^{\mathsf T}\boldsymbol{\beta}$. A realized outcome also includes an error term. Bayesian prediction must account for both that outcome-level variation and uncertainty about the regression parameters.

This distinction matters in applications such as forecasting an individual firm's sales, a household's expenditure, or a region's employment. An interval for the conditional mean describes uncertainty about the average outcome at the given covariates. An interval for a new outcome must additionally allow for the fact that individual outcomes vary around that mean.

Definition: Suppose the observed data are $D$, and a new outcome $Y_0$ has covariate vector $\mathbf{x}_0$. Under a specified likelihood and posterior distribution for the parameters $\boldsymbol{\theta}$, the posterior predictive distribution of $Y_0$ is $$ p(y_0\mid D,\mathbf{x}_0) = \int p(y_0\mid \boldsymbol{\theta},\mathbf{x}_0)\, p(\boldsymbol{\theta}\mid D)\,d\boldsymbol{\theta}. $$ It averages the outcome distribution over posterior uncertainty in the parameters. A posterior predictive interval is an interval formed from quantiles of this distribution.

For the normal linear regression model, write $\boldsymbol{\theta}=(\boldsymbol{\beta},\sigma^2)$. Assume that, conditional on these parameters and the covariates, the observed outcomes follow the regression model with independent normal errors:

Y_i\mid\boldsymbol{\beta},\sigma^2,\mathbf{x}_i \sim N(\mathbf{x}_i^{\mathsf T}\boldsymbol{\beta},\sigma^2).

For prediction, assume the new outcome follows the same conditional model and is independent of the observed outcomes given $\boldsymbol{\beta}$, $\sigma^2$, and $\mathbf{x}_0$. Treat $\mathbf{x}_0$ as known. The posterior predictive distribution is then

p(y_0\mid D,\mathbf{x}_0) = \int \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left( -\frac{(y_0-\mathbf{x}_0^{\mathsf T}\boldsymbol{\beta})^2}{2\sigma^2} \right) p(\boldsymbol{\beta},\sigma^2\mid D)\, d\boldsymbol{\beta}\,d\sigma^2.

The integral, rather than a single plug-in regression line, is the defining operation. It averages across plausible coefficient and variance values, giving more posterior weight to parameter values that are more plausible given $D$.

Two Sources of Predictive Uncertainty

Let $\mu_0=\mathbf{x}_0^{\mathsf T}\boldsymbol{\beta}$ denote the conditional mean at the new covariates. Conditional on the parameters, $Y_0$ has mean $\mu_0$ and variance $\sigma^2$. Averaging over the posterior gives the predictive mean and variance:

E(Y_0\mid D,\mathbf{x}_0) = E(\mu_0\mid D) = \mathbf{x}_0^{\mathsf T}E(\boldsymbol{\beta}\mid D),

\operatorname{Var}(Y_0\mid D,\mathbf{x}_0) = E(\sigma^2\mid D) + \operatorname{Var}(\mu_0\mid D).

The first variance term is expected residual variation for an individual outcome. The second is posterior uncertainty about the conditional mean. The identity does not require the posterior for $\boldsymbol{\beta}$ and $\sigma^2$ to be independent.

Key distinction: An interval for $\mu_0=\mathbf{x}_0^{\mathsf T}\boldsymbol{\beta}$ uses only posterior uncertainty about the conditional mean. A predictive interval for $Y_0$ also includes residual variation, so it is generally wider. They answer different questions: uncertainty about an average versus uncertainty about a new realized outcome.

Known Error Variance: A Normal Predictive Distribution

When $\sigma^2$ is known and the posterior for $\boldsymbol{\beta}$ is multivariate normal, the predictive distribution is also normal. Suppose $\boldsymbol{\beta}\mid D\sim N(\mathbf{m},V)$ and $\sigma^2$ is fixed. The posterior distribution of the conditional mean is normal, with mean $\mathbf{x}_0^{\mathsf T}\mathbf{m}$ and variance $\mathbf{x}_0^{\mathsf T}V\mathbf{x}_0$. Adding the independent normal outcome error gives

Y_0\mid D,\mathbf{x}_0 \sim N\!\left( \mathbf{x}_0^{\mathsf T}\mathbf{m}, \sigma^2+\mathbf{x}_0^{\mathsf T}V\mathbf{x}_0 \right).

Worked Example: Known Error Variance and a New Outcome

Suppose the outcome is measured in thousands of dollars and the fitted model has an intercept and one covariate. At covariate value $x_0=3$, let $\mathbf{x}_0=(1,3)^{\mathsf T}$. Assume the posterior coefficient distribution and known error variance are

\boldsymbol{\beta}\mid D\sim N(\mathbf{m},V),\qquad \mathbf{m}= \begin{pmatrix}1.2\\0.8\end{pmatrix},\qquad V= \begin{pmatrix}0.04&-0.005\\-0.005&0.01\end{pmatrix}, \qquad \sigma^2=0.64.

The posterior mean of the conditional mean is $\mathbf{x}_0^{\mathsf T}\mathbf{m}=1.2+3(0.8)=3.6$ thousand dollars. Its posterior variance is

\mathbf{x}_0^{\mathsf T}V\mathbf{x}_0 =0.04+2(3)(-0.005)+9(0.01) =0.10.

Thus the predictive variance is $0.64+0.10=0.74$, and the predictive standard deviation is $\sqrt{0.74}=0.8602$ thousand dollars. A central 95% normal predictive interval is

3.6\pm1.96\sqrt{0.74} = [1.914,\ 5.286]\text{ thousand dollars}.

This interval describes uncertainty about a new outcome at $x_0=3$, under the stated model and posterior. For comparison, a central 95% interval for the conditional mean is $3.6\pm1.96\sqrt{0.10}=[2.980,\ 4.220]$ thousand dollars. It is narrower because it excludes the error variance $0.64$.

Unknown Error Variance: Student t Prediction

A common conjugate specification gives a closed-form predictive distribution while accounting for uncertainty in both coefficients and error variance. Use the normal-inverse-gamma posterior parameterization

\boldsymbol{\beta}\mid\sigma^2,D \sim N(\mathbf{m},\sigma^2V), \qquad \sigma^2\mid D\sim\operatorname{InverseGamma}(a,b),

where $a>0$, $b>0$, and the inverse-gamma density is proportional to $(\sigma^2)^{-(a+1)}\exp(-b/\sigma^2)$. Here $V$ is the coefficient covariance matrix conditional on $\sigma^2$, apart from its scale factor. Conditional on $\sigma^2$, integrating out $\boldsymbol{\beta}$ gives

Y_0\mid\sigma^2,D,\mathbf{x}_0 \sim N\!\left( \mathbf{x}_0^{\mathsf T}\mathbf{m}, \sigma^2(1+\mathbf{x}_0^{\mathsf T}V\mathbf{x}_0) \right).

Integrating out $\sigma^2$ gives a Student $t$ distribution. With $t_\nu(\mu,s)$ denoting a Student $t$ distribution with $\nu$ degrees of freedom, location $\mu$, and scale $s$, the result is

Y_0\mid D,\mathbf{x}_0 \sim t_{2a}\!\left( \mathbf{x}_0^{\mathsf T}\mathbf{m}, \sqrt{\frac{b}{a}(1+\mathbf{x}_0^{\mathsf T}V\mathbf{x}_0)} \right).

Why the Student t Formula Holds

Set $c=1+\mathbf{x}_0^{\mathsf T}V\mathbf{x}_0$, which is positive because $V$ is a covariance-scale matrix. Conditional on $\sigma^2$, the outcome has a normal density centered at $\mu_0=\mathbf{x}_0^{\mathsf T}\mathbf{m}$ with variance $c\sigma^2$. Multiplying that density by the inverse-gamma density and integrating over $\sigma^2$ yields a density proportional to

\int_0^\infty (\sigma^2)^{-(a+3/2)} \exp\!\left( -\frac{b+(y_0-\mu_0)^2/(2c)}{\sigma^2} \right)d\sigma^2 \ \propto\ \left[ b+\frac{(y_0-\mu_0)^2}{2c} \right]^{-(a+1/2)}.

After factoring out $b$, the kernel is $\left[1+(y_0-\mu_0)^2/(2bc)\right]^{-(a+1/2)}$. This is the Student $t$ density kernel with $2a$ degrees of freedom and squared scale $bc/a$, which establishes the stated distribution. The heavier tails than a normal distribution reflect uncertainty about $\sigma^2$, in addition to coefficient uncertainty.

Worked Example: A Predictive Interval With Unknown Variance

Suppose a hypothetical model predicts an outcome measured in thousands of dollars. At the covariates of interest, take an intercept-only illustration with posterior parameters $m=10$, $V=0.25$, $a=4$, and $b=6$. The posterior predictive location is $10$, and its degrees of freedom are $2a=8$. Its scale is

\sqrt{\frac{b}{a}(1+V)} = \sqrt{\frac{6}{4}(1.25)} = \sqrt{1.875} = 1.3693 \text{ thousand dollars}.

Using $t_{0.975,8}=2.306$, the central 95% predictive interval is

10\pm2.306(1.3693) = 10\pm3.158 = [6.842,\ 13.158]\text{ thousand dollars}.

The posterior distribution of the conditional mean has the same location and degrees of freedom, but its scale is $\sqrt{(b/a)V}=\sqrt{0.375}=0.6124$. Its central 95% interval is therefore $10\pm2.306(0.6124)=[8.588,\ 11.412]$ thousand dollars. The predictive interval is wider because it also includes the variation of a new outcome around its conditional mean.

Prediction From Posterior Draws

A closed-form distribution is not always available or convenient. The defining integral can instead be approximated using posterior draws. For each draw $(\boldsymbol{\beta}^{(s)},\sigma^{2(s)})$ from $p(\boldsymbol{\beta},\sigma^2\mid D)$, generate a new error $\varepsilon_0^{(s)}$ from $N(0,\sigma^{2(s)})$, then calculate

Y_0^{(s)} = \mathbf{x}_0^{\mathsf T}\boldsymbol{\beta}^{(s)} +\varepsilon_0^{(s)}.

The resulting simulated outcomes approximate draws from the posterior predictive distribution. Their empirical quantiles estimate predictive quantiles. In contrast, the values $\mathbf{x}_0^{\mathsf T}\boldsymbol{\beta}^{(s)}$, without the generated errors, approximate the posterior distribution of the conditional mean. Leaving out the errors therefore answers a different question.

Worked Example: Averaging a Tail Probability Over Posterior Draws

Consider four equally weighted hypothetical posterior draws for a regression's conditional mean and error standard deviation at a fixed covariate value. Suppose the pairs $(\mu_0,\sigma)$ are $(10,1)$, $(10,2)$, $(12,1)$, and $(12,2)$, with outcomes measured in thousands of dollars. To calculate the posterior predictive probability that a new outcome exceeds 12, calculate the conditional tail probability at each draw and average:

P(Y_0>12\mid D,\mathbf{x}_0) \approx \frac{1}{4}\sum_{s=1}^4 \left[ 1-\Phi\!\left(\frac{12-\mu_0^{(s)}}{\sigma^{(s)}}\right) \right].

For the four draws, these probabilities are approximately $1-\Phi(2)=0.0228$, $1-\Phi(1)=0.1587$, $1-\Phi(0)=0.5000$, and $1-\Phi(0)=0.5000$. Therefore,

P(Y_0>12\mid D,\mathbf{x}_0) \approx \frac{0.0228+0.1587+0.5000+0.5000}{4} = 0.2954.

Under these four equally weighted draws, the estimated predictive probability that the new outcome exceeds 12 thousand dollars is about 0.295. The calculation averages over both the variation in conditional means and the different error standard deviations; using only the average conditional mean would not produce this predictive probability.

Intervals, Assumptions, and Interpretation

A central 95% posterior predictive interval is formed from the 2.5th and 97.5th percentiles of $p(Y_0\mid D,\mathbf{x}_0)$. Under the assumed model, it is a posterior probability statement about the new outcome, conditional on the observed data and specified covariates. It is not a claim that there is a 95% frequentist probability that a fixed parameter lies in the interval. A credible interval for the conditional mean instead summarizes posterior uncertainty about $\mathbf{x}_0^{\mathsf T}\boldsymbol{\beta}$.

The conclusions depend on the likelihood and on the posterior used in the integration. In the normal regression setup, the new observation is assumed to follow the same conditional outcome model as the observed data, with the same error variance structure. If the covariates themselves are uncertain, a further predictive calculation must account for their distribution; the formulas here condition on $\mathbf{x}_0$. A narrow interval is not automatically evidence of accurate forecasting if the model assumptions are poor.

Common Pitfall

Wrong reasoning: “I have a posterior interval for the regression mean at this covariate value, so it is also a 95% interval for the next observed outcome.” This omits residual variation. Correction: To predict a new outcome, integrate the distribution $p(y_0\mid\boldsymbol{\beta},\sigma^2,\mathbf{x}_0)$ over the joint posterior for $(\boldsymbol{\beta},\sigma^2)$. With posterior simulation, add a new error draw for every parameter draw. The conditional-mean interval and the outcome-prediction interval should be labelled and interpreted separately.

Key takeaway: Bayesian prediction averages the entire outcome distribution over posterior parameter uncertainty. Coefficient uncertainty affects the predicted conditional mean; residual uncertainty affects a new realization around that mean. Both belong in a posterior predictive distribution.

Check Your Understanding

Answer each question using the distinction between a conditional mean and a new realized outcome.

Write the integral defining $p(y_0\mid D,\mathbf{x}_0)$ when the parameter vector is $\boldsymbol{\theta}$.
In the normal regression model, identify the two components of $\operatorname{Var}(Y_0\mid D,\mathbf{x}_0)$.
Why is a posterior interval for $\mathbf{x}_0^{\mathsf T}\boldsymbol{\beta}$ generally narrower than a predictive interval for $Y_0$?
Under the stated normal-inverse-gamma parameterization, what are the degrees of freedom of the posterior predictive Student $t$ distribution?
When generating posterior predictive draws, what additional random quantity must be generated after drawing the regression parameters?