Bayesian Updating For Econometric Models

From a Sampling Model to a Posterior

Econometric models describe how observed data arise given parameters. Bayesian updating adds a probability distribution over those parameters before the data are observed, then combines the two pieces to describe parameter uncertainty after observing the data. The central calculation is compact, but its interpretation depends on being clear about the sampling model, the prior, and what counts as the observed data.

Let \(y\) denote observed data and \(\theta\) an unknown parameter or parameter vector. A sampling model specifies the likelihood \(L(\theta;y)=p(y\mid\theta)\), the probability mass or density of the data conditional on \(\theta\). A prior density \(\pi(\theta)\) represents uncertainty about \(\theta\) before observing \(y\). For a discrete parameter, the corresponding quantities are probability masses.

Definition: For a continuous parameter with prior density \(\pi(\theta)\), the posterior density is \[ p(\theta\mid y)=\frac{p(y\mid\theta)\pi(\theta)}{m(y)},\qquad m(y)=\int p(y\mid\theta)\pi(\theta)\,d\theta. \] The quantity \(m(y)\) is the normalizing constant, also called the marginal likelihood or prior predictive density of the data. For a discrete parameter, replace the integral by a sum. The prior must be a proper probability distribution, and the integral or sum defining \(m(y)\) must be finite and positive for this formula to define a proper posterior.

The numerator combines how well each parameter value predicts the observed data with how much prior weight that value receives. The denominator makes the resulting density integrate to one. In particular, integrating the numerator over \(\theta\) gives \(m(y)\), so division by \(m(y)\) normalizes it. For continuous data, \(m(y)\) is a density at the observed value, not generally the probability of observing that exact value; a single exact value has probability zero under a continuous distribution.

When deriving a posterior, likelihood terms that do not depend on \(\theta\) can be dropped because they cancel between numerator and denominator. But those terms must be retained when calculating \(m(y)\) as a data density. The normalizing constant depends on the stated data representation and sampling model, so the likelihood must be specified on a consistent scale.

Updating a Normal Mean

Suppose \(Y_1,\ldots,Y_n\) are independent observations from \(N(\mu,\sigma^2)\), with known \(\sigma^2\), and the unknown parameter is the population mean \(\mu\). Assume the prior is \(\mu\sim N(\mu_0,\tau^2)\). The sampling distribution of the sample mean is \(\bar{Y}\mid\mu\sim N(\mu,\sigma^2/n)\). We can therefore update using \(\bar{Y}\) as the data summary; here the normalizing constant calculated below is its prior predictive density.

Worked Example: Updating a Normal Mean

Suppose four hypothetical measurements are 4, 5, 5, and 6 units. Assume independent normal sampling with known standard deviation 2 units, and use the prior \(\mu\sim N(0,9)\), where the prior variance is 9 square units. Find the posterior distribution of \(\mu\) and the normalizing constant for the observed sample mean.

The sample mean is \(\bar{y}=(4+5+5+6)/4=5\) units. Since \(\sigma^2=4\) and \(n=4\), the sampling variance of \(\bar{Y}\) is \(4/4=1\) square unit. Thus \(\bar{Y}\mid\mu\sim N(\mu,1)\), and the likelihood for this summary is proportional to \(\exp[-(\bar{y}-\mu)^2/2]\).

Multiplying the likelihood by the \(N(0,9)\) prior gives a posterior precision equal to the sum of the likelihood precision and prior precision:

\frac{1}{v_{\text{post}}}=\frac{1}{1}+\frac{1}{9}=\frac{10}{9}, \qquad v_{\text{post}}=\frac{9}{10}=0.9.

The posterior mean is the precision-weighted average of the sample mean and prior mean:

\mu_{\text{post}} = \frac{\bar{y}/1+0/9}{1+1/9} = \frac{5}{10/9} = 4.5.

Therefore \(\mu\mid\bar{y}=5\sim N(4.5,0.9)\), with mean in measurement units and variance in square units. The prior predictive distribution of the sample mean is \(N(0,9+1)=N(0,10)\): it follows by adding the independent prior uncertainty in \(\mu\) and sampling uncertainty in \(\bar{Y}\mid\mu\). The normalizing constant at the observed mean is

m(\bar{y}=5) = \frac{1}{\sqrt{2\pi(10)}}\exp\!\left(-\frac{5^2}{2(10)}\right) \approx 0.03615

per unit of sample-mean measurement. This is a density, not the probability that the sample mean equals exactly 5. The calculation uses the sample mean as the observed data summary; the full-data density would be a different density with a different unit.

Why the Normalizing Constant Is a Predictive Quantity

The integral defining \(m(y)\) averages the likelihood over parameter values according to the prior. Before seeing the data, the model assigns prior probability or density to possible observations by first drawing \(\theta\) from \(\pi(\theta)\), then drawing \(y\) from \(p(y\mid\theta)\). Integrating over \(\theta\) gives the resulting prior predictive distribution of \(y\). This explains why the denominator is not an arbitrary mathematical correction: it measures how the model, averaged over its prior uncertainty, predicts the observed data.

A short derivation also shows why the posterior is normalized. For a proper prior and a positive, finite \(m(y)\), integrate the proposed posterior over the parameter space:

\int p(\theta\mid y)\,d\theta = \frac{\int p(y\mid\theta)\pi(\theta)\,d\theta}{m(y)} = \frac{m(y)}{m(y)} = 1.

The same reasoning uses a sum for a discrete parameter. If the prior is improper, its density is known only up to an arbitrary multiplicative constant; a posterior kernel may sometimes still be normalized, but \(m(y)\) is not defined as a prior predictive density in the usual way. A proper posterior alone does not make an improper prior a probability distribution.

Worked Example: Normalizing a Discrete Parameter Posterior

Consider five hypothetical binary outcomes \(y=(1,1,0,1,0)\), modeled as independent Bernoulli observations with success probability \(\theta\). Suppose \(\theta\) can take only \(0.2\), \(0.5\), or \(0.8\), with prior probabilities \(0.2\), \(0.5\), and \(0.3\), respectively. Calculate the posterior probabilities and the normalizing constant.

There are three successes and two failures. For this particular ordered outcome vector, the likelihood is \(L(\theta;y)=\theta^3(1-\theta)^2\); no binomial coefficient is included because the data are the specified sequence, not just its success count. The likelihoods are

L(0.2;y)=0.2^3(0.8)^2=0.00512,\quad L(0.5;y)=0.5^3(0.5)^2=0.03125,\quad L(0.8;y)=0.8^3(0.2)^2=0.02048.

Multiplying each likelihood by its prior probability gives unnormalized posterior weights \(0.001024\), \(0.015625\), and \(0.006144\). Their sum is the normalizing constant:

$$ m(y)=0.001024+0.015625+0.006144=0.022793. $$

Dividing each weight by this sum gives posterior probabilities of approximately \(0.04493\), \(0.6853\), and \(0.2698\), respectively. For example, \(P(\theta=0.5\mid y)=0.015625/0.022793\approx0.6853\). These probabilities sum to one, and the middle parameter value receives most posterior weight because its likelihood is largest and its prior weight is also substantial. Here \(m(y)\) is the prior predictive probability of this particular outcome sequence.

Building a Posterior in Gaussian Regression

The same construction applies to regression. State the conditional sampling model first, including which quantities are treated as fixed or known, then specify a prior for the coefficient or coefficient vector. The posterior is proportional to the regression likelihood times the coefficient prior. If the prior and sampling model are Gaussian, their product has a Gaussian kernel in the coefficients; the denominator is the integral of the full likelihood times the prior over all coefficient values.

The following deliberately small model has one unknown slope, no intercept, and a known error variance. It illustrates the construction and the meaning of the denominator without requiring a general regression formula. The regression is a model for conditional association; this example alone does not establish a causal interpretation.

Worked Example: A One-Slope Gaussian Regression Posterior

For three hypothetical observations, let the fixed regressor values be \(x=(-1,0,1)\) and the outcomes be \(y=(-1,1,2)\), measured in outcome units. Assume \(Y_i=\beta x_i+\varepsilon_i\), with independent errors \(\varepsilon_i\sim N(0,1)\). The intercept is fixed at zero, the error variance is known, and the prior is \(\beta\sim N(0,4)\). Find the posterior for \(\beta\) and interpret its normalizing constant.

The likelihood and prior densities, including their parameter-independent factors, are

p(y\mid\beta) = (2\pi)^{-3/2} \exp\!\left[-\frac{1}{2}\sum_{i=1}^{3}(y_i-\beta x_i)^2\right], \qquad \pi(\beta) = \frac{1}{\sqrt{8\pi}}\exp\!\left(-\frac{\beta^2}{8}\right).

From the data, \(\sum_i y_i^2=6\), \(\sum_i x_i y_i=3\), and \(\sum_i x_i^2=2\). Consequently,

\sum_i(y_i-\beta x_i)^2 = 6-6\beta+2\beta^2.

After multiplying likelihood and prior, the terms involving \(\beta\) in the exponent are \(3\beta-1.125\beta^2\). Completing the square gives

3\beta-1.125\beta^2 = -\frac{9}{8}\left(\beta-\frac{4}{3}\right)^2+2.

Therefore the normalized posterior is \(\beta\mid y\sim N(4/3,4/9)\). Its posterior mean is about 1.333 outcome units per regressor unit, and its posterior variance is \(4/9\) in squared slope units.

For the normalizing constant, the prior predictive distribution is \(y\sim N(0,I+4xx^\mathsf{T})\), where \(I\) is the \(3\times3\) identity matrix. This follows because \(y=x\beta+\varepsilon\), with independent \(\beta\sim N(0,4)\) and \(\varepsilon\sim N(0,I)\). The covariance has eigenvalue \(1+4x^\mathsf{T}x=9\) in the direction of \(x\), and eigenvalue 1 in each of the two perpendicular directions, so its determinant is 9. The squared length of \(y\) in the \(x\) direction is \((x^\mathsf{T}y)^2/(x^\mathsf{T}x)=9/2\); the squared perpendicular length is \(6-9/2=3/2\). Thus the quadratic form in the multivariate normal density is \((9/2)/9+3/2=2\), and the data density is

m(y) = \frac{1}{(2\pi)^{3/2}\sqrt{9}}\exp\!\left(-\frac{2}{2}\right) \approx 0.007787.

This is the prior predictive density for the three-dimensional outcome vector, in inverse outcome-units cubed. It accounts for the full likelihood constants; dropping those constants would preserve the posterior but would not give this value of \(m(y)\).

Common Mistakes and Practical Checks

Bayesian updating is algebraically simple once the model is specified, but several errors can change the meaning or validity of the result. Check each of the following before interpreting a posterior:

Confusing a likelihood with a probability distribution over parameters. The likelihood describes data conditional on a parameter. It becomes part of a parameter distribution only when combined with a prior and normalized.
Calling a continuous-data density a probability. A value such as \(m(y)=0.03615\) is a density at the observation, not a probability of that exact observation.
Dropping constants when calculating the normalizer. Terms independent of \(\theta\) cancel from the posterior kernel but may be essential to the numerical value of the data density.
Using an improper prior as though it were a probability distribution. An improper prior has no finite total probability, so it does not define the prior predictive density in the usual sense.
Interpreting a regression coefficient causally without a design or identification argument. A posterior describes uncertainty under the model and prior; it does not by itself establish that changing a regressor causes an outcome change.

A useful final check is to verify that posterior masses sum to one or that the posterior density integrates to one, and to verify that the normalizing constant is calculated for the same data representation and likelihood used in the update. The posterior summarizes uncertainty conditional on the specified model, prior, and observed data; it is not a probability that the model assumptions themselves are correct.

Check Your Understanding

Answer these questions using the model and data specified in each case.

In Bayes’ rule, what does the normalizing constant integrate or sum over, and why does it make the posterior normalized?
For continuous data, why is the normalizing constant a density rather than generally the probability of the exact observed value?
In the normal-mean example, which two sources of uncertainty add to give the prior predictive variance of the sample mean?
Why can a parameter-independent likelihood factor be omitted when finding a posterior kernel but not when reporting the data density?
In the regression example, what assumptions allow the prior predictive outcome vector to have covariance \(I+4xx^\mathsf{T}\)?