Metropolis-Hastings For Nonconjugate Econometric Posteriors

Why Metropolis–Hastings Is Useful

In Bayesian regression, some likelihood-and-prior combinations yield familiar conditional posterior distributions that can be sampled directly. Others do not. A prior with a density that is not conjugate to the regression likelihood can make the posterior nonstandard, even when evaluating its density is straightforward. Metropolis–Hastings (MH) provides a way to generate draws from such a posterior using proposals that are easier to simulate than the target distribution.

An MH update proposes a new parameter value, compares its posterior density with the current value, and either accepts the proposal or stays at the current value. The proposal need not resemble the posterior exactly, but its density must be accounted for correctly. This tutorial develops random-walk and independence proposals for a simple nonconjugate regression and works through the arithmetic for both.

A Nonconjugate Regression Target

Suppose the outcome $y_i$ follows a linear regression with fixed covariates $x_i$ and two coefficients, an intercept $\beta_0$ and a slope $\beta_1$. Assume independent, normally distributed errors with known variance $\sigma^2$. For a concrete example, use three hypothetical observations, $x=(0,1,2)$ and $y=(1,2,2)$, with $\sigma^2=1$. Assign independent priors: $\beta_0\sim N(0,100)$ and a Laplace prior for $\beta_1$ with density $\frac12\exp(-|\beta_1|)$.

The Laplace prior on the slope is not conjugate to the Gaussian regression likelihood. The posterior is nevertheless available up to a normalizing constant, which is all MH requires. Omitting constants that do not depend on the coefficients, its log kernel is

\ell(\beta_0,\beta_1) = -\frac{1}{2\sigma^2}\sum_{i=1}^n (y_i-\beta_0-\beta_1x_i)^2 -\frac{\beta_0^2}{200} -|\beta_1|.

Here $\sigma^2=1$. The quadratic likelihood term follows from the assumed normal errors, the intercept term is the log kernel of $N(0,100)$, and the final term is the log kernel of the Laplace prior. The absolute value makes the posterior non-Gaussian. Write $\pi(\beta\mid y)$ for the normalized posterior density and $K(\beta)=\exp\{\ell(\beta)\}$ for an unnormalized density kernel, where $\beta=(\beta_0,\beta_1)$. Ratios of $K$ equal ratios of $\pi$, so calculations can use $\ell$ without finding the posterior's normalizing constant.

Definition: A Metropolis–Hastings update at current state $\beta$ first draws a candidate $\beta'$ from a proposal density $q(\beta'\mid\beta)$. It accepts the candidate with probability $$ \alpha(\beta,\beta') = \min\left\{1,\, \frac{\pi(\beta'\mid y)\,q(\beta\mid\beta')} {\pi(\beta\mid y)\,q(\beta'\mid\beta)} \right\}. $$ If the candidate is accepted, the next state is $\beta'$; otherwise, the next state remains $\beta$.

Random-Walk Metropolis–Hastings

A random-walk proposal adds a random perturbation to the current value. For example, a joint Gaussian random walk proposes $\beta'=\beta+\varepsilon$, where $\varepsilon\sim N(0,\Sigma_q)$, with a fixed covariance matrix $\Sigma_q$. Because the density of a centered Gaussian perturbation is unchanged when its sign is reversed, $q(\beta'\mid\beta)=q(\beta\mid\beta')$. The proposal terms therefore cancel, leaving

\alpha(\beta,\beta') = \min\left\{1,\frac{\pi(\beta'\mid y)}{\pi(\beta\mid y)}\right\} = \min\{1,\exp[\ell(\beta')-\ell(\beta)]\}.

The same cancellation holds for a symmetric random walk that updates one coefficient at a time, with the other coefficient held fixed. A candidate with higher posterior density is accepted with probability one. A candidate with lower density can still be accepted, but only with probability equal to the posterior density ratio. That possibility is essential: simply moving uphill would not, in general, produce draws from the desired posterior.

Worked Example: A Joint Random-Walk Proposal

At the current state $\beta=(1,0.5)$, consider the candidate $\beta'=(1.2,0.8)$. Suppose it was generated by a symmetric Gaussian random walk. Compute its acceptance probability.

Check the regression likelihood. At the current state, the fitted values at $x=0,1,2$ are $1,1.5,2$. The residuals are $0,0.5,0$, so the residual sum of squares is $0.25$. At the candidate state, the fitted values are $1.2,2,2.8$, the residuals are $-0.2,0,-0.8$, and the residual sum of squares is $0.68$.

Calculate the log-posterior change. The likelihood contribution changes by $-\frac12(0.68-0.25)=-0.215$. The intercept-prior contribution changes by $-(1.2^2-1^2)/200=-0.0022$, and the slope-prior contribution changes by $-(|0.8|-|0.5|)=-0.3$. Thus

\ell(\beta')-\ell(\beta) = -0.215-0.0022-0.3 = -0.5172.

Apply the MH rule. Symmetry cancels the proposal densities, so $\alpha=\min\{1,\exp(-0.5172)\}\approx0.5962$. If the algorithm draws $u=0.40$ from a uniform distribution on $[0,1]$, it accepts because $0.40\leq0.5962$. If instead $u=0.70$, it rejects and retains $(1,0.5)$.

Independence Metropolis–Hastings

An independence proposal draws a candidate from a fixed density $q(\beta')$, regardless of the current state. It is useful when a convenient approximation to the posterior is available, although a poor approximation can lead to low acceptance. Since $q(\beta'\mid\beta)=q(\beta')$, the acceptance probability is

\alpha(\beta,\beta') = \min\left\{1,\, \frac{\pi(\beta'\mid y)}{\pi(\beta\mid y)} \frac{q(\beta)}{q(\beta')} \right\}.

Unlike a symmetric random walk, an independence proposal generally does not have canceling proposal terms. The factor $q(\beta)/q(\beta')$ corrects for how likely each point is under the proposal. It can reduce the acceptance probability for a candidate that the proposal generates unusually often, or increase it for a candidate that the proposal rarely generates. Omitting this factor generally changes the target distribution.

Worked Example: An Independence Proposal

Use the same current state $(1,0.5)$ and candidate $(1.2,0.8)$ from the previous example, but now draw candidates independently from $q(\beta)=N(0,4I)$, where $I$ is the two-dimensional identity matrix. The log-posterior difference is $-0.5172$, as calculated above. Find the proposal correction and acceptance probability.

For this centered Gaussian proposal, the log density, up to a constant, is $-\|\beta\|^2/8$. The squared norms are $\|(1,0.5)\|^2=1.25$ and $\|(1.2,0.8)\|^2=2.08$. Therefore

\log\frac{q(\beta)}{q(\beta')} = -\frac{1.25}{8}-\left(-\frac{2.08}{8}\right) = 0.10375.

The total log acceptance ratio is the posterior log ratio plus the proposal log ratio: $-0.5172+0.10375=-0.41345$. Hence $\alpha=\min\{1,\exp(-0.41345)\}\approx0.6614$. A uniform draw of $u=0.65$ results in acceptance, since $0.65\leq0.6614$. This example shows why the posterior comparison alone is not enough for an independence proposal.

Why the Acceptance Rule Targets the Posterior

The acceptance formula is chosen to make probability flow between any two states balance under the posterior. For two distinct states $\beta$ and $\beta'$, the probability density of proposing and accepting the move from $\beta$ to $\beta'$, weighted by the target density at $\beta$, is

\pi(\beta\mid y)q(\beta'\mid\beta)\alpha(\beta,\beta') = \min\{\pi(\beta\mid y)q(\beta'\mid\beta), \pi(\beta'\mid y)q(\beta\mid\beta')\}.

To see this, substitute the definition of $\alpha$. Multiplying $q(\beta'\mid\beta)$ by the minimum of 1 and the acceptance ratio gives the minimum of the forward proposal density and the reverse density weighted by $\pi(\beta'\mid y)/\pi(\beta\mid y)$; multiplying by $\pi(\beta\mid y)$ yields the displayed expression. Its right-hand side is unchanged when $\beta$ and $\beta'$ are exchanged. Thus the weighted accepted flow from $\beta$ to $\beta'$ equals the weighted accepted flow in the reverse direction. Rejections contribute transitions that stay at the current state. Together, these transitions preserve the posterior distribution. In practice, the proposal must also allow the chain to reach the relevant parts of the target; a proposal that rules out regions with positive posterior density cannot explore the full target.

Key result: For any proposal, compute the log acceptance ratio as $$ r=\ell(\beta')-\ell(\beta) +\log q(\beta\mid\beta') -\log q(\beta'\mid\beta). $$ Then accept if $r\geq0$, or, when $r<0$, accept if $\log u\leq r$ for $u\sim\text{Uniform}(0,1)$. This log-scale comparison avoids calculating very small density ratios directly.

Proposal Scale and Practical Implementation

For a random walk, the proposal covariance determines the scale and direction of candidate changes. A small scale tends to produce nearby proposals, which can have relatively high acceptance probabilities, but each accepted move may change the coefficients only slightly. A large scale tends to produce more distant candidates, which may have lower posterior density and be rejected more often. Neither a high acceptance rate nor a large jump by itself guarantees effective exploration.

For a fixed pair of states under a symmetric random walk, the proposal scale does not appear in the acceptance formula: only the posterior ratio remains. Scale matters because it changes which candidate states are proposed and how often. The following comparison uses the same posterior as above.

Worked Example: Comparing Small and Larger Moves

From $(1,0.5)$, consider first a small slope-only proposal to $(1,0.6)$, as might occur with a small-scale coordinate random walk. The fitted values are $1,1.6,2.2$, giving residuals $0,0.4,-0.2$ and a residual sum of squares of $0.20$. The intercept is unchanged. The log-posterior difference is

-\frac12(0.20-0.25)-(|0.6|-|0.5|) = 0.025-0.1 = -0.075.

The symmetric-proposal acceptance probability is $\exp(-0.075)\approx0.9277$. This candidate is likely to be accepted, but its slope changes by only $0.1$. The larger joint move to $(1.2,0.8)$, calculated earlier, has acceptance probability about $0.5962$. It is less likely to be accepted but changes both coefficients more. These two calculations illustrate the tradeoff; they do not establish a universal relationship between scale and average acceptance, which also depends on the posterior shape and current state.

Common Mistakes

Dropping the proposal correction. This is valid for a symmetric random walk, not for a general proposal. For an independence proposal, include $q(\beta)/q(\beta')$.
Accepting every proposal with a higher likelihood. The comparison is based on the posterior, which includes both likelihood and prior. A proposal with higher posterior density is accepted, but a lower-density proposal can also be accepted with a probability below one.
Using an unnormalized posterior as if its normalizing constant were known. Posterior ratios are valid because the common constant cancels. The same is true of log-posterior differences.
Comparing $u$ to the log acceptance ratio. Compare $u$ to $\alpha$, or compare $\log u$ to the log acceptance ratio when that ratio is negative. Do not compare a uniform draw directly with $r$.
Treating a high acceptance probability as proof of good exploration. Very small random-walk steps can be accepted often while moving slowly. Proposal scale should be judged by the resulting movement as well as acceptance.

Key takeaway: Define the posterior kernel first, then combine its log-density change with the proposal-density correction. Symmetric random walks cancel that correction; independence proposals generally do not. Proposal scale changes the candidates presented to the acceptance rule, not the posterior distribution the rule targets.

Check Your Understanding

Try these questions using the definitions and calculations above.

Why can a posterior normalizing constant be omitted when computing an MH acceptance probability?
A symmetric random-walk candidate has a log-posterior difference of $-0.8$. What is its acceptance probability, to three significant figures?
For an independence proposal, why must the ratio $q(\beta)/q(\beta')$ appear in the acceptance probability?
If a proposal has acceptance probability $0.30$ and the uniform draw is $u=0.25$, is it accepted? What is the next state if the current state is $\beta$ and the candidate is $\beta'$?
Explain why reducing the random-walk proposal scale may increase acceptance while still making exploration less efficient.