Proper Prior Sequences for Generalized Bayes Rules

Why Proper Priors Enter a Generalized Bayes Argument

A generalized Bayes rule may be defined using a prior measure \(\nu\) whose total mass is infinite. Such a measure can still yield a finite posterior distribution after observing data, but it is not a proper prior probability distribution. Blyth’s method uses a sequence of proper priors to connect this generalized Bayes rule to comparisons of integrated risk. The construction of the sequence is only part of the argument: the integrated risks and the scale of the risk gap must also be controlled.

This tutorial separates three questions that are easy to conflate. Can the sigma-finite measure be approximated by proper priors? Do the corresponding posterior distributions approach the generalized posterior? And is the risk gap small enough, on the correct scale, to support a Blyth argument? The first question has a general answer; the latter two require additional conditions.

Definition: A measure \(\nu\) on the parameter space \(\Theta\) is sigma-finite if \(\Theta\) is a countable union of measurable sets of finite \(\nu\)-measure. A proper prior is a probability measure on \(\Theta\). For a measurable weight \(w\) with \(0\leq w\leq 1\) and \(0<\int w\,d\nu<\infty\), the normalized measure \(\pi(d\theta)=w(\theta)\nu(d\theta)/\int w\,d\nu\) is a proper prior.

Constructing Proper Priors by Restriction

Let \(\nu\) be sigma-finite. By definition, there are measurable sets \(C_1,C_2,\ldots\) whose union is \(\Theta\), each with finite \(\nu\)-measure. Define \(E_k=C_1\cup\cdots\cup C_k\). Then \(E_k\) increases to \(\Theta\) and \(\nu(E_k)<\infty\). If \(\nu(\Theta)>0\), some \(E_k\) has positive measure; discard any initial zero-mass sets. The normalized restrictions \(\pi_k(A)=\nu(A\cap E_k)/\nu(E_k)\) are proper priors. This construction needs no density for \(\nu\); it applies to any sigma-finite measure.

Result: Every nonzero sigma-finite measure \(\nu\) admits a sequence of proper priors formed from normalized restrictions to increasing measurable sets \(E_k\), with \(\bigcup_k E_k=\Theta\). More generally, any measurable weights \(w_k\) satisfying \(0\leq w_k\leq1\), \(w_k\uparrow1\) \(\nu\)-almost everywhere, and \(0<Z_k=\int w_k\,d\nu<\infty\) give proper priors \(\pi_k(d\theta)=w_k(\theta)\nu(d\theta)/Z_k\).

For the general weights, properness follows directly: \(\pi_k\) is a nonnegative measure with total mass \(\int d\pi_k=Z_k/Z_k=1\). For restrictions, take \(w_k=1_{E_k}\). The condition \(Z_k>0\) matters: a finite-mass set of measure zero cannot be normalized. This proves the result and explains exactly where sigma-finiteness is used. \(\square\)

The weights need not be indicator functions. Using them allows a gradual taper rather than a sharp cutoff, provided their integrals remain finite. In either case, the proper priors are approximations in the sense that their unnormalized measures \(w_k\nu\) retain more of \(\nu\) as \(k\) increases. The normalized probabilities \(\pi_k\), however, change scale as \(Z_k\) changes.

Posterior Approximation and a Finite-Action Result

Suppose the sampling model has density or mass function \(f(x\mid\theta)\) with respect to a measure \(\mu\). For a prior measure \(\nu\), the generalized posterior normalizer is \(m_\nu(x)=\int f(x\mid\theta)\nu(d\theta)\). Where \(0<m_\nu(x)<\infty\), the generalized posterior is the probability measure with density \(f(x\mid\theta)/m_\nu(x)\) relative to \(\nu\). A proper prior sequence approximates this posterior only where its own normalizers are well behaved.

Result: Let \(w_k\uparrow1\) \(\nu\)-almost everywhere, and define \(m_k(x)=\int f(x\mid\theta)w_k(\theta)\nu(d\theta)\). If \(0<m_\nu(x)<\infty\), then \(m_k(x)\uparrow m_\nu(x)\). If the action set is finite, the loss is measurable and bounded, and the generalized posterior normalizer is finite and positive, the minimum posterior expected loss under \(w_k\nu\) converges to the minimum posterior expected loss under \(\nu\).

To verify the claim about losses, write the finite action set as \(a_1,\ldots,a_J\), and suppose \(0\leq L(\theta,a_j)\leq M\). Define \(q_{j,k}(x)=\int L(\theta,a_j)f(x\mid\theta)w_k(\theta)\nu(d\theta)\) and \(q_j(x)=\int L(\theta,a_j)f(x\mid\theta)\nu(d\theta)\). The monotone convergence theorem gives \(m_k(x)\uparrow m_\nu(x)\) and \(q_{j,k}(x)\uparrow q_j(x)\). Each \(q_j(x)\) is finite because \(q_j(x)\leq M m_\nu(x)\). Since \(m_\nu(x)>0\), the posterior expected loss for action \(a_j\) converges: \(q_{j,k}(x)/m_k(x)\to q_j(x)/m_\nu(x)\). A minimum over finitely many convergent real sequences converges to the minimum of their limits. Thus the minimum posterior expected losses converge. This proves the result. \(\square\)

This is a statement about posterior expected losses at a fixed data value \(x\). It is not a statement that integrated risks converge, or that a Blyth risk gap vanishes. For an infinite action set, even convergence of the posterior expected loss for each fixed action need not imply convergence of the infimum over actions: actions that vary with \(k\) can affect the infimum. Additional hypotheses are then needed.

Worked Example: Normal Location with a Flat Prior

Let \(X\mid\theta\sim N(\theta,1)\), where \(\theta\in\mathbb R\), and take \(\nu(d\theta)=d\theta\), Lebesgue measure. It is sigma-finite but has infinite total mass. Set \(E_k=[-k,k]\). Then \(\nu(E_k)=2k\), so the proper prior \(\pi_k\) is uniform on \([-k,k]\), with density \(1/(2k)\) there.

At an observed value \(x\), the generalized posterior density is proportional to \(\exp\{-(x-\theta)^2/2\}\) on \(\mathbb R\). Its normalizer is \(\int_{\mathbb R}\exp\{-(x-\theta)^2/2\}\,d\theta=\sqrt{2\pi}\), so the generalized posterior is \(N(x,1)\). Under \(\pi_k\), the posterior is proportional to the same likelihood restricted to \([-k,k]\). For every fixed \(x\), the restricted normalizer increases to \(\sqrt{2\pi}\), and the posterior approaches the generalized posterior as the interval expands.

Under squared-error loss \(L(\theta,a)=(a-\theta)^2\), the generalized posterior mean is \(a=x\). Its sampling risk is \(R(\theta,x)=E_\theta[(X-\theta)^2]=1\). Consequently, its integrated risk under the unnormalized restriction is \(\int_{-k}^k R(\theta,x)\,d\theta=\int_{-k}^k1\,d\theta=2k\), which is finite for every \(k\), though it grows without bound. This example shows both why a proper sequence is available and why finiteness on each restriction should not be confused with a uniform bound over the sequence.

Risk Integrability and the Scale of the Blyth Gap

For a measurable decision rule \(\delta\), its risk is \(R(\theta,\delta)=\int L(\theta,\delta(x))f(x\mid\theta)\mu(dx)\). Assume the loss is nonnegative, so Tonelli’s theorem applies without first assuming integrability. For \(\tau_k(d\theta)=w_k(\theta)\nu(d\theta)\), the unnormalized integrated risk satisfies

\int R(\theta,\delta)\,\tau_k(d\theta) = \int\!\int L(\theta,\delta(x))f(x\mid\theta)w_k(\theta)\nu(d\theta)\,\mu(dx).

The equality follows by exchanging the order of integration for a nonnegative measurable integrand. The common risk-integrability requirement is that this quantity be finite for each \(k\) under consideration, both for the generalized Bayes rule and for the competing Bayes rules used in the comparison. A convenient sufficient condition is a bound \(R(\theta,\delta)\leq H(\theta)\) with \(\int w_kH\,d\nu<\infty\) for each \(k\). If losses can be signed, absolute integrability is needed before exchanging integrals or subtracting integrated risks.

Suppose \(\delta_k\) is Bayes under the proper prior \(\pi_k=\tau_k/Z_k\), and both integrated risks are finite. Its Bayes property gives the nonnegative gap

D_k=\int \bigl\{R(\theta,\delta)-R(\theta,\delta_k)\bigr\}\,\pi_k(d\theta)\geq0.

For Blyth’s method, the corresponding gap on the unnormalized scale is \(G_k=Z_kD_k=\int\{R(\theta,\delta)-R(\theta,\delta_k)\}\tau_k(d\theta)\). The distinction is essential. If \(\nu\) is infinite, \(Z_k\) often tends to infinity; \(D_k\to0\) alone does not imply \(G_k\to0\). For example, \(D_k=1/Z_k\) tends to zero when \(Z_k\to\infty\), while \(G_k=1\) does not. In a Blyth argument, it is the unnormalized gap that must be compared with any fixed positive integrated improvement on a region where \(w_k\) approaches one.

Risk condition for a Blyth comparison: For each \(k\), require finite integrated risks under \(\tau_k=w_k\nu\) for the rules being compared, so that \(G_k\) is well-defined. The vanishing-gap condition used in the argument is \(G_k\to0\), not merely \(D_k\to0\) for the normalized proper priors. Posterior convergence and finite risk on each individual restriction do not by themselves establish this condition.

Worked Example: A Proper Sequence with an Infinite Generalized Risk

Let \(X\mid\theta\) have an exponential distribution with rate \(\theta>0\), so \(f(x\mid\theta)=\theta e^{-\theta x}\) for \(x>0\). Use the sigma-finite prior \(\nu(d\theta)=d\theta/\theta\). For \(x>0\), the posterior kernel is \(\theta e^{-\theta x}d\theta/\theta=e^{-\theta x}d\theta\). Its normalizer is \(\int_0^\infty e^{-\theta x}d\theta=1/x\), so the generalized posterior is exponential with rate \(x\). Under squared-error loss, its posterior mean and generalized Bayes rule are \(\delta(x)=1/x\).

The prior restrictions \([1/k,k]\), for \(k>1\), have finite mass \(\int_{1/k}^k d\theta/\theta=2\log k\). Normalizing each restriction therefore gives a proper prior. Nevertheless, the risk of \(\delta\) is infinite for every fixed \(\theta>0\). Indeed, \[ R(\theta,\delta)=\int_0^\infty (1/x-\theta)^2\theta e^{-\theta x}\,dx. \] For sufficiently small \(x>0\), \(1/x\geq2\theta\), so \((1/x-\theta)^2\geq1/(4x^2)\). Also, for \(0<x<1/\theta\), \(e^{-\theta x}\geq e^{-1}\). Hence the risk integral is bounded below, over a sufficiently small interval \((0,\varepsilon)\), by \(\theta e^{-1}\int_0^\varepsilon (4x^2)^{-1}dx=\infty\). Thus the integrated risk under every restricted prior is infinite. A proper sequence and a proper posterior exist, but the risk comparison needed for Blyth’s method is not available with this loss and rule.

Worked Example: Exact Bayes Rules in a Countable Model

Let \(\Theta=\{1,2,\ldots\}\), let \(\nu\) be counting measure, and observe \(X=\theta\) with probability one. Use actions \(a\in\Theta\) and zero-one loss \(L(\theta,a)=1_{\{a\ne\theta\}}\). Counting measure is sigma-finite, and \(E_k=\{1,\ldots,k\}\) has mass \(k\). The normalized restriction is the uniform proper prior \(\pi_k(\theta)=1/k\) on \(E_k\).

After observing \(X=x\), the generalized posterior places probability one on \(\theta=x\). The generalized Bayes rule is \(\delta(x)=x\), with risk \(R(\theta,\delta)=0\) for every \(\theta\). Under each \(\pi_k\), the observation also identifies the parameter exactly, so the proper-prior Bayes rule \(\delta_k(x)=x\) has risk zero on \(E_k\). Therefore \(\int\{R(\theta,\delta)-R(\theta,\delta_k)\}\,\tau_k(d\theta)=0\) for every \(k\), where \(\tau_k\) is counting measure restricted to \(E_k\). Here risk integrability and the unnormalized vanishing-gap condition both hold exactly.

Common Pitfall: Properness Is Not a Risk Bound

A frequent error is to argue that because each \(\pi_k\) is a probability distribution and each posterior is proper, all relevant integrated risks are finite and the Blyth gap must vanish. Neither conclusion follows. A proper prior can assign positive mass to parameters at which a rule has infinite risk, as the exponential example shows. Even when risks are finite for each \(k\), they can grow with \(k\), as in the normal example. Finally, a small gap averaged under \(\pi_k\) may still be large after multiplication by \(Z_k\).

The correction is to check the ingredients separately: establish properness of the priors; verify posterior normalizers where posterior comparisons are made; establish finite integrated risk for each rule on each weighted restriction; and evaluate the unnormalized gap \(G_k=Z_kD_k\) on the scale required by Blyth’s method. When posterior convergence is used, also check that the required risk comparison follows from more than pointwise convergence alone.

Choose finite-mass weights.
Construct \(w_k\nu\) with \(0<Z_k<\infty\) and \(w_k\uparrow1\) \(\nu\)-almost everywhere.

Check posterior quantities.
Verify that the generalized and proper-prior posterior normalizers are finite and positive at the data values used in the argument.

Check risk integrability.
Use Tonelli’s theorem for nonnegative losses, and establish finite integrated risk under each \(w_k\nu\) for every rule in the comparison.

Check the correct gap.
Translate the normalized gap \(D_k\) into \(G_k=Z_kD_k\), and verify the required unnormalized limit rather than inferring it from \(D_k\to0\).

Check Your Understanding

Use the definitions and risk conditions above to assess each claim.

Why does sigma-finiteness allow a sequence of proper priors formed by normalized restrictions?
For increasing weights \(w_k\), what does monotone convergence establish about the posterior normalizers \(m_k(x)\)? What does it not establish about integrated risk gaps?
If \(Z_k\to\infty\) and the normalized gap \(D_k\to0\), is it necessarily true that the unnormalized gap \(Z_kD_k\to0\)? Explain.
What additional risk property fails in the exponential-rate example, despite the existence of proper restricted priors and proper posteriors?
State a sufficient condition that guarantees finite integrated risk under each weighted restriction for a rule \(\delta\).