Jeffreys Priors for Trial Probabilities and Scales

Why Fisher Information Can Guide a Prior

A prior distribution can look different when the same parameter is expressed on a different scale. The earlier material on uniform priors showed why a density that is uniform in one parameterization need not be uniform in another. Jeffreys priors offer a specific response: for a scalar parameter, they use the square root of the model’s Fisher information. Because Fisher information changes with the parameterization in a particular way, this construction is invariant under smooth one-to-one reparameterizations.

This property is useful when specifying a prior for a clinical-trial event probability or a measurement scale, but it does not make the prior universally neutral. The prior depends on the likelihood model, can place appreciable mass near boundaries, and may be improper. With nuisance parameters, the precise Jeffreys construction also matters.

Definition: For a scalar parameter \(\theta\) with likelihood \(L(\theta)\), the Jeffreys prior has density \(\pi_J(\theta)\propto\sqrt{I(\theta)}\), where \(I(\theta)\) is the expected Fisher information for one observation, or the total expected Fisher information for the specified sample. The proportionality constant is chosen to normalize the density if its integral is finite.

Fisher information measures the expected curvature of the log likelihood, or equivalently the expected squared score under regularity conditions. The Jeffreys construction uses that curvature to assign prior density: regions in which the likelihood is more sensitive to the parameter receive more density per unit of that parameter. The result is not a claim that all parameter values are equally plausible.

Binomial Event Probabilities

Suppose \(Y\sim\operatorname{Binomial}(n,p)\), where \(p\) is the event probability and \(n\) is fixed. The model assumes independent Bernoulli trials with the same probability \(p\), and \(0<p<1\). For one Bernoulli observation \(Y_i\), the log likelihood, apart from a constant, is

\ell(p)=Y_i\log(p)+(1-Y_i)\log(1-p).

The score is \(Y_i/p-(1-Y_i)/(1-p)\). Taking its expected squared value under the Bernoulli model gives the Fisher information for one observation, \(I(p)=1/[p(1-p)]\). For \(n\) independent observations, the information is \(n/[p(1-p)]\). The factor \(n\) is constant with respect to \(p\), so it affects only the normalizing constant of the prior.

\pi_J(p)\propto\sqrt{\frac{1}{p(1-p)}}=\frac{1}{\sqrt{p(1-p)}}, \qquad 0<p<1.

This is the Beta\((1/2,1/2)\) density. It is proper: although the density grows without bound as \(p\) approaches 0 or 1, both boundary singularities are integrable. The prior is symmetric about \(p=1/2\), but it is not uniform and does not encode a belief that probabilities near the boundaries are impossible.

Worked Example: Jeffreys Prior for a Hypothetical Trial Event Rate

Suppose a hypothetical trial records whether each participant experiences an adverse event during follow-up. Let \(p\) denote the common event probability. The binomial model is appropriate here if participants’ event indicators are independent and share the same \(p\), with the number of participants fixed.

From the binomial Fisher information, the Jeffreys prior is \(\operatorname{Beta}(1/2,1/2)\). Suppose, hypothetically, that 6 of 20 participants experience the event. The likelihood is proportional to \(p^6(1-p)^{14}\). Multiplying by the prior density gives

\pi(p\mid Y=6)\propto p^6(1-p)^{14}p^{-1/2}(1-p)^{-1/2} =p^{5.5}(1-p)^{13.5}.

Therefore \(p\mid Y=6\sim\operatorname{Beta}(6.5,14.5)\). Its posterior mean is \(6.5/(6.5+14.5)=6.5/21\approx0.3095\), or about 31 events per 100 participants on average under this posterior. This is a posterior summary, not a guarantee about the event rate in a future trial; its interpretation relies on the binomial model and the prior.

Why the Scalar Prior Is Invariant

Let \(\eta=g(\theta)\) be a smooth, one-to-one reparameterization with nonzero derivative. A density must transform with the change-of-variable factor: \(\pi_\eta(\eta)=\pi_\theta(\theta)|d\theta/d\eta|\). Fisher information transforms as \(I_\eta(\eta)=I_\theta(\theta)(d\theta/d\eta)^2\). Applying the Jeffreys rule in the new parameter therefore gives

\sqrt{I_\eta(\eta)}\,d\eta =\sqrt{I_\theta(\theta)}\left|\frac{d\theta}{d\eta}\right|d\eta =\sqrt{I_\theta(\theta)}\,|d\theta|.

Thus the prior measure is the same under either coordinate: deriving the prior in \(\eta\) and transforming it back yields the prior derived in \(\theta\). This is the sense in which a scalar Jeffreys prior is invariant. It does not mean the density has the same algebraic shape in every parameterization.

Worked Example: Re-expressing the Binomial Prior on the Logit Scale

For an event probability \(p\), define the logit parameter \(\eta=\log\{p/(1-p)\}\). Then \(dp/d\eta=p(1-p)\). Starting from the Jeffreys density \(\pi_p(p)\propto[p(1-p)]^{-1/2}\), change variables:

\pi_\eta(\eta) =\pi_p(p)\left|\frac{dp}{d\eta}\right| \propto[p(1-p)]^{-1/2}p(1-p) =\sqrt{p(1-p)}.

The density on the logit scale is therefore proportional to \(\sqrt{p(1-p)}\), with \(p=1/(1+e^{-\eta})\). It is not constant in \(\eta\). Invariance means that this transformed density represents the same prior measure as the Beta\((1/2,1/2)\) density on \(p\); it does not mean that the prior is uniform on both scales.

Normal Scale When the Mean Is Known

Consider independent observations \(X_1,\ldots,X_n\sim N(\mu,\sigma^2)\), where the mean \(\mu\) is known and the standard deviation \(\sigma>0\) is the scalar parameter of interest. This model assumes normally distributed observations with a common, positive scale. The log likelihood, apart from a constant, is

\ell(\sigma)=-n\log(\sigma)-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(X_i-\mu)^2.

Writing \(Q=\sum_i(X_i-\mu)^2\), the score is \(-n/\sigma+Q/\sigma^3\). Under the model, \(E(Q)=n\sigma^2\). The expected squared score, or equivalently the negative expected second derivative of the log likelihood, is \(I(\sigma)=2n/\sigma^2\). Since \(2n\) is constant with respect to \(\sigma\), the scalar Jeffreys prior is

\pi_J(\sigma)\propto\frac{1}{\sigma}, \qquad \sigma>0.

This prior is improper because \(\int_0^\infty d\sigma/\sigma\) diverges. Improper means it cannot be normalized to have total probability one. It can still yield a proper posterior for particular data and models, but that must be checked rather than presumed. A transformation also makes its form clearer: for \(v=\sigma^2\), the density is \(\pi(v)\propto1/v\); for \(\lambda=\log\sigma\), it is uniform in \(\lambda\). Neither statement makes the prior proper.

Worked Example: Updating a Known-Mean Normal Scale Prior

Suppose, hypothetically, four independent measurements in millimetres follow \(X_i\sim N(9,\sigma^2)\), with the mean known to be 9 mm. The observed measurements are 8, 9, 10, and 11 mm. The model assumptions are independence, a common normal distribution, and a known mean. Here \(Q=(8-9)^2+(9-9)^2+(10-9)^2+(11-9)^2=6\ \mathrm{mm}^2\).

The likelihood is proportional to \(\sigma^{-4}\exp\{-6/(2\sigma^2)\}\). Multiplying by the Jeffreys prior \(1/\sigma\) gives the posterior kernel

\pi(\sigma\mid x)\propto\sigma^{-5}\exp\left(-\frac{3\ \mathrm{mm}^2}{\sigma^2}\right),\qquad \sigma>0.

This posterior is proper: it is integrable near zero because the exponential term tends to zero faster than any power, and integrable at infinity because the kernel behaves as \(\sigma^{-5}\). Its mode is found by differentiating its log kernel: \(-5/\sigma+6/\sigma^3=0\), so \(\sigma^2=6/5\ \mathrm{mm}^2\) and \(\sigma\approx1.095\ \mathrm{mm}\). The mode is a posterior summary under this model and prior; it is not a direct measurement of a fixed, known scale.

When the Normal Mean Is Unknown

The known-mean derivation must not be silently reused when the mean is unknown. For the joint parameter vector \((\mu,\sigma)\), the expected Fisher information matrix for \(n\) independent normal observations is diagonal, with entries \(n/\sigma^2\) and \(2n/\sigma^2\). The joint Jeffreys-rule prior is proportional to the square root of the determinant:

\pi_J(\mu,\sigma)\propto \sqrt{\det\begin{pmatrix}n/\sigma^2&0\\0&2n/\sigma^2\end{pmatrix}} =\frac{\sqrt{2}\,n}{\sigma^2} \propto\frac{1}{\sigma^2}.

This is a joint prior for \((\mu,\sigma)\), not the scalar prior \(1/\sigma\) derived with \(\mu\) fixed. Jeffreys-rule construction for a parameter vector and taking a scalar Jeffreys prior while treating the mean as fixed are distinct operations. They need not give the same scale prior when nuisance parameters are present.

Worked Example: Checking the Joint Normal Posterior

Suppose, hypothetically, that \(n\) independent measurements in millimetres follow a normal distribution with both \(\mu\) and \(\sigma\) unknown, and use the joint Jeffreys-rule prior \(1/\sigma^2\). The posterior kernel is proportional to \(\sigma^{-(n+2)}\exp\{-\sum_i(x_i-\mu)^2/(2\sigma^2)\}\). Integrating over \(\mu\) contributes a factor proportional to \(\sigma/\sqrt{n}\), so the marginal kernel for \(\sigma\) is proportional to

\sigma^{-(n+1)} \exp\left(-\frac{\sum_i(x_i-\bar{x})^2}{2\sigma^2}\right).

If at least two observations differ, then \(\sum_i(x_i-\bar{x})^2>0\): the exponential controls the integral near zero, while the power \(\sigma^{-(n+1)}\) is integrable at infinity for \(n>0\). The posterior is then proper. If all observations are identical, the residual sum of squares is zero, and the kernel has no exponential control near zero; its integral diverges there. This example shows why propriety depends on the data as well as on the prior.

Use, Alternatives, and Failure Modes

A uniform prior on \(p\) is \(\operatorname{Beta}(1,1)\), whereas the binomial Jeffreys prior is \(\operatorname{Beta}(1/2,1/2)\). The first is uniform per unit of probability; the second is determined by binomial Fisher information and is invariant under smooth reparameterization. For normal scale, the scalar Jeffreys prior \(1/\sigma\) is uniform per unit of \(\log\sigma\), but is improper. These alternatives answer different modeling choices; invariance alone does not establish that one is clinically appropriate.

In a trial, an event probability may be near zero or one, or investigators may have reliable historical knowledge about its plausible range. The Jeffreys prior’s integrable boundary singularities can matter in small samples, especially when there are no events or all participants experience the event. Its invariance does not encode clinical knowledge, resolve model misspecification, or ensure a proper posterior in every model. For a scale parameter, the improper prior additionally requires a posterior-propriety check before posterior probabilities or summaries are used.

Useful diagnostics include checking that the likelihood model matches the sampling structure, verifying posterior propriety, and examining whether inference is sensitive to the prior choice in a small or boundary-heavy data set. A common mistake is to say, “Jeffreys priors are invariant, so they are objective and need no checking.” The correction is that invariance is a mathematical property of the construction, not a guarantee of clinical suitability, robustness, or a proper posterior.

Key takeaway: For a binomial probability, the Jeffreys prior is \(\operatorname{Beta}(1/2,1/2)\). For a normal standard deviation with known mean, it is proportional to \(1/\sigma\). If the normal mean is unknown, the joint Jeffreys-rule prior is proportional to \(1/\sigma^2\). State the model and parameterization, and check posterior propriety.

Check Your Understanding

Answer each question, then compare with the solutions below.

For a hypothetical binomial trial with 3 events among 10 participants, what is the posterior distribution using the Jeffreys prior?
Why is the Jeffreys prior density for a binomial probability allowed to diverge at 0 and 1?
If \(v=\sigma^2\) and the known-mean normal scale prior is proportional to \(1/\sigma\), what is the corresponding density in \(v\), up to proportionality?
Why should the prior \(1/\sigma\) not automatically be used as the joint Jeffreys-rule prior when both the normal mean and scale are unknown?

Solutions. (1) The posterior is \(\operatorname{Beta}(3.5,7.5)\), by adding the observed 3 events and 7 non-events to the prior’s two shape parameters of \(1/2\). (2) The singularities are integrable: the density behaves like \(p^{-1/2}\) near 0 and \((1-p)^{-1/2}\) near 1, and each has a finite integral over a neighborhood of its boundary. (3) Since \(\sigma=\sqrt{v}\) and \(|d\sigma/dv|=1/(2\sqrt{v})\), \(\pi(v)\propto v^{-1/2}/(2\sqrt{v})\propto1/v\). (4) With both parameters unknown, the joint information matrix gives a determinant whose square root is proportional to \(1/\sigma^2\); the scalar known-mean calculation and the joint construction are different.