Why Summarize Posterior Uncertainty?
A credible set gives a useful count: how many variants are needed to accumulate a chosen amount of posterior probability? But that count does not show how probability is distributed among those variants. A set of four variants might have one variant with most of the support, or it might spread support almost evenly across all four. These are different forms of uncertainty, even if the credible-set sizes match.
Posterior entropy summarizes the spread of a probability distribution over candidate variants. It uses all the probabilities, not only the variants inside a thresholded credible set. It is therefore best interpreted alongside, not instead of, credible-set size and the largest variant posterior probability.
This definition concerns a probability distribution over mutually exclusive candidate variants for one signal. In a model that allows multiple causal signals, marginal posterior inclusion probabilities (PIPs) across all variants generally need not sum to one: more than one variant may be causal. Such PIPs cannot be inserted directly into this formula as though they formed one distribution. Use signal-specific probabilities that sum to one, or define a different distribution whose categories and interpretation are explicit.
If the model allows a possibility that the signal is absent, the distribution must also account for that possibility if the goal is to summarize uncertainty across both presence and variant identity. Alternatively, one can calculate entropy conditional on the signal being present, as in the definition above. These summaries answer different questions, so the conditioning should be stated.
Calculating Entropy and Reading Its Scale
Each variant contributes \(-q_j\ln(q_j)\). A probability near one contributes little entropy; a probability near zero also contributes little, although many small probabilities together can matter. Entropy is zero when one variant has probability one and all others have probability zero. For \(m\) candidate variants, its maximum is \(\ln(m)\), attained when all \(m\) probabilities equal \(1/m\).
A useful transformation is the effective number of variants, \(\exp(H)\). This is the number of equally probable variants that would have the same entropy. It is not a count of variants with nonzero probability, nor does it imply that exactly that many variants are causal. For example, an entropy of \(\ln(4)\) corresponds to an effective number of four, whether the distribution is exactly uniform over four variants or has another shape with the same entropy.
Worked Example: A Concentrated Distribution With Four Variants
Suppose a hypothetical single-signal analysis gives posterior probabilities \(0.70, 0.10, 0.10, 0.10\) for four candidate variants. They sum to one, so they form a valid distribution for the defined calculation.
The entropy is $$ \begin{aligned} H &=-[0.70\ln(0.70)+3\{0.10\ln(0.10)\}]\\ &=-(0.70)(-0.356675)-3(0.10)(-2.302585)\\ &=0.249672+0.690776\\ &=0.940448\ \text{nats}. \end{aligned} $$ The effective number is \(\exp(0.940448)\approx 2.561\). Although four variants have positive probability, the distribution has the same entropy as a uniform distribution over about 2.56 variants.
For a 95% credible set, order the probabilities from largest to smallest and include variants until their cumulative probability is at least 0.95. The cumulative probabilities here are 0.70, 0.80, 0.90, and 1.00, so the 95% set contains all four variants. The set size alone does not reveal that the first variant carries 70% of the posterior support; entropy reflects the overall allocation.
Worked Example: Same Credible-Set Size, More Diffuse Support
Consider a second hypothetical distribution over four variants: \(0.40, 0.30, 0.20, 0.10\). It also sums to one. The 95% credible set again contains all four variants, because the first three sum to 0.90 and the fourth raises the total to 1.00.
Calculate entropy using the unrounded component values: $$ \begin{aligned} H &=-[0.40\ln(0.40)+0.30\ln(0.30)+0.20\ln(0.20)+0.10\ln(0.10)]\\ &=0.3665162927+0.3611918413+0.3218875825+0.2302585093\\ &=1.2798542258\\ &\approx 1.279854\ \text{nats}. \end{aligned} $$ The effective number is \(\exp(1.2798542258)\approx 3.596\). This is higher than the first example’s 2.561, so support is more diffuse across the four candidates. Both 95% credible sets have size four, but their entropy values and largest posterior probabilities differ.
The contrast illustrates why the set size should not be read as a complete measure of uncertainty. A thresholded set records how many variants are needed to reach a target probability; entropy records the shape of the full distribution.
Why the Maximum Entropy Is \(\ln(m)\)
The upper bound follows from the concavity of the logarithm. For positive \(q_j\), apply Jensen’s inequality to the values \(1/q_j\), weighted by \(q_j\): $$ \begin{aligned} H(q) &=\sum_{j:q_j>0}q_j\ln(1/q_j)\\ &\leq \ln\left(\sum_{j:q_j>0}q_j(1/q_j)\right)\\ &=\ln(r), \end{aligned} $$ where \(r\) is the number of positive probabilities. Since \(r\leq m\), \(H(q)\leq\ln(m)\). Equality in the first inequality requires all \(1/q_j\) with positive weights to be equal, so the positive probabilities must be equal. The maximum \(\ln(m)\) occurs when all \(m\) candidates have probability \(1/m\). At the other extreme, if one probability is one, then \(H=-1\ln(1)=0\).
This bound is also a useful arithmetic check. A computed entropy should be nonnegative and no larger than \(\ln(m)\), allowing for rounding. If it falls outside that range, check that the probabilities sum to one, that the logarithms and signs were handled correctly, and that the same candidate distribution was used throughout.
Worked Example: A Small Credible Set With Residual Uncertainty
In a third hypothetical analysis, four variants have probabilities \(0.82, 0.08, 0.06, 0.04\). These sum to one. For a 95% credible set, the first two sum to 0.90; adding the third gives 0.96. Thus the set has size three, and its posterior mass is 0.96.
Entropy includes the fourth variant as well as the three in the set: $$ \begin{aligned} H &=-[0.82\ln(0.82)+0.08\ln(0.08)+0.06\ln(0.06)+0.04\ln(0.04)]\\ &=0.162730+0.202058+0.168805+0.128755\\ &\approx 0.662348\ \text{nats}. \end{aligned} $$ The effective number is \(\exp(0.662348)\approx 1.939\). The 95% set size of three and the effective number of about 1.94 describe different features: the former is the number needed to cross a specified cumulative-probability threshold, while the latter summarizes the concentration of the whole distribution.
Entropy Alongside Other Summaries
A maximum posterior probability is easy to interpret: it reports the largest single-variant probability. It can miss how the remaining support is arranged. Credible-set size is tied to a chosen probability threshold, and small changes in probabilities near that threshold can change the set size. Entropy uses the full probability vector, but its single number does not identify which variants carry support or whether nearby variants are highly correlated.
| Summary | What it describes | What it does not describe by itself |
|---|---|---|
| Maximum variant posterior probability | Support for the leading candidate | How the remaining probability is distributed |
| Credible-set size at a stated target | Number of variants needed to reach the target posterior mass | How evenly support is divided within the set |
| Posterior entropy | Overall spread of the full variant probability distribution | Variant identity, biological function, or whether the model is well specified |
| Effective number, \(\exp(H)\) | Entropy expressed as an equivalent number of equally supported variants | A literal count of plausible or causal variants |
In reporting, give enough context to make the summary reproducible: identify the signal, state that the probabilities are normalized over candidate variants, report the logarithm base or entropy units, and specify the credible-set target. The candidate universe matters. Adding many variants with tiny probabilities may change entropy slightly, and changing which variants are included or how probabilities were normalized can change it more substantially.
Diagnostics and Common Pitfalls
Before interpreting entropy, verify that the input is the intended posterior distribution. The probabilities should be finite, nonnegative, and sum to one, up to numerical rounding. If the sum is not approximately one, determine whether the values are marginal PIPs, probabilities for a truncated subset, or weights that still require normalization. Normalizing a truncated subset changes the question: it describes uncertainty conditional on the variant being among those retained, not uncertainty over the original candidate universe.
- Using multi-signal PIPs as one distribution. PIPs may sum to more than one when multiple variants can be causal. Do not calculate the defined entropy from them without specifying a different interpretation.
- Dropping variants outside the credible set. The entropy in this tutorial is calculated over all candidates in the signal-specific distribution. Restricting to the set and renormalizing gives a conditional, within-set entropy, which is not the same summary.
- Calling the effective number a causal-variant count. It is an entropy-equivalent number, not an estimate that exactly that many variants are causal.
- Comparing values without checking the candidate universe. Entropy depends on the probability distribution and its categories. Comparisons are most interpretable when candidate definitions and conditioning are comparable.
- Rounding terms too early. Keep adequate precision in component calculations and round the final sum. For example, the entropy of \(0.40,0.30,0.20,0.10\) is 1.2798542258 nats, which rounds to 1.279854, not 1.279855.
Entropy is a summary of posterior uncertainty under the specified fine-mapping model and its inputs. A low value does not establish that the leading variant is causal in reality, and a high value does not by itself identify why support is diffuse. It should be read with posterior probabilities, credible-set construction, and the model assumptions in view.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- For probabilities \(0.50, 0.30, 0.20\), calculate the entropy to three decimal places and give the effective number to two decimal places.
- For probabilities \(0.60, 0.25, 0.15\), what is the size of the 90% credible set? Explain why its size does not give the entropy.
- Why is it generally invalid to calculate this entropy directly from marginal PIPs for a model with multiple causal variants?
- If the probabilities are uniform over eight variants, what are the entropy and effective number?
- A researcher removes variants outside a 95% credible set, renormalizes the probabilities inside it, and reports the result as the full-distribution entropy. What is the error?
Solutions. (1) \(H=-[0.50\ln(0.50)+0.30\ln(0.30)+0.20\ln(0.20)]=1.029653\), or 1.030 nats; \(\exp(H)\approx2.80\). (2) The first two probabilities sum to 0.85, and adding the third gives 1.00, so the 90% set has size three. Its size states only the count needed to cross 0.90; it does not encode the distribution’s full shape. (3) Multiple variants may each be causal, so the marginal PIPs need not sum to one and are not mutually exclusive categories of one variant-identity distribution. (4) \(H=\ln(8)\approx2.079\) nats and \(\exp(H)=8\). (5) The calculation describes uncertainty conditional on being inside the selected set, not entropy over all candidates; omitted probability mass and variants must be accounted for to describe the full distribution.