Choosing Alternative Distributions for Test Statistics

Why the Alternative Distribution Matters

In a large genomic analysis, each test produces a statistic such as a z-score. The null distribution describes z-scores when the null hypothesis is true; the alternative distribution describes them when it is false. Choosing the alternative distribution is consequential: it determines how much probability the model assigns to an observed score, including a score far out in the tails. That density contributes directly to the posterior probability that the test is null.

The choice is not simply between a “light-tailed” and a “heavy-tailed” curve. A single normal distribution, a mixture of normal distributions, and a flexible density estimate can describe different forms of heterogeneity among nonnull tests. Each can fit some data well and misrepresent other data, especially where observations are sparse. The goal is to select and assess a model that represents the distribution of nonnull test statistics on the scale being analyzed.

This formula presumes that the null proportion and both densities have been specified or estimated for the population of tests being modeled. In genomic applications, tests are often treated as draws from a common two-group distribution for this purpose. Dependence between tests may matter for estimating the model and its uncertainty, but it does not change the pointwise Bayes formula once the marginal model is specified. If the null is an empirical normal distribution rather than the theoretical standard normal, \(f_0\) must use that empirical center and scale.

Three Ways to Represent the Alternative

A single normal alternative uses \(f_1(z)\) from one normal distribution, often with a shifted mean, a variance different from the null variance, or both. For example, \(Z\mid\text{nonnull}\sim N(\mu_1,\sigma_1^2)\). This is compact and easy to interpret, but it assumes one bell-shaped alternative population. It can be inadequate if nonnull tests include a range of effect sizes or both weak and strong signals.

A mixture-normal alternative represents that heterogeneity as a weighted sum of normal densities: $$ f_1(z)=\sum_{k=1}^{K}w_k\phi(z;\mu_k,\sigma_k^2), \qquad w_k\geq 0,\quad \sum_{k=1}^{K}w_k=1, $$ where \(\phi(z;\mu_k,\sigma_k^2)\) is the normal density with mean \(\mu_k\) and variance \(\sigma_k^2\). A component can represent a subgroup with a particular location and spread. A broad component can give the mixture substantially more tail density than a narrow component, even if its weight is small. A mixture is more adaptable than one normal curve, but too many components can make the fit unstable or difficult to interpret.

A flexible alternative density estimates \(f_1\) without committing to a fixed number of normal components. For example, a kernel density estimate smooths a distribution of values around the observed nonnull statistics; other smooth density estimators use basis functions or constraints. The earlier material on nonparametric estimation of the alternative density covers how such estimates can be constructed. Here the key issue is how their fitted shape, particularly in the tails, affects posterior null probabilities.

How Tail Fit Changes the Posterior

At a fixed observed \(z\), hold \(f_0(z)\) and \(\pi_0\) fixed. The local false discovery rate is a decreasing function of \(f_1(z)\). To see this, write \(A=\pi_0f_0(z)\) and \(B=(1-\pi_0)f_1(z)\); then \(\operatorname{lfdr}(z)=A/(A+B)\). Differentiating with respect to \(f_1(z)\) gives

when \(0<\pi_0<1\) and \(f_0(z)>0\). Thus, a model assigning more alternative density at that score yields a lower posterior null probability, all else equal. This is a pointwise comparison: the relevant quantity is \(f_1(z)\), not merely the probability that an alternative score exceeds a threshold. Two fitted alternatives can have similar overall tail probabilities but different densities at a particular observed score.

The effect is often clearest for extreme scores. If an alternative curve is too light-tailed, it may assign very little density to a large \(|z|\); the null probability in the numerator can then remain relatively important compared with the alternative contribution. A model with more tail density can instead assign that score more plausibly to the nonnull group. The reverse problem is possible too: an overly heavy-tailed alternative can assign excessive density to extreme scores and drive posterior null probabilities down too far.

Mixtures and Broad Tail Components

A mixture can put substantial density in the tails without requiring every nonnull test to come from a broad distribution. The weights determine how much probability each component contributes; at a particular score, the contribution also depends on that component’s density there. For this reason, a component with a modest weight may dominate the fitted alternative density at an extreme score.

Flexible Fits and Tail Diagnostics

A flexible estimate can follow features that a single normal or a small mixture misses, but flexibility does not guarantee accurate tail probabilities or densities. There may be few observations in the far tails, and smoothing can spread mass into or away from a region where the data are sparse. A fitted density should be nonnegative, integrate to one, and be evaluated on the same statistic scale as the null density. Its tail behavior should also be checked for sensitivity to reasonable fitting choices.

Useful diagnostics for choosing among these representations include overlaying fitted alternative densities on the portion of the statistic distribution attributed to nonnull tests, checking whether fitted curves reproduce the observed tail behavior, and examining how estimates at scientifically relevant scores change under plausible model or smoothing choices. Such checks are difficult when the null and alternative overlap heavily: the nonnull observations are not directly labeled, so the fitted \(f_1\) is inferred within the two-group model rather than read off from a known set of alternative tests.

A broad alternative should not be selected merely because it makes large scores appear more convincingly nonnull. Equally, a narrow normal should not be retained for convenience if its tail is visibly inadequate. The estimate of \(\pi_0\), the null density, and the alternative density interact in the posterior formula. A comparison that changes only \(f_1\) is useful for isolating the tail-fit effect, but real model fits may change more than one quantity at once.

Common Mistake and Practical Choice

Wrong reasoning: “The mixture has a 20% broad component, so every extreme z-score has a 20% chance of coming from that component; therefore its local false discovery rate is 20%.” This confuses a mixture weight with a posterior probability for a particular observation. The weight is a prior component proportion within the alternative. The posterior null probability depends on the null and alternative densities at the observed score, as well as on \(\pi_0\).

Correction: Evaluate the full \(f_1(z)\) as the weighted sum of component densities, then substitute it and \(f_0(z)\) into the two-group posterior formula. A component’s effect at \(z\) depends on both its weight and its density there. The same principle applies when \(f_1\) is estimated flexibly.

A single normal is a reasonable starting representation when the alternative distribution is approximately unimodal and symmetric on the chosen scale. A mixture is useful when there is substantive or empirical evidence of groups with different locations or spreads. A flexible density can be attractive when these shapes are not well captured by a small mixture, provided the tail estimate is stable enough for the intended use. No representation is universally preferable: assess fit where the posterior calculations will rely on it, and report sensitivity when plausible alternatives yield materially different results.