Why the Choice of Null Distribution Matters
A test statistic is interpreted relative to a null distribution: the distribution it would follow if the tested hypothesis were true. In genomic analyses, thousands of statistics are considered together, so even modest errors in that reference distribution can alter p-values, rankings, and estimated posterior null probabilities. A familiar reference such as the standard normal distribution is not automatically appropriate merely because a statistic is called a \(Z\)-score.
The key distinction is between a theoretical null, derived from the assumptions and sampling behavior of a specified test, and an empirical null, modeled from the observed collection of test statistics as an estimate of the distribution of null statistics in that collection. An empirical null can accommodate a center or scale different from the theoretical reference, but it is not automatically more correct. Its interpretation depends on whether the observed data contain enough information to distinguish null behavior from alternatives and other sources of variation.
Models and Assumptions
For a test statistic \(Z\), a standard theoretical null in a large-sample analysis is \(Z\mid H_0 \sim N(0,1)\). This reference is justified when the test’s null distribution is exactly standard normal or is adequately approximated by it. For example, an asymptotic normal test may rely on a sufficiently large sample, a correctly specified model, and appropriate variance estimation. If those conditions fail, the standard normal reference may not give valid tail probabilities.
An empirical normal null instead uses \(Z\mid H_0 \sim N(\mu_0,\sigma_0^2)\). The model says that null statistics in the collection are approximately normal with center \(\mu_0\) and standard deviation \(\sigma_0\). A location shift can arise when statistics are systematically displaced from zero; a scale change can arise when their variability differs from the theoretical reference. Neither feature alone identifies the cause. Confounding, unmodeled heterogeneity, technical artifacts, dependence, and imperfect standard-error estimates are among possible explanations.
The two-group mixture framework makes the distinction explicit. If a proportion \(\pi_0\) of tests are null, and the remaining tests follow an alternative density \(f_1\), the marginal density of statistics is
Here \(f_0\) is the chosen null density and \(f\) is the overall density of observed statistics. The theoretical-null choice fixes \(f_0\) from the test’s reference model; the empirical-null approach estimates or specifies a different \(f_0\) to represent null behavior in the collection. The alternative density and \(\pi_0\) are distinct parts of the mixture: changing the null is not the same operation as changing the proportion of null tests.
An empirical null is most interpretable when its model is a reasonable description of the null component and the collection provides information about that component. In practice, this often requires that null statistics are sufficiently represented and that alternatives do not overwhelm or strongly distort the part of the distribution used to learn the null. These are substantive assumptions, not guaranteed consequences of fitting a distribution to observed values. A fitted center and scale do not prove that the tests are null or that the assumed model is appropriate.
Worked Example: A Shifted Null Changes a Two-Sided P-Value
Suppose a hypothetical genomic analysis reports an observed statistic \(z=2.00\). Compare a theoretical null \(N(0,1)\) with a hypothetical empirical null \(N(0.25,1)\). Assume that each candidate null is normal and that the two-sided departure is measured as distance from that null’s center. For a normal null \(N(\mu_0,\sigma_0^2)\), the corresponding two-sided tail probability is
where \(\Phi\) is the standard normal cumulative distribution function. Under the theoretical null, the standardized distance is \(|2.00-0|/1=2.00\), so
Under the shifted null, the distance from its center is \(|2.00-0.25|=1.75\), giving
The observed statistic has a larger tail probability under the shifted reference. The difference follows from the changed center, not from a change in the observed statistic. These calculations are conditional on treating each candidate null as the correct reference; the example does not establish which null is appropriate.
Center and Scale Have Different Effects
For a normal null with center \(\mu_0\) and standard deviation \(\sigma_0\), the standardized distance is \(|z-\mu_0|/\sigma_0\). A center shift changes the numerator, while a scale change alters the denominator. Holding the observed statistic fixed, increasing \(\sigma_0\) makes the standardized distance smaller and the two-sided tail probability larger. Moving \(\mu_0\) toward \(z\) has the same direction of effect; moving it away increases the standardized distance and reduces the tail probability.
These are statements about a statistic’s tail area under a selected null model. They do not say that the statistic’s biological effect has changed. Nor does a larger p-value alone show that the empirical null is preferable: a null model should be selected on statistical and scientific grounds, not because it yields a desired number of discoveries.
Worked Example: A Broader Null Changes the Tail Area
Consider a hypothetical observed statistic \(z=2.40\). Compare a theoretical null \(N(0,1)\) with a centered empirical null \(N(0,1.30^2)\). Assume both normal models are being used as two-sided references centered at zero. Under the theoretical null,
For the broader empirical null, standardize using its standard deviation:
The wider reference distribution assigns more probability to values at least this far from zero, so its p-value is larger. The calculation assumes the scale 1.30 is a suitable description of null variability. If that scale instead reflects a mixture of null and alternative statistics, treating it as a null scale could make the resulting tail areas misleading.
Consequences for Local False Discovery Rates
The null choice also affects the local false discovery rate, the posterior probability that a test is null given its statistic, within the two-group model. For densities \(f_0\) and \(f\), Bayes’ rule gives
This quantity depends directly on the null density at the observed value, as well as on the alternative density and null proportion. Thus replacing a theoretical null with an empirical one can change local false discovery rates even when the observed statistics and the alternative model are held fixed. A p-value is a tail probability under a null; a local false discovery rate is a posterior probability under a mixture model. They answer different questions.
Worked Example: The Null Density Changes a Local False Discovery Rate
For illustration, suppose a hypothetical two-group model has \(\pi_0=0.90\), \(z=2.00\), and alternative density \(f_1\) equal to the density of \(N(2,1)\). Compare theoretical null \(N(0,1)\) with empirical null \(N(0,1.2^2)\). This assumes that the mixture model, including the stated alternative density and null proportion, is appropriate. At \(z=2.00\), the alternative density is \(f_1(2)=0.3989\). Under the theoretical null, \(f_0(2)=0.0540\), so
Under the broader empirical null, the normal density is
and therefore
In this specified model, the broader null assigns more density to \(z=2\), increasing the posterior null probability there. The numerical result is not a general rule that empirical nulls always increase local false discovery rates; the result depends on the densities and mixture proportion.
When Each Reference Is Appropriate
A theoretical null is attractive when the test is correctly specified and its reference distribution is well justified. It is transparent, has a direct connection to the test’s null hypothesis, and avoids estimating extra location or scale parameters from the same collection of statistics. If these assumptions hold, replacing it with a fitted empirical distribution can add uncertainty and may absorb real signals into the null.
An empirical null can be useful when a common theoretical reference appears systematically miscentered or mis-scaled for a collection of tests and there is a defensible reason to model the null component differently. Its purpose is to represent null behavior in that collection, not simply to make the histogram look normal or to force a preferred rate of discoveries. Since the null parameters are estimated, plug-in calculations typically treat them as known when they are not; uncertainty in those estimates can affect tail probabilities and downstream posterior quantities.
| Feature | Theoretical null | Empirical null |
|---|---|---|
| Source of reference | Test assumptions and sampling theory | Model for null behavior in the observed collection |
| Center and scale | Fixed by the stated reference, often zero and one for a normal statistic | May be estimated or specified differently from the theoretical values |
| Main strength | Clear test-based interpretation when assumptions hold | Can represent systematic displacement or excess null variability |
| Main vulnerability | Miscalibration if test assumptions fail | Confounding of null behavior with alternatives or other heterogeneity |
Diagnostics, Failure Modes, and Limits
A collection of statistics whose central bulk is visibly displaced or more dispersed than the theoretical reference can motivate closer investigation of the null model. That pattern is a warning, not a diagnosis. A large number of non-null tests, many weak alternatives, dependence, or data-quality problems can also affect the observed distribution. In particular, dependence among genomic tests does not by itself invalidate every marginal null distribution; it can, however, complicate the distribution of the collection and the estimation or interpretation of a shared null model.
The empirical-null approach is limited by identifiability: the overall density \(f\) alone does not uniquely determine \(\pi_0\), \(f_0\), and \(f_1\) without additional modeling assumptions. A central region may be enriched for null tests, but enrichment is not proof that all central observations are null. A normal empirical null may also be a poor approximation if the null statistics are skewed, heavy-tailed, or otherwise non-normal. Tail conclusions can be especially sensitive to such misspecification.
The main practical alternatives are to retain a justified theoretical null, use a defensible empirical null, or revise the test or data model when neither reference is credible. They are not interchangeable corrections. If a test’s own assumptions fail, estimating a new center and scale may conceal the problem rather than fix it. Empirical-null p-values and local false discovery rates should therefore be described as conditional on the fitted null model, with appropriate caution about estimation uncertainty and model adequacy.
Common mistake: “The theoretical-null histogram is wider than standard normal, so the empirical null must be the true null.” The reasoning confuses a feature of the observed mixture with a property of its latent null component. The correction is to treat excess width as a diagnostic signal, then assess whether a wider null is scientifically and statistically defensible; alternatives and other sources of variation can also widen the observed distribution.
Identify the theoretical null distribution and the assumptions under which it applies to the statistic.
Look for systematic center or scale discrepancies, while recognizing that the mixture of nulls and alternatives also shapes the observed distribution.
If an empirical null is used, state its form and interpret resulting tail areas or local false discovery rates as conditional on that model.
Check Your Understanding
Answer each question, then compare your work with the solutions.
- For \(z=2.10\), calculate the two-sided p-value under \(N(0,1)\), then under \(N(0,1.2^2)\).
- A test statistic is \(z=-1.80\), and a proposed empirical null is \(N(-0.20,1.10^2)\). Calculate its two-sided tail probability using distance from the null center.
- Why does an observed collection wider than \(N(0,1)\) not prove that its null component has standard deviation greater than one?
- In the two-group model, which quantities are needed to calculate \(\operatorname{lfdr}(z)\), and how can changing \(f_0\) affect it?
Solutions. (1) Under \(N(0,1)\), \(p=2\{1-\Phi(2.10)\}\approx0.0357\). Under \(N(0,1.2^2)\), the standardized distance is \(2.10/1.20=1.75\), so \(p=2\{1-\Phi(1.75)\}\approx0.0801\). (2) The distance from the proposed center is \(|-1.80-(-0.20)|=1.60\); dividing by 1.10 gives \(1.4545\). Thus \(p=2\{1-\Phi(1.4545)\}\approx0.1458\). (3) The observed distribution is a mixture of null and alternative statistics and may also reflect dependence or other heterogeneity. Its width need not equal the scale of the latent null component. (4) The local false discovery rate requires \(\pi_0\), \(f_0(z)\), and \(f_1(z)\), combined as \(\pi_0f_0(z)/[\pi_0f_0(z)+(1-\pi_0)f_1(z)]\). Changing \(f_0(z)\) changes both the numerator and denominator, so the posterior null probability can change even when the observed statistic and other model components remain fixed.