Null Calibration Checks for Association Tests

Why Check Null Calibration?

An association test is calibrated when, under its null hypothesis and stated assumptions, its test statistic follows the reference distribution used to calculate its p-value. If that distribution is wrong, the reported p-values can be too small or too large even when there is no genetic effect. In a genome-wide analysis, even a small systematic excess of small p-values can produce many apparently compelling findings.

A calibration check asks whether a particular analysis procedure behaves as intended under a genetic null that reflects the study design. It is not the same as deciding whether a particular variant is associated, and a genome-wide excess of small p-values does not by itself reveal its cause. True associations, population structure, relatedness, technical artifacts, and model misspecification can all affect the observed distribution.

What a Well-Calibrated Null Looks Like

For a single continuously distributed test statistic with a correct null reference distribution, its p-value is uniform under the null. Thus, among \(m\) tests that are all truly null, the expected number with p-values at most \(a\) is \(ma\). For example, among 10,000 null tests, about 500 are expected to have p-values at most 0.05. This is an expectation, not a promise that every set of 10,000 tests will contain exactly 500 such p-values. Dependence between genetic variants affects the variability of the count, although it does not change the marginal null expectation when each test is calibrated.

The uniformity result follows directly from the definition of a continuous p-value. If \(T\) has null cumulative distribution function \(F_0\), and a one-sided p-value is \(P(T \geq t)\), then \(P(T \geq T_{\mathrm{obs}})=1-F_0(T_{\mathrm{obs}})\). Since \(F_0(T)\) is uniform on 0 to 1 for a continuous null distribution, its complement is also uniform. The same principle applies to a two-sided p-value when it is calculated from the correct null distribution.

In common association tests, a statistic such as a score or Wald statistic may be compared with a chi-square distribution. For a one-degree-of-freedom test, the expected median of the null chi-square statistics is approximately 0.4549. This provides a reference for a quantile-quantile (QQ) plot or a simple inflation summary. A QQ plot compares observed test-statistic quantiles with the corresponding null quantiles. Systematic upward deviation indicates that the observed statistics are larger than the reference predicts; it does not identify why.

Design-Preserving Null Simulations and Resampling

A useful null check preserves the features that affect the distribution of the test statistic, while generating outcomes under no genetic effect for the tested association. Depending on the analysis, those features can include covariates, ancestry structure, relatedness, case-control sampling, missingness, and the phenotype model. The association procedure is then rerun on each generated or resampled data set. If the analysis includes data-dependent steps, those steps should generally be repeated in each replicate when their selection uncertainty affects the test.

The simulation method must match the null hypothesis and design. For example, simulating a binary phenotype independently for each participant from fitted null-model probabilities is appropriate only if the model's conditional independence assumptions are adequate. If outcomes are correlated within families, independent Bernoulli draws do not preserve that dependence; the null-generation procedure must account for it. Likewise, randomly shuffling phenotypes across all participants is not generally valid when phenotype distributions differ by covariates or ancestry.

Permutation is one possible resampling approach, but its validity depends on exchangeability: under the null, the observations being permuted must have the same joint distribution after the permitted rearrangements. If exchangeability holds only within strata, permutation should be restricted to those strata. Residual-permutation methods can sometimes accommodate covariates, but their validity also depends on the model and residual structure; they are not a universal fix for confounding or dependence.

Genome-Wide Inflation: A Diagnostic, Not a Diagnosis

A common summary for one-degree-of-freedom chi-square association statistics is the genomic inflation factor, often denoted \(\lambda\). It is the median observed chi-square statistic divided by the median expected under the null:

A value above 1 indicates that the median observed statistic exceeds the one-degree-of-freedom null median. It does not establish that a test is miscalibrated. If many variants have genuine effects, their contribution can raise the median. Population structure, relatedness not adequately modeled, or other systematic errors can also raise it. Conversely, a value near 1 does not guarantee that every part of the p-value distribution is well behaved: problems concentrated in a tail or in a subset of variants may be obscured by a median summary.

In particular, an observed genome-wide shift and a failure of the test procedure under its intended null are related but distinct findings. Simulations generated under a credible null model assess the procedure conditional on the modeled design. Genome-wide summaries describe the actual analyzed data, which may contain true effects as well as unwanted sources of association. Neither a QQ plot nor \(\lambda\) alone can assign the excess to one source.

Quantifying Calibration in Simulations

A direct calibration check is to estimate the rejection rate at one or more nominal levels. If \(B\) independent null simulations are run and \(R\) tests reject at level \(\alpha\), the empirical rejection rate is \(R/B\). Under exact calibration, \(R\) has a binomial distribution with \(B\) trials and success probability \(\alpha\), provided the simulation replicates are independent and generated under the specified null. Its expected value is \(B\alpha\), and its standard error as a rate is approximately \(\sqrt{\alpha(1-\alpha)/B}\). A deviation from \(B\alpha\) should be judged against this simulation variability, as well as uncertainty in the fitted null model.

Common Mistakes and Practical Interpretation

• Calling every excess of small p-values a calibration failure. The reasoning is that a uniform null distribution should have few small p-values, so any excess must mean the test is broken. The correction is that uniformity applies to true null tests; real genetic effects and a mixture of null and non-null variants can also alter the genome-wide distribution.
• Using unrestricted permutation when exchangeability fails. Shuffling outcomes across ancestry groups, covariate levels, or related participants can destroy features that the null distribution must retain. Use a justified restricted permutation or a null simulation that models those features.
• Interpreting \(\lambda\) as a universal correction factor. A value above 1 is a diagnostic summary, not proof of confounding and not a complete description of calibration. Examine the full distribution and compare it with credible null simulations.
• Treating a finite simulation estimate as exact. A small number of replicates gives substantial Monte Carlo uncertainty, particularly for very small tail probabilities. Report the number of replicates and increase it when the question concerns rare tail events.