Null Calibration Diagnostics for Gene-Based Tests

Why Check Null Calibration?

A gene-based test is calibrated when, under its null hypothesis and the conditions it assumes, its reported p-values produce the expected rejection rate. For example, a valid test at significance level \(\alpha=0.05\) should reject about 5% of repeated null data sets. Calibration matters because a method that rejects too often can produce an excess of apparently significant genes even when none has an effect. A conservative method rejects less often than its nominal level, which can reduce sensitivity.

Burden tests, SKAT, and omnibus procedures can rely on different approximations or combinations of component tests. Their calibration can be affected by features such as rare alleles, linkage disequilibrium, covariate adjustment, imbalanced case-control sampling, or small samples. Null simulations provide a direct way to examine whether the complete analysis procedure behaves as intended in a specified design. They diagnose performance under the simulated conditions; they do not establish that those conditions perfectly represent a real study.

Designing a Null Simulation

A useful calibration simulation reproduces the design features that matter to the test while generating outcomes under no genetic association. One common approach holds the observed genotype matrix fixed and simulates phenotypes under a fitted null model containing the covariates but no gene effect. This preserves the observed allele frequencies, linkage disequilibrium, and variant-set structure. It assesses calibration conditional on those genotypes, not on a hypothetical new sample of genotypes.

For a quantitative outcome, the null outcome model might specify a mean determined by covariates and an appropriate residual distribution. For a binary outcome, it might specify null disease probabilities conditional on covariates, using a logistic model. The simulation should reflect important design features such as case-control sampling or relatedness when these affect the analysis. A simple permutation of outcomes is not automatically valid: unrestricted permutation requires exchangeability under the null, which can fail when covariates affect the outcome or when observations are related.

For each simulated replicate, apply the analysis pipeline being assessed: fit the required null model, calculate the burden, SKAT, or omnibus statistic, and obtain its reported p-value using the intended procedure. Keep prespecified genotypes, variant sets, and weights fixed. If filtering, weight selection, or another step is data-dependent in the intended analysis, repeat that step in each replicate; otherwise the simulation may assess a simpler procedure than the one actually used.

Empirical Type I Error and Simulation Uncertainty

Suppose \(B\) independent null replicates produce p-values \(p_1,\ldots,p_B\). At a chosen nominal level \(\alpha\), the empirical type I error estimate is the fraction of replicates that reject:

where \(I(\cdot)\) is 1 when its condition is true and 0 otherwise. If the replicates are independent and the true null rejection probability is \(r(\alpha)\), then the number of rejections has a binomial distribution with \(B\) trials and probability \(r(\alpha)\). The estimated Monte Carlo standard error is therefore

This standard error reflects simulation noise, not uncertainty about whether the simulated design matches the study. A rough 95% interval is \(\hat{r}(\alpha)\pm1.96SE\), although a binomial interval such as a Wilson interval is preferable when the number of rejections is very small or the estimate is near 0 or 1. A modest difference from \(\alpha\) may be consistent with Monte Carlo variation; a large, repeated discrepancy deserves investigation.

Using the Whole P-Value Distribution

A single threshold can miss important patterns. Under a continuous null with calibrated p-values, the p-value distribution is uniform: the proportion at or below \(u\) should be about \(u\) for many values of \(u\). Comparing observed proportions with nominal values at thresholds such as 0.01, 0.05, 0.10, and 0.50 gives a compact calibration profile. A plot of these cumulative proportions against the nominal thresholds should track the diagonal. A p-value histogram should be roughly flat, subject to simulation variability.

An excess of small p-values suggests inflation; a deficit suggests conservatism. A deviation confined to the extreme tail can be missed by a histogram, so tail thresholds should be chosen before inspecting results and should include levels relevant to the intended use. Conversely, a histogram can reveal broad shape problems not evident at one threshold. When comparing procedures, inspect burden, SKAT, and omnibus p-values separately: an apparently satisfactory pooled pattern can hide a problem limited to one method or one class of gene sets.

The following justification explains the uniform benchmark. If \(P_{\text{test}}\) has a continuous null distribution function \(F\), and its p-value is \(1-F(T)\) for a test statistic \(T\) with larger values more extreme, then \(F(T)\) is uniform on \([0,1]\) under the null. Consequently, for \(0\leq\alpha\leq1\),

When a test uses an approximation to the null distribution, or when the statistic has a discrete distribution, this equality need not hold exactly. Valid discrete p-values are generally super-uniform: \(P(P_{\text{test}}\leq\alpha\mid H_0)\leq\alpha\). Thus, a small departure from a flat histogram is not automatically evidence of a defect; the relevant question is whether the observed departure is larger than expected from simulation and appropriate for the test's null assumptions.

Simulation-Based Tail Probabilities

Null simulations can also estimate a tail probability for a particular observed test statistic. If \(T_{\text{obs}}\) is the study statistic and \(T_1,\ldots,T_B\) are simulated null statistics, count \(R=\sum_{b=1}^{B}I(T_b\geq T_{\text{obs}})\). A common finite-simulation estimate is \((R+1)/(B+1)\), rather than \(R/B\). The added one avoids reporting a zero p-value when no simulated statistic exceeds the observed one and gives the rank-based estimate under exchangeability of the observed and simulated statistics under the null.

This is distinct from checking whether an analytic p-value procedure is calibrated across many simulated replicates. The first estimates a tail probability for one observed statistic under a chosen simulation model; the second evaluates repeated-test rejection rates and p-value distributions. Simulation-based probabilities also have limited resolution: with \(B\) replicates, the smallest value from this formula is \(1/(B+1)\). They should not be presented as precise beyond what the number of simulations supports.

Interpreting Diagnostics and Choosing Comparisons

Calibration checks are most informative when the simulated design is explicit. Holding genotypes fixed can reveal whether outcome-model and test approximations behave well for the observed allele frequencies and linkage disequilibrium, but it cannot assess variation in those genetic features across new samples. A parametric bootstrap can represent covariate-adjusted outcome generation under a fitted null model, but its conclusions depend on that model. Permutation can avoid some outcome-distribution assumptions when exchangeability is justified, but naive permutations may break covariate, relatedness, or sampling structure.

Apply the same simulated outcomes to each procedure where possible, so differences in estimated rejection rates are not needlessly confounded by different random draws. Still, gene tests evaluated on the same simulated individuals and variants are dependent. If rates are aggregated across many genes, report how aggregation was done and examine whether particular genes or properties—such as allele count or variant-set size—drive the result. An overall rate can conceal local miscalibration. Simulation uncertainty for a simple per-gene rate uses the number of independent null replicates, not the number of correlated genes treated as if they were independent observations.

A common mistake is to conclude that a test is calibrated because its empirical rate at 0.05 is close to 0.05 in a small simulation. For example, with only 200 null replicates, one rejection changes the estimated rate by 0.5 percentage points, and a rate near 5% has an approximate standard error of 1.54 percentage points. The correction is to report the number of replicates, rejection count, estimated rate, and Monte Carlo uncertainty, and to inspect additional prespecified thresholds. Another mistake is to call every excess of small p-values a failure of the test itself. A misspecified null outcome model, an invalid permutation scheme, or a pipeline that was not fully repeated can also cause the diagnostic to fail.

Null calibration is a diagnostic, not a guarantee of power, causal interpretation, or validity under alternatives. It does not determine which gene-based method is best for a particular genetic architecture, and a test that is well calibrated under one design may not be so under another. The most defensible conclusion is conditional: state the simulated design, the procedure, the observed empirical rates, and whether the deviations exceed plausible Monte Carlo variation.