Why Calibration Matters
A small p-value is evidence against a gene-level null hypothesis only to the extent that the test produces small p-values at the advertised rate when that null is true. Calibration asks whether the null distribution assumed by the test agrees with the distribution generated by the data and analysis procedure. It does not ask whether a particular gene is truly differentially expressed, nor does it establish that a model is correct in every respect.
In pseudobulk analysis, each gene is tested using counts aggregated within biological samples, with a design that may include condition, batch, pairing, or donor-level effects. Let \(\theta_g\) denote the condition contrast for gene \(g\), with null hypothesis \(H_0:\theta_g=0\). A typical statistic is a standardized contrast, \(T_g=\hat{\theta}_g/SE(\hat{\theta}_g)\). If its null reference distribution is standard normal, a two-sided p-value is \(2\Phi(-|T_g|)\), where \(\Phi\) is the standard normal cumulative distribution function. Other models may use a \(t\), likelihood-ratio, or other reference distribution; calibration must be assessed against the reference actually used.
The assumptions behind a reference distribution matter. A normal approximation to \(T_g\) requires the fitted model and its standard error to describe the sampling variation adequately, and the approximation must be accurate at the sample size available. A \(t\)-reference test additionally relies on its specified degrees of freedom being appropriate. For count models, the mean-variance relationship, design matrix, and handling of sample-level dependence must be sufficiently well specified for the inferential procedure. Calibration checks can reveal discrepancies, but they cannot by themselves identify which assumption failed.
What a Null P-Value Distribution Should Look Like
For a continuous exact test, the probability integral transform gives a simple justification for uniform p-values. If \(T\) has a continuous null cumulative distribution function \(F_0\), then its one-sided upper-tail p-value is \(p=1-F_0(T)\). For any \(u\) between zero and one, \(P(p\leq u)=P(1-F_0(T)\leq u)=P(F_0(T)\geq1-u)=u\), because \(F_0(T)\) is uniform under the null. For a two-sided test, the same result holds when the p-value is computed from the correct continuous null distribution using the appropriate two-sided tail probability.
Thus, among many true null tests, roughly 5% should have p-values below 0.05, and roughly 1% below 0.01. “Roughly” matters: finite collections fluctuate, and p-values for different genes are generally dependent because genes share samples and biological pathways. A histogram that is approximately flat is one possible sign of calibration when most tested genes are null, but it is not a requirement when many genes have true effects.
A pile-up of small p-values in the observed data has at least two explanations: true differential expression, or anti-conservative testing (inflation). A deficit of small p-values can result from conservative tests, limited power, or a discrete test. Therefore, the p-value distribution from observed genes alone is not a clean calibration reference unless the tested genes are known or credibly assumed to be null.
Worked Example: Deriving the Two-Sided Normal Null Distribution
Suppose, under \(H_0\), that \(T_g\sim N(0,1)\) and the reported p-value is \(p_g=2\Phi(-|T_g|)\). Verify the probability of a p-value at most 0.05.
The inequality \(2\Phi(-|T_g|)\leq0.05\) is equivalent to \(|T_g|\geq z_{0.975}\), where \(z_{0.975}\approx1.960\). Under the stated null model,
The equality depends on the standard normal null distribution being correct. If the actual null statistic has heavier tails, more than 5% of tests may cross the nominal 0.05 threshold, even when all tested effects are zero.
Reading Diagnostics Without Overinterpreting Them
Useful summaries include p-value histograms, empirical cumulative distribution functions, and rejection proportions at prespecified thresholds. For a threshold \(u\), the empirical rejection proportion among \(m\) tested genes is \(\hat{F}(u)=m^{-1}\sum_{g=1}^{m}I(p_g\leq u)\). Under \(m\) independent true null tests with exact p-values, its expectation is \(u\), and its standard error is \(\sqrt{u(1-u)/m}\). In gene-wide data the independence assumption is usually not justified, and true effects also change the proportion. Treating that standard error as a definitive diagnostic can therefore be misleading.
The pattern across thresholds is informative. Excess over the uniform reference near zero may reflect a mixture of true effects and inflation. Excess across a broad range, including moderate p-values, is more concerning for systematic miscalibration, but still does not prove it. An approximately uniform distribution does not guarantee calibrated individual tests: different errors can offset in an aggregate display, and a small number of null tests may have little diagnostic power.
Inspect diagnostics in relation to the analysis pipeline. Calibration can differ by average expression, dispersion, cell number per pseudobulk sample, or design subgroup. Filtering changes which genes are included; if a filter uses the same test statistic or p-value being assessed, the retained null p-values need not remain uniform. A filter independent of the test under the null is less likely to create this particular distortion, but still should be documented. Prior checks of residuals, dispersion, fitted means, influential samples, and model convergence help investigate possible causes; a favorable p-value plot does not replace them.
Simulated Null Contrasts
A more direct diagnostic is to create data or contrasts for which the null is known, then run the complete analysis and compare the resulting statistics with their claimed reference distribution. The simulation must preserve features relevant to the real test: numbers of donors, pairing or repeated measures, batch allocation, unequal sample sizes, count means and dispersions, and any model-fitting or filtering steps being evaluated. A simulation that breaks the donor structure can make a procedure look calibrated under a design it will not encounter.
There are two common routes. In a model-based simulation, generate counts under a fitted model with the condition contrast constrained to zero, retaining nuisance structure such as baseline expression, dispersion, and donor effects as appropriate. Refit the model and calculate the same test statistic and p-value used in the analysis. In a design-respecting permutation, reassign labels only in ways allowed by the null and experimental design—for example, swap condition labels within each matched donor in a paired study. Permutations are not automatically valid: unrestricted label shuffling is wrong when it breaks pairing, batch restrictions, or exchangeability.
State which contrast is zero, which nuisance effects remain, and which sample relationships must be preserved.
Use a model-based simulation or a permutation scheme justified by the design; do not remove dependence that the real test must handle.
Refit, calculate the same statistic and p-value, and apply the same prespecified filtering and convergence rules.
Check null p-value proportions, statistic quantiles, and tail behavior, accounting for simulation uncertainty and dependence.
Worked Example: Checking a Nominal Five-Percent Rejection Rate
In a hypothetical calibration exercise, one null contrast is generated and tested in each of 1,000 independently simulated datasets. The fitted procedure reports \(p<0.05\) in 87 datasets. Under exact 5% calibration, the count \(R\) has a binomial distribution \(R\sim\mathrm{Binomial}(1000,0.05)\), assuming independent simulation replicates.
The observed rate is \(87/1000=0.087\), or 8.7%. Its standardized departure from the expected count is \((87-50)/6.892\approx5.37\). This is substantially more than ordinary binomial fluctuation would usually explain under the stated assumptions, so the exercise suggests anti-conservative behavior at this threshold. It does not identify the cause: for example, an inaccurate standard error or a misspecified null simulation could be responsible. The conclusion is about this hypothetical procedure and design, not a claim about any real dataset.
Worked Example: Comparing an Observed Excess With a Null Simulation
Suppose a hypothetical analysis of 10,000 genes finds 900 p-values below 0.01. That 9.0% rate is much larger than 1%, but the observed genes may include genuine differential expression, so it cannot by itself establish inflation. The analyst therefore simulates 2,000 null contrasts, preserving the study design, and finds 22 simulated p-values below 0.01.
The simulated null rate is \(22/2000=0.011\), or 1.1%. Under a 1% rate, the expected count is \(2000(0.01)=20\), and the binomial standard deviation under independent replicates is \(\sqrt{2000(0.01)(0.99)}\approx4.450\). The simulated count is only \((22-20)/4.450\approx0.449\) standard deviations above expectation. This gives no indication of notable inflation at the 0.01 threshold in this simulation. The observed 9.0% may therefore reflect real effects, though this comparison alone cannot prove that explanation or verify calibration for every gene subgroup or tail threshold.
Comparing Calibration Approaches and Their Limits
| Approach | What it checks | Main caution |
|---|---|---|
| Analytic null reference | Whether test statistics or p-values follow the distribution assumed by the model | Approximation quality may be poor with few donors or model misspecification |
| Observed p-value distribution | Whether the collection of results has an unexpected overall pattern | True effects and dependence complicate interpretation |
| Design-respecting permutation | Null behavior under label assignments allowed by the design | Requires a valid exchangeability argument and enough allowable permutations |
| Model-based null simulation | Performance under a specified fitted data-generating model | Can reproduce errors in the model used to generate the simulations |
No approach is a universal gold standard. Analytic calibration is efficient but relies on the reference distribution and its approximations. Permutation can avoid some parametric assumptions, but only when the assignment or exchangeability structure justifies it; it may also have limited resolution in small studies. Model-based simulation can preserve complex nuisance structure and evaluate the complete pipeline, but its conclusions are conditional on the data-generating model. Agreement across a justified analytic check and a well-designed simulation is more persuasive than either alone.
Simulation-based tail proportions also have Monte Carlo error. If \(B\) independent null replicates produce \(r\) rejections, the estimated rate is \(r/B\), with approximate standard error \(\sqrt{\hat{q}(1-\hat{q})/B}\), where \(\hat{q}=r/B\). When genes or tests within a replicate are dependent, they should not be treated as independent replicates to obtain an artificially small uncertainty estimate. Simulation uncertainty describes finite Monte Carlo precision; it does not include uncertainty from choosing the wrong null model.
Common Pitfalls
- Calling every excess of small observed p-values inflation. That reasoning assumes all tested genes are null. Genuine differential expression also creates small p-values. Compare with known-null or simulated-null tests before attributing an excess to miscalibration.
- Using a histogram as a pass-fail test. A flat-looking plot can hide subgroup or tail problems, while a non-flat plot can be expected when effects are present. Examine thresholds and strata, and use null simulations where feasible.
- Simulating the wrong design. Independently generated samples do not assess a paired procedure if donor pairing is present. Preserve the dependence and assignment restrictions relevant to the test.
- Assuming a successful simulation proves validity. A simulation generated from a misspecified model can make the fitted procedure appear well calibrated by construction. Check model fit and vary plausible nuisance assumptions.
- Ignoring selection. Filtering or choosing genes based on their test results can alter the distribution among retained genes. Include the relevant selection steps in the calibration exercise or account for their effect.
Check Your Understanding
Answer each question, then compare with the solutions below.
- Under a continuous exact null test, what is \(P(p_g\leq0.10\mid H_0)\), and what does that statement not say about an individual observed p-value?
- A paired study simulates null data by shuffling condition labels across all samples, without regard to donor. What is the problem, and what would a more appropriate permutation preserve?
- Among 500 independent null simulation replicates, 35 yield \(p<0.05\). Compute the empirical rate and compare it with the nominal rate.
- An observed analysis has many small p-values, but simulated null contrasts show a rate near the nominal rate. Give one plausible interpretation and one reason not to conclude definitively that every observed small p-value reflects a real effect.
Solutions. (1) The probability is 0.10 under the stated null; it is a repeated-sampling property, not the probability that the null is true for a particular gene. (2) Unrestricted shuffling breaks the within-donor pairing and may violate exchangeability. A valid scheme should preserve matched donor sets, such as swapping labels within donors when that assignment is justified. (3) The empirical rate is \(35/500=0.070\), or 7.0%, which is above 5%; the excess is a diagnostic signal, but its uncertainty and the simulation assumptions should be examined before concluding inflation. (4) Genuine effects are a plausible source of the observed excess if the null simulations are calibrated. However, simulation validity is conditional on its model, and an aggregate null check cannot establish the cause or truth of every individual gene result.