Q-Q Diagnostics for Empirical Null Models

What a Null Q-Q Plot Compares

Thousands of genomic tests can produce a distribution of test statistics that looks different from the nominal null distribution. A null quantile-quantile plot, or null Q-Q plot, makes the discrepancy visible by comparing ordered observed statistics with quantiles expected under a specified null model. Its shape can indicate whether the main discrepancy concerns the null center, the null scale, or departures concentrated in the tails.

For example, suppose the statistics are approximately standard normal under the theoretical null, but many central observations are slightly positive. A plot against the theoretical null can show a location shift. If both positive and negative statistics spread farther from zero, it can show scale inflation. If the center follows the reference but only the most extreme observations depart, the pattern is concentrated in the tails. These are diagnostic patterns, not proof of their cause: genuine non-null tests can also create tail departures.

The plotting positions are a convenient approximation to the quantiles of ordered observations from a continuous distribution. Other plotting-position conventions are also used; they can change the exact plotted coordinates slightly, especially in small samples, but do not change the main interpretation. The plot is descriptive: a line is a visual reference, not automatically a fitted statistical model or a formal test.

Reading Center, Scale, and Tail Patterns

For a normal null with center \(\mu_0\) and standard deviation \(\sigma_0\), its population quantile at probability \(p\) is \(\mu_0+\sigma_0\Phi^{-1}(p)\). Therefore, if the observed statistics follow that null, their Q-Q points should approximately track the line \(z=\mu_0+\sigma_0q\), subject to sampling variation. Against a standard normal reference, the target line is \(z=q\); against an empirical null, it is \(z=\hat{\mu}_0+\hat{\sigma}_0q\).

This quantile relationship gives the basic diagnostic interpretation. A positive intercept indicates that the plotted statistics tend to lie above the reference near the center, consistent with a positive location shift; a negative intercept indicates the reverse. A slope greater than one indicates more spread than the reference, while a slope below one indicates less spread. These readings describe the overall pattern only when the points are reasonably linear. Curvature or localized departures mean a single intercept and slope may conceal important differences.

Tail patterns are read relative to the central trend or the chosen reference line. With \(q_i\) on the horizontal axis and \(z_{(i)}\) on the vertical axis, heavy tails typically place the lower-tail points below the line and the upper-tail points above it: observed values are more extreme than expected at both ends. Light tails typically reverse that pattern. A departure at only one end may indicate asymmetry or a one-sided excess. Tail deviations in genomic data may also arise because some tests correspond to genuine effects; a Q-Q plot alone cannot label those observations as null or non-null.

Worked Examples

Why the Interpretation Works—and What It Assumes

The line interpretation follows directly from the quantile function of a normal distribution. If \(Z\sim N(\mu_0,\sigma_0^2)\), then \((Z-\mu_0)/\sigma_0\sim N(0,1)\), for \(\sigma_0>0\). For a probability \(p\), the standardized variable has quantile \(\Phi^{-1}(p)\). Multiplying by \(\sigma_0\) and adding \(\mu_0\) gives the original-scale quantile:

Using \(p_i\) as the plotting positions gives the reference values \(\mu_0+\sigma_0q_i\). If the sample is generated from that normal distribution, its ordered observations should fluctuate around those quantiles, rather than match them exactly. This justification assumes the normal model is a suitable null for the statistics being compared. In genomic applications, test statistics may only be approximately normal under the theoretical null, for example because a large-sample approximation is used.

A diagnostic plot also depends on what is plotted. Compare like with like: the statistics should have a common interpretation and scale, and the proposed null should apply to the set or subset under examination. If an empirical null is used, its estimated center and scale must be obtained by a defensible method. Earlier material on estimating the null center and scale covers those estimation approaches; here, the plot is used to inspect the resulting distributional fit. Estimating parameters and assessing fit on the same observations can make the apparent agreement look better than it would on independent data.

The points are not generally independent simply because there are many tests. Genomic statistics may be dependent, and ordered values are themselves constrained by their ranks. Therefore, visual scatter around a line should not be interpreted using an independent-normal sampling argument without justification. In particular, a Q-Q plot is not a formal test with a universal cutoff for declaring the null model wrong. Simulation-based envelopes can help quantify expected variation when the full design and dependence structure can be represented adequately, but an envelope also depends on the simulation assumptions.

Diagnostics, Comparisons, and Common Pitfalls

Inspect the central pattern and the tails separately. First check whether the bulk of the points is approximately linear and whether its center is displaced. Then assess whether the slope differs from the intended reference and whether deviations are concentrated at either end. Finally, compare the plot with how the null parameters were estimated and with the scientific setting: real signal is often most visible in the tails, so a tail departure need not imply that the central null model is unsuitable.

• Check the axes and reference. Confirm whether the horizontal axis uses standard normal quantiles or empirical-null quantiles. A line with slope one is the target only when the reference has already incorporated the intended location and scale.
• Do not let a few extremes define the center. The largest statistics can strongly affect a line fitted to all points. Describe the central trend and tail departures separately.
• Account for finite-sample variability. The most extreme order statistics are naturally variable. A visible end departure in a small set of tests is not, by itself, strong evidence against a null model.
• Remember dependence. Correlated tests can alter the variability and appearance of the ordered statistics. The apparent number of plotted points is not necessarily the number of independent pieces of information.

A histogram summarizes the distribution in bins and can reveal broad asymmetry or multiple features, but its appearance depends on bin width and it does not directly map observed quantiles to reference quantiles. A null Q-Q plot instead shows which quantiles depart, and in what direction. A local FDR calibration plot addresses a different question: how estimated posterior null risks compare with observed outcomes across groups. It does not replace a distributional comparison of test statistics with a null reference. These displays can complement one another, but they answer different diagnostic questions.

A common mistake is to say, “The upper tail lies above the line, so the empirical null has too large a variance.” That conclusion does not follow: an upper-tail-only departure is not the symmetric spread pattern expected from a simple scale increase. Correction: check the lower tail and central trend too. A scale discrepancy should affect the distribution broadly, whereas a one-ended departure may reflect asymmetry or an excess of positive non-null effects.

The main limitation is that the plot is descriptive and not uniquely diagnostic of cause. Location, scale, skewness, heavy tails, mixtures, dependence, and genuine effects can overlap in their visual consequences. A Q-Q plot can guide model assessment, but it cannot prove that an empirical null is correct, establish which tests are truly null, or replace the assumptions required to estimate that null.