Why Diagnose Dependence?
In a genomic analysis, nearby tests may share genetic variation, samples, or features of the experimental design. Their test statistics and p-values can therefore move together. This matters because a list of marginal p-values does not show how the tests behave jointly, and procedures for multiple testing may have guarantees that depend on assumptions about that joint behavior.
A dependence diagnostic is evidence about the data and analysis, not a theorem about the conditions under which a procedure controls the false discovery rate (FDR). Correlations, p-value histograms, and regional patterns can help describe dependence and identify problems worth investigating. None of them, by itself, proves independence or a condition such as positive regression dependence on a subset (PRDS). The earlier material on marginal p-value validity versus independence distinguishes the validity of individual null p-values from assumptions about their joint distribution.
A null p-value is valid if it is super-uniform: for every \(t\) between 0 and 1, \(P(P_i\leq t)\leq t\) when test \(i\)'s null hypothesis is true. An exact continuous null p-value is uniform on \([0,1]\), so \(P(P_i\leq t)=t\). Exact uniformity is stronger than validity. Some diagnostics require the stronger assumption, and the distinction must be kept explicit.
Correlations Between Test Statistics
A direct approach is to estimate correlations between test statistics across tests or, more commonly, across resamples designed to represent their joint null behavior. For genomic measurements, correlations between raw features are not automatically correlations between the statistics used for testing. Shared covariates, standardization, sample size, and the test construction can all affect the latter.
Suppose each of \(B\) resamples produces a vector of test statistics, with \(T_{bi}\) denoting the value for resample \(b\) and test \(i\). An empirical correlation between tests \(i\) and \(j\) is the ordinary sample correlation of \((T_{1i},\ldots,T_{Bi})\) and \((T_{1j},\ldots,T_{Bj})\). The resampling scheme should preserve the dependence-generating structure relevant under the null: for example, it should retain the observed relationships among genotypes while generating null outcomes in a way justified by the study design.
Worked Example: Correlation Across Null Resamples
In a small hypothetical illustration, five valid null resamples produce the following statistics for two genomic tests:
\(T_1=(-2,-1,0,1,2)\) and \(T_2=(-1,0,1,0,0)\). Both sample means are zero. The sums of squares are \(\sum_b T_{b1}^2=10\) and \(\sum_b T_{b2}^2=2\), and the sum of cross-products is \(\sum_b T_{b1}T_{b2}=2\).
The sample correlation is
This positive sample correlation describes these five resamples; it is not a precise estimate with only five replicates. In a real analysis, many suitable resamples would be needed, and uncertainty in the estimated correlations should be considered. The value does not establish PRDS: pairwise correlations summarize only part of a joint distribution and do not verify a regression-dependence condition.
Correlations are useful for identifying broad patterns, such as stronger dependence between nearby tests than between distant tests. Their interpretation depends on the statistic and the resampling design. A correlation estimated across individuals from observed outcomes, for example, may reflect true signals as well as null dependence. Conversely, a small estimated pairwise correlation does not rule out higher-order or nonlinear dependence.
What a P-Value Histogram Can and Cannot Show
A p-value histogram summarizes the marginal distribution of the observed \(P_i\)'s. Under a complete null with exact continuous p-values, each marginal p-value is uniform, whether or not the p-values are independent. Thus, for any interval \([a,b)\subseteq[0,1]\), the probability for a single null p-value to fall in that interval is \(b-a\). Dependence does not change this marginal probability, but it can change how variable the histogram counts are across repeated data sets.
Worked Example: Expected Counts in P-Value Bins
Suppose a hypothetical analysis has \(m=1{,}000\) tests, and for this calculation assume all null hypotheses are true and all null p-values are exact and continuous uniform. Divide the p-values into \([0,0.2)\), \([0.2,0.5)\), and \([0.5,1]\). Their probabilities are \(0.2\), \(0.5-0.2=0.3\), and \(1-0.5=0.5\), respectively. The expected counts are therefore
Imagine that the observed counts are 320, 280, and 400. The first bin has 120 more p-values than its expectation under this complete-null calculation. That excess might reflect nonnull tests, a failure of the assumed null calibration, or chance variation. It does not, by itself, identify which explanation is responsible, nor does it establish dependence. In particular, dependence can make departures in counts more variable even when the marginal expectations remain as calculated.
The exact-uniform assumption in that calculation is essential. Validity and continuity alone do not determine the probability of falling in an interval such as \([0.2,0.5)\). For example, a p-value uniform on \([0.5,1]\) is continuous and valid: its cumulative probability is zero below \(0.5\), and \(2t-1\leq t\) for \(0.5\leq t\leq1\). Its probability of falling in \([0.2,0.5)\) is zero, not \(0.3\). For merely valid null p-values, the probability of \([a,b)\) is \(P(P_i<b)-P(P_i<a)\); the validity inequalities do not fix this difference.
A histogram with excess near zero is often compatible with a mixture of null and nonnull tests, but it is not a dependence diagnostic on its own. Nor does a visually flat histogram prove independence. A useful justification for this caution comes from considering a bin count. Let \(I_i\) equal 1 when null p-value \(P_i\) falls in a specified bin and 0 otherwise, and let \(C=\sum_i I_i\). Then
The first expression depends on marginal bin probabilities. The second also includes pairwise covariances. Consequently, the expected count under exact null uniformity can remain \(m(b-a)\) while dependence changes the distribution of that count. Treating bins as if their counts had the variability expected under independence can therefore be misleading.
Genomic Region Patterns
Genomic dependence often has an ordered structure: tests correspond to nearby positions, genes, or other related features. A run of small p-values in one region may be consistent with shared signal or local dependence, but it can also arise from chance, model misspecification, or technical artifacts. A regional display makes this structure visible; a formal summary can use adjacent-statistic correlations, run lengths, or the number of regions meeting a prespecified criterion.
Worked Example: Assessing a Long Run of Small P-Values
In a hypothetical scan, researchers summarize each analysis by the longest run of consecutive genomic tests with \(P_i<0.01\). The observed longest run is four tests. To assess how unusual that pattern is under a complete null, they conduct 999 phenotype-label permutations. Assume the study design makes the labels exchangeable under this null, and that each permutation retains the observed genotypes and hence their local dependence. In 18 permutations, the longest run is at least four.
Using the common plus-one permutation calculation, the estimated tail probability is
Under the stated exchangeability assumption and this permutation scheme, a run at least this long is relatively unusual under the complete-null reference distribution. This result is about the chosen run statistic, not a probability that the observed region is truly associated and not an FDR guarantee. The conclusion could change if the permutation scheme failed to preserve a relevant design feature or if the statistic had been selected after exploring many alternatives without accounting for that selection.
Regional summaries should be chosen with the biological and statistical structure in mind. A window that is too narrow may miss broad dependence; one that is too wide may combine unrelated regions. Plotting results by genomic position can reveal clusters, but visual inspection is exploratory and can be affected by the scale, thresholds, and regions selected for display.
Combining Diagnostics Without Overclaiming
Each diagnostic answers a different descriptive question. Test-statistic correlations summarize linear co-movement for selected test pairs under the data or resampling scheme. P-value histograms summarize marginal distributions and can reveal calibration concerns or an excess of small values. Region patterns show where observed statistics or p-values cluster along genomic coordinates. Their agreement can strengthen the case that the data have structured dependence, but agreement still does not prove a particular formal condition.
| Diagnostic | What it describes | Important limitation |
|---|---|---|
| Test-statistic correlations | Pairwise linear association under the chosen sampling or resampling scheme | Do not capture all dependence or establish PRDS |
| P-value histogram | Marginal distribution of the p-values | Does not show which tests co-vary; shape can reflect alternatives or miscalibration |
| Genomic region pattern | Spatial clustering or runs in genomic order | Can reflect chance, artifacts, or the chosen window and threshold |
| Dependence-preserving permutation | Reference distribution for a prespecified summary under a stated null | Its validity depends on the exchangeability and design assumptions |
The main alternative is not a single competing plot: it is to rely on a procedure whose guarantee is justified under a specified dependence condition, or to use a procedure designed for broader dependence assumptions. The earlier material on PRDS and FDR control describes why formal guarantees require the corresponding assumptions. An observed positive correlation is not a substitute for showing PRDS, and a lack of visible correlation is not proof of independence. Likewise, diagnostic evidence cannot turn a guarantee under one set of conditions into a guarantee under another.
Useful checks include examining whether correlations change with genomic distance, comparing summaries from appropriately constructed null resamples, checking whether unusual regional patterns coincide with known technical features, and assessing sensitivity to reasonable plotting bins or region definitions. These checks help expose failure modes, but should be reported as diagnostics with their assumptions, not as proofs.
Common Mistakes and Limitations
- Equating validity with exact uniformity. Validity gives \(P(P_i\leq t)\leq t\), not equality. A bin probability such as \(0.3\) for \([0.2,0.5)\) follows only under exact uniformity.
- Reading a histogram as a dependence plot. A histogram contains no information about which p-values belong to neighboring or otherwise related tests. It describes the marginal distribution.
- Assuming a correlation proves a formal dependence condition. Pairwise correlations are incomplete summaries; PRDS is a joint-distribution condition, not a synonym for positive correlations.
- Ignoring the resampling design. Permuting labels is justified only when the relevant exchangeability assumptions hold. Resampling that breaks structure important under the null may produce misleading diagnostics.
- Calling a cluster a discovery guarantee. A cluster of small p-values may be scientifically interesting, but a regional pattern alone does not establish an FDR-controlled set of discoveries.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Under an exact continuous uniform null, what is the probability that a p-value falls in \([0.2,0.5)\)? Does validity and continuity alone determine it?
- Why can dependence alter the variability of a p-value histogram bin count without changing its expectation under exact null uniformity?
- A sample correlation of \(0.45\) is found between two test statistics across null resamples. What does this show, and what does it not show?
- In a permutation assessment with 999 permutations, 18 have a run statistic at least as large as observed. Calculate the plus-one estimate and interpret it.
Solutions. (1) Under exact uniformity, the probability is \(0.5-0.2=0.3\). Validity and continuity alone do not determine it; a continuous valid p-value uniform on \([0.5,1]\), for example, has probability zero in that interval. (2) The expected count is the sum of the marginal bin probabilities, whereas its variance includes covariance terms between test indicators; dependence changes those terms. (3) It describes positive linear association in those resamples, but does not establish independence or PRDS, and its precision depends on the number and validity of resamples. (4) The estimate is \((18+1)/(999+1)=0.019\). Under the stated permutation assumptions, it summarizes how often the run statistic is at least as large under the null reference scheme; it is not an FDR guarantee.