Calibration Diagnostics in Admixed Association Studies

Why Calibration Needs More Than One Check

An association test is calibrated when its reported null distribution gives the intended error rates under the model assumptions. In an admixed cohort, those assumptions can be difficult to assess: ancestry composition may be related to both genotype and phenotype, ancestry-specific sample sizes can differ, and the association model may not capture every relevant feature of the data. A small p-value can reflect a real association, a calibration problem, or both. Calibration diagnostics help assess the null behavior of the tests; they do not, by themselves, identify why that behavior is observed.

This tutorial focuses on three complementary checks: QQ plots stratified by ancestry, genomic inflation factors, and tests using variants or outcomes expected to be null. Each check summarizes a different aspect of the evidence. Their interpretation depends on how the tests and strata were constructed, what is known about the null, and how dependent the test statistics are.

Definition: A set of association tests is calibrated under the null when, for hypotheses that are truly null and under the stated model assumptions, their test statistics follow the reference null distributions used to compute p-values. A p-value is valid if \(P(p \leq t) \leq t\) for every \(t\) between 0 and 1 under the null. An exactly continuous null p-value has \(P(p \leq t)=t\) for every such \(t\), so it is Uniform(0,1).

Validity is weaker than exact uniformity: a valid test can be conservative, so its p-values tend to be larger than uniform null p-values. Also, even exactly uniform marginal p-values need not be independent. Nearby variants can be correlated through linkage disequilibrium, and tests can share individuals and covariates. Thus, a diagnostic reference must distinguish what follows from marginal calibration from what additionally requires independence.

Ancestry-Stratified QQ Plots

A QQ plot compares observed association p-values with the values expected under a specified null reference. For a stratum of \(m\) tests, sort the observed p-values as \(p_{(1)} \leq \cdots \leq p_{(m)}\). Under independent, exactly Uniform(0,1) null p-values, the expected value of the \(i\)th order statistic is \(i/(m+1)\). A common plot places \(-\log_{10}(i/(m+1))\) on the horizontal axis and \(-\log_{10}(p_{(i)})\) on the vertical axis. Points near the diagonal are broadly consistent with that reference; points above it indicate smaller observed p-values than expected.

In an admixed study, make separate plots for prespecified ancestry strata or ancestry-composition groups when the analysis design supports that comparison. The precise grouping should be defined in advance and reported: for example, a set of broad global-ancestry bins is not the same as a set of study recruitment groups. A stratum must contain enough tests and adequate sample information for its diagnostic to be useful. Stratification can reveal whether a departure is concentrated in one group or appears across groups, but it does not guarantee that confounding has been removed. Nor does a difference between stratum plots establish that genetic effects differ.

The order-statistic reference above is exact for independent, exactly uniform null p-values. With dependent tests, it is a useful visual benchmark, not an exact description of the variability of the plotted ranks. If formal uncertainty bands are needed, generate them under the null while preserving the dependence and analysis structure, for example using a suitable model-based null simulation. Moreover, a genome-wide plot usually mixes null and non-null variants. A tail above the diagonal can therefore reflect genuine associations as well as miscalibration.

Worked Example: Reading a Tail Departure in One Stratum

Suppose a prespecified ancestry stratum has \(m=1{,}000\) tested variants. In a hypothetical dataset, the tenth-smallest p-value is \(0.00050\). Under the independent, exactly uniform null reference, its expected plotting position is \(10/(1{,}000+1)\).

\text{Expected }p_{(10)}=\frac{10}{1001}=0.009990,\qquad -\log_{10}(0.009990)\approx 2.000

The observed coordinate is \(-\log_{10}(0.00050)=3.301\), so the tenth point lies above the reference diagonal: this stratum has more small p-values in this part of the tail than the simple null benchmark predicts. That is a diagnostic signal, not a conclusion that the model is confounded. A true association signal, dependence among variants, or a mismatch between the actual and assumed null behavior could contribute. Inspection of null controls and the fitted model is needed before interpreting the departure.

Genomic Inflation Factors

A genomic inflation factor summarizes the center of a set of association test statistics relative to its null distribution. For one-degree-of-freedom tests, convert each two-sided association p-value to its corresponding chi-square statistic, \(\chi^2_j\), with one degree of freedom, and calculate the ratio of the observed median to the theoretical null median. The latter is approximately \(0.4549\).

Definition: For \(m\) one-degree-of-freedom tests, the genomic inflation factor is

\lambda_{\mathrm{GC}}= \frac{\operatorname{median}(\chi^2_1,\ldots,\chi^2_m)} {0.4549}.

If the test statistics follow their one-degree-of-freedom null distributions, the population median is \(0.4549\), and \(\lambda_{\mathrm{GC}}\) is expected to be near 1, subject to sampling variation and dependence.

The factor is a compact summary, not a complete calibration assessment. A value above 1 indicates that the median statistic is larger than the reference median; it does not say which variants drive the excess or why. A value below 1 can indicate conservative test behavior, but can also arise from sampling variation. The statistic is intended for comparable one-degree-of-freedom tests: do not apply the \(0.4549\) denominator to a test with a different null distribution. For example, a joint test with more degrees of freedom needs its own null median.

In a large study, widespread real associations can move the median upward, so a value above 1 is not automatically evidence of confounding. Conversely, a factor near 1 does not rule out problems in a tail or in a particular ancestry stratum. Report the factor alongside stratum-specific QQ plots and null-based checks, and interpret it in light of the test type, sample size, and proportion of truly associated variants.

Worked Example: Calculating a Stratum-Specific Inflation Factor

Consider seven hypothetical one-degree-of-freedom statistics in one ancestry stratum: \(0.20,\ 0.35,\ 0.50,\ 0.70,\ 1.00,\ 1.40,\ 2.10\). The median is the fourth value, \(0.70\). Using the standard one-degree-of-freedom null median gives

\lambda_{\mathrm{GC}}=\frac{0.70}{0.4549}=1.539.

For this small illustrative set, the median statistic is about 1.54 times the theoretical null median. Seven tests are far too few to treat that ratio as a stable estimate of genome-wide behavior; the calculation only demonstrates the definition. In a real analysis, calculate the factor within each suitable ancestry stratum and for the pooled analysis, then compare these summaries with the QQ plots and null controls. A larger stratum-specific factor identifies a departure worth investigating, not its cause.

Null-Variant Tests and Error-Rate Checks

A null-variant check evaluates tests for variants known, by design or by a credible external basis, to have no effect on the outcome under the target model. The usefulness of the check depends on that null claim and on the test being applied under conditions matching the main analysis. If the variants are selected because their observed p-values were large, the check cannot assess calibration fairly. If a supposed null variant is actually associated, its small p-value is not evidence of miscalibration.

A related strategy is to generate a null outcome or null datasets under a model that preserves the relevant features of the design. The procedure must respect covariates, relatedness, ancestry structure, and any other features used by the association analysis. A naive permutation that breaks an important dependence or design feature may produce a misleading benchmark. Where possible, examine calibration separately by ancestry stratum as well as overall.

For any single valid null p-value, \(P(p \leq t) \leq t\). Thus the expected number below threshold \(t\) among \(m\) null tests is at most \(mt\), by adding the individual crossing probabilities; independence is not required for this expectation. The count is exactly Binomial(\(m,t\)) only when the \(m\) null p-values are independent and each is exactly Uniform(0,1) (or, for this particular threshold, when each independent test has crossing probability \(t\)). Validity and continuity alone do not imply that distribution. If tests are dependent, even exact uniformity of each p-value does not make the count binomial; dependence changes the count's variability.

Worked Example: Checking a Prespecified Null Set

Suppose 400 hypothetical, independently generated null tests have exactly Uniform(0,1) p-values. At threshold \(t=0.05\), each test has probability \(0.05\) of crossing the threshold. Therefore the number \(K\) of p-values below \(0.05\) follows a Binomial(\(400,0.05\)) distribution. Its expectation and standard deviation are

E(K)=400(0.05)=20,\qquad SD(K)=\sqrt{400(0.05)(0.95)}=\sqrt{19}=4.359.

If 24 of these tests cross the threshold, the count is four above its expectation, or \(4/4.359=0.918\) standard deviations above it. This example does not indicate a striking departure from the stated reference. In genomic data, however, tests at nearby variants are often dependent, so this binomial calculation should not be applied to a correlated null-variant set without justification. A dependence-preserving null simulation or a different calibration assessment may be more appropriate.

Why the Count Reference Requires Its Assumptions

The binomial result follows directly from the assumptions. Let \(I_j\) equal 1 if null test \(j\) has \(p_j \leq t\), and 0 otherwise. If the p-values are independent and each is exactly Uniform(0,1), then the indicators are independent Bernoulli variables with success probability \(t\). Their sum \(K=\sum_{j=1}^{m} I_j\) is therefore Binomial(\(m,t\)). If p-values are only valid, each success probability \(q_j=P(p_j \leq t)\) may be less than \(t\); if the indicators are independent, the count is then a sum of Bernoulli variables with possibly different success probabilities, not generally Binomial(\(m,t\)). Without independence, even this sum-of-independent-Bernoulli result does not apply.

E(K)=\sum_{j=1}^{m}P(p_j\leq t)\leq mt

This expectation bound requires validity of each null p-value but not independence. It does not specify the distribution or variability of \(K\). In particular, it cannot by itself justify a binomial tail probability. This distinction matters when interpreting the number of small p-values in a correlated set of genomic tests.

Putting the Diagnostics Together

The three checks answer related but distinct questions. QQ plots show the shape and location of departures across the p-value distribution. The genomic inflation factor compresses the center of one-degree-of-freedom statistics into a single number. Null-variant or null-outcome checks compare observed error behavior with a defensible null benchmark. A plot can reveal a tail pattern that the median-based factor misses; a factor can summarize a broad shift that is visually subtle; and a credible null set can help distinguish real association signals from departures in tests expected to be null.

For an admixed cohort, calculate and display diagnostics by the ancestry strata relevant to the analysis as well as for the pooled result, when sample sizes permit. Record how strata were defined, which variants or outcomes are treated as null, the test-statistic null distributions, and how dependence was handled. Compare diagnostics across strata cautiously: differences can reflect sample size, allele frequencies, test precision, or the composition of the tested variants, not only model performance.

A visible departure is a prompt for further checks, not a license to apply a single correction mechanically. Before using any adjustment, establish that its assumptions fit the analysis and that it does not obscure real signal. These diagnostics can reveal that test behavior deserves scrutiny, but a QQ plot, an inflation factor, or a null-set count alone cannot determine whether the source is residual confounding, genuine associations, or another model or data feature.

Common Mistakes and Failure Modes

Treating every departure as confounding. Small p-values can arise from real associations, while calibration problems can have multiple sources. Use the diagnostics to flag patterns, then investigate the model and null controls.
Assuming valid means exactly uniform. Valid p-values can be conservative. For \(m\) independent conservative tests, the count below \(t\) is not necessarily Binomial(\(m,t\)); its expected count is at most \(mt\).
Assuming exact marginal uniformity means independent tests. Linkage disequilibrium can induce dependence. It affects uncertainty in QQ ranks and the distribution of threshold-crossing counts.
Relying on one pooled summary. A pooled inflation factor can conceal a departure concentrated in one ancestry stratum, and a near-one factor does not rule out tail miscalibration.
Calling a selected set a null control. The null status must be justified independently of the observed association results; otherwise the check can be circular.

Key takeaway: Assess calibration with complementary, ancestry-aware diagnostics. State the null assumptions behind each reference, distinguish validity from exact uniformity, and account for dependence before assigning probabilities to the number of small p-values.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

A one-degree-of-freedom stratum has a median chi-square statistic of \(0.5459\). Calculate \(\lambda_{\mathrm{GC}}\) and state what it does and does not establish.
Why can a pooled QQ plot look acceptable while an ancestry-stratified plot shows a departure?
For 100 independent valid null p-values, what can be said about the expected number below \(0.05\)? Is that number necessarily Binomial(\(100,0.05\))?
Why is a count of small p-values among correlated variants not generally evaluated with a Binomial(\(m,t\)) reference?
Name one condition a null-outcome simulation should preserve for its calibration comparison to be credible.

Solutions. (1) \(\lambda_{\mathrm{GC}}=0.5459/0.4549=1.200\) (approximately). The median is elevated relative to the one-degree-of-freedom null reference, but the ratio does not identify the cause. (2) Pooling can average over strata with different test behavior, so a localized departure may be diluted in the combined distribution. (3) The expectation is at most \(100(0.05)=5\), since validity gives each crossing probability at most \(0.05\). The count is not necessarily Binomial(\(100,0.05\)); that exact reference requires independent tests with crossing probability \(0.05\), as under independent, exactly uniform null p-values. (4) Dependence means the threshold indicators are not independent, so their sum need not have a binomial distribution even if each marginal p-value is exactly uniform. (5) Examples include covariates, ancestry structure, relatedness, or another feature used in the fitted association model; the simulation should preserve the relevant design features.