Misspecification and Calibration in LMM Association

Why Calibration Matters

A fixed-effect association test is calibrated when, under its null hypothesis, its p-values behave as claimed. For example, if a test is calibrated at significance level \(\alpha=0.05\), then repeated studies satisfying the null and the model assumptions reject that null about 5% of the time. Calibration matters because an incorrectly small standard error or an inappropriate reference distribution can make null variants appear associated.

In a genetic linear mixed model, calibration depends on more than the fixed-effect test formula. It also depends on whether the covariance model represents the dependence among observations, whether relevant mean differences are controlled, and whether the phenotype is compatible with the assumed error distribution. A model can converge and produce plausible variance-component estimates while still yielding poorly calibrated association tests.

How Covariance Misspecification Changes a Test

Let \(y\) be an \(n\)-vector of phenotypes and let \(X\) be a fixed-effects design matrix that includes a tested genetic predictor and any adjustment covariates. The working linear mixed model is \(y=X\beta+e\), with \(e\sim N(0,V)\). The covariance \(V\) may include residual variance and one or more random-effect covariance components. Generalized least squares uses this covariance to estimate \(\beta\) and to calculate its model-based standard error.

For the calibration argument, suppose the mean model is correct but the true error covariance is \(V_0\), not the fitted working covariance \(V\). Treat \(V\) as fixed for the moment. The generalized least-squares estimator and its actual covariance are

The usual model-based covariance replaces \(V_0\) by \(V\), giving \((X^\mathsf{T}V^{-1}X)^{-1}\). If these covariances differ in the direction relevant to the tested coefficient, the reported standard error does not match the estimator's actual variability. An estimate can still target the correct mean coefficient when the mean model is correct, yet its Wald statistic can have the wrong null distribution.

A Short Derivation of the Calibration Effect

Consider a single coefficient or contrast \(\theta=c^\mathsf{T}\beta\). Its estimate is a linear function \(a^\mathsf{T}y\), where \(a^\mathsf{T}=c^\mathsf{T}(X^\mathsf{T}V^{-1}X)^{-1}X^\mathsf{T}V^{-1}\). Under the null, assume the mean of this estimate is zero and, for this derivation, that \(V\) and \(V_0\) are known and the errors are Gaussian. The model-based test divides the estimate by its standard deviation calculated from \(V\). Its actual variance is instead determined by \(V_0\). Therefore its null distribution is

This follows because a linear function of a Gaussian vector is Gaussian, with variance \(a^\mathsf{T}V_0a\); dividing by \(\sqrt{a^\mathsf{T}Va}\) divides that variance by \(a^\mathsf{T}Va\). Correct calibration requires the ratio to equal one. When variance components are estimated, Wald statistics generally rely on an approximation, so covariance misspecification can compound the approximation error.

Unmodeled Structure Is Not Only a Covariance Problem

Genetic relatedness and other dependencies can make observations correlated, which a suitable random-effect covariance may represent. But population structure can also create differences in the expected phenotype across groups. If group membership is associated with allele dosage and phenotype, omitting group-related fixed effects can create an apparent genetic association even when there is no within-group genetic effect. A covariance model for random dependence does not automatically remove a systematic difference in the conditional mean.

For example, ancestry covariates or other measured confounders may belong in the fixed-effects design, while relatedness may be represented in the covariance. The appropriate specification depends on the scientific design and data. Adding covariates without thought is not a universal remedy: covariates should represent plausible sources of mean differences and be chosen without using the observed association result to manufacture a desired p-value.

Phenotype Outliers and Test Stability

A phenotype outlier is an observation unusually far from what the fitted model predicts. Under a Gaussian mixed model, a very extreme residual may be improbable, and a single observation can have substantial influence on the estimated fixed effect, the residual variance, or the variance components. Influence is especially important when an observation also has an unusual genotype value or high leverage. An outlier does not automatically mean the observation is erroneous: it may reflect a valid extreme phenotype, a recording problem, or a missing feature of the model.

Residual checks should respect the fitted covariance. When possible, inspect residuals standardized or transformed using the fitted covariance, alongside ordinary residuals and relevant covariates. A large residual may indicate a phenotype issue or a mean-model problem; a pattern across related individuals or groups may instead suggest that the covariance or fixed-effects structure is inadequate. Removing observations solely because their removal improves a p-value is not a defensible calibration strategy.

Using Null Simulations and Diagnostic Plots

A direct way to assess empirical calibration is to generate repeated datasets under a null model and apply the intended association-testing procedure to each. For a parametric null simulation, keep the observed genotypes and design covariates fixed, generate phenotypes under a model with the tested genetic effect set to zero, and retain relevant dependence among individuals. The complete analysis pipeline should be repeated, including any variance-component estimation used in practice. If the simulation uses only the fitted covariance, it assesses the procedure under that assumed model; it cannot by itself show that the assumed covariance is correct. Stress tests can instead generate data under plausible alternative covariance or mean structures.

For each simulation, record whether the null p-value is below a prespecified \(\alpha\). The proportion of rejections estimates the empirical type I error for that simulation design. Its Monte Carlo uncertainty matters: with \(R\) independent simulations and estimated rejection proportion \(\hat{p}\), the approximate Monte Carlo standard error is \(\sqrt{\hat{p}(1-\hat{p})/R}\). A small apparent departure from \(\alpha\) may be simulation noise, especially when \(R\) is small.

Diagnostic plots help locate, but do not prove, the source of a problem. A quantile-quantile plot of null association p-values compares observed values with those expected under a uniform null distribution. Broad upward departure can indicate excess small p-values; departure concentrated in the extreme tail may have a different explanation, including genuine effects in a genome-wide analysis. A p-value histogram can show broad distortion or isolated excess near zero. Residual plots against fitted values, genotype, ancestry variables, or relatedness summaries can reveal patterns, unequal spread, and influential observations. These plots should be interpreted together with the scientific design and simulation results.

A common summary is the genomic inflation factor, often calculated as the median observed association chi-square statistic divided by the median of a one-degree-of-freedom chi-square distribution under the null. A value above one can be consistent with test-statistic inflation, but it is not a diagnosis: true polygenic effects can also raise the genome-wide median. It should not replace examination of null simulations, covariates, residuals, and the relationship structure.

A Practical Calibration Review

A disciplined review separates distinct failure modes rather than treating every excess of small p-values as the same problem. First inspect whether the fixed-effects mean structure includes plausible confounders. Then assess whether the covariance captures dependence relevant to the cohort and whether residual diagnostics suggest outliers or unmodeled patterns. Finally, test the full procedure on null simulations designed to represent the assumptions and plausible departures that matter for the study.