Calibration Diagnostics for Cell-Type QTL Results

Why Calibration Diagnostics Matter

A small p-value from a cell-type QTL analysis is useful only if the fitted model produces trustworthy uncertainty estimates under its assumptions. A donor-level pseudobulk regression can still be miscalibrated if the mean model is wrong, the residual variance is poorly represented, genotype groups have inadequate overlap with covariates, or a small number of donors dominate the fit. These problems may make test statistics too dispersed and p-values too small, creating more apparent discoveries than the analysis warrants.

Calibration diagnostics do not establish that an association is causal, nor do they replace a model appropriate to the outcome and study design. They ask a narrower question: does the analysis behave as expected when its assumptions should apply? The most informative assessment combines diagnostics at the donor level with checks across a collection of null or control association tests.

The Model and Its Assumptions

For a donor-level pseudobulk expression outcome, a common starting point is a linear regression. Let \(Y_i\) be the expression summary for donor \(i\) in one cell type, \(G_i\) the donor’s genotype dosage, and \(C_i\) a vector of included covariates. The fitted model is

Here \(\beta_G\) is the expected change in the modeled expression outcome per additional effect allele, conditional on the covariates. The regression is an association model: its coefficient alone does not show that genotype caused the expression difference. For the usual linear-model standard errors and tests to be valid, the conditional mean must be correctly specified, donors must contribute independent errors conditional on the model, and the variance assumptions behind the chosen standard error must be reasonable. Exact small-sample \(t\)-tests additionally rely on normally distributed errors; large-sample procedures may use robust standard errors, but those do not fix a misspecified mean or dependent donors.

The observational unit for these diagnostics is the donor, not each cell. A donor with many cells may yield a more precise pseudobulk summary, but those cells do not become independent donors. For a cell-level mixed model, residual and influence checks must respect its fitted conditional structure; the same diagnostic question applies, but the residual definition and reference distribution depend on that model.

Check Genotype Groups and Cell-Count Support

Start by tabulating donor counts at each genotype dosage and checking how those donors are distributed across important covariates, such as ancestry components, batch, sex, or collection site when relevant to the design. The purpose is not to demand a particular genotype ratio. Allele frequencies, ancestry, sampling, and study design all affect observed counts, so a departure from a simple expected ratio is not by itself proof of a problem. Instead, look for sparse dosage groups, coding errors, or near-separation: for example, if nearly all donors with one dosage are from a single batch, genotype and batch effects may be difficult to distinguish.

For each cell type, summarize the number of contributing cells per donor, including the number of donors with no usable cells, and examine these summaries across genotype groups and relevant technical covariates. Unequal cell counts can mean unequal precision in pseudobulk expression and may be associated with dropout or selection into the analyzed sample. A cell-count pattern is a diagnostic signal, not an instruction to adjust automatically: the right response depends on how the cells were sampled and on the intended estimand.

Residual Patterns and Donor Influence

For each donor, the fitted value \(\hat{Y}_i\) is the outcome predicted by the regression, and the residual \(e_i=Y_i-\hat{Y}_i\) is the observed-minus-predicted difference. Plot residuals against fitted values, genotype dosage, and key covariates. A roughly patternless cloud centered around zero is compatible with a suitable mean model. Curvature can signal a nonlinear relationship; a widening or narrowing spread can signal nonconstant variance; and clusters by batch or other covariate can reveal structure the model has not captured. A residual Q–Q plot can help assess whether unusually heavy tails or skewness make normal-theory small-sample inference questionable. No single plot proves that assumptions hold.

A donor may be influential because its covariate pattern gives it high leverage, because its residual is large, or both. In ordinary least squares, let \(h_i\) be the donor’s diagonal leverage, \(p\) the number of fitted regression coefficients including the intercept, and \(\mathrm{MSE}=\sum_i e_i^2/(n-p)\). Cook’s distance summarizes the change in fitted values that would result from deleting that donor:

Large values flag donors for closer examination, not automatic removal. Rules of thumb such as comparing Cook’s distance with \(4/n\) are screening guides, not decision rules. Check the underlying sample, genotype, cell count, covariates, and measurement quality. Then assess whether conclusions are sensitive to the donor, while retaining the prespecified analysis and explaining any justified data correction or sensitivity analysis.

Null Association Plots

Residual checks concern individual fitted models; null association plots assess whether a collection of tests behaves as expected. Useful inputs include known null or negative-control tests and permutations that preserve the study’s relevant structure. A permutation scheme must be valid for the design: freely shuffling genotypes across donors, for example, may be inappropriate when ancestry or other structure makes donors nonexchangeable. A plot made from an invalid null construction cannot establish calibration.

A p-value histogram should be approximately flat under a collection of true nulls with valid continuous tests. A quantile–quantile (Q–Q) plot compares observed test-statistic or p-value quantiles with their null expectations. A departure in the extreme tail can reflect genuine associations as well as miscalibration; broad excess over much of the distribution is more concerning, especially in null controls. The familiar genomic inflation factor for one-degree-of-freedom chi-square tests is

because 0.4549 is the median of a chi-square distribution with one degree of freedom. A value above 1 indicates that the observed median statistic exceeds this reference, but it is not a stand-alone diagnosis: widespread real effects, dependence among tests, or an inappropriate reference set can also shift the distribution. Interpret it alongside the Q–Q plot, the analysis design, and known-null checks. Correcting or rescaling statistics without finding the source of a discrepancy may conceal rather than resolve a modeling problem.

A Practical Diagnostic Sequence

A useful sequence keeps each diagnostic tied to a question and avoids treating one statistic as a pass–fail test.

Common Mistakes

• Calling every excess small p-value a true discovery. A tail excess can reflect association, but null controls and broader distributional patterns help distinguish it from systematic miscalibration.
• Treating a genotype ratio as a universal pass–fail criterion. Genotype frequencies vary with population and sampling. Check coding, group support, and covariate overlap rather than assuming one ratio must hold.
• Interpreting low cell counts as a reason to delete donors automatically. Counts inform precision and possible selection or technical problems. Filtering or adjustment must be justified by the design and analysis target.
• Removing an influential donor because the p-value becomes less favorable. Influence is a prompt to investigate data quality and sensitivity, not a license for outcome-driven exclusion.
• Using a single inflation factor as proof of calibration. It summarizes one feature of the test-statistic distribution and can be shifted by real effects, dependence, or a poor null reference.