Calibrating SKAT with Few Cases or Small Samples

Why Calibration Needs Attention

SKAT assesses whether a set of genetic variants is associated with a trait by comparing a kernel-based score statistic with its null distribution. For binary traits, the commonly used reference is a weighted mixture of chi-square distributions derived using a large-sample approximation. With few cases, a small total sample, or highly imbalanced case and control counts, that approximation may be inaccurate. The issue is calibration: under the null hypothesis, does the procedure report small p-values at the intended frequency?

The approximation can be strained when the binary outcome contains little information for estimating the null model, or when rare variants produce scores dominated by only a few carriers. A mathematically computed mixture-distribution p-value is not necessarily a well-calibrated small-sample p-value. Resampling can provide a useful check, but only if the simulated or permuted outcomes are generated under a null model that matches the analysis and sampling design.

The Asymptotic SKAT Reference

Let \(Y_i\) be the binary outcome for individual \(i\), let \(X\) be the \(n\)-by-\(q\) matrix of covariates including an intercept, and let \(G\) be the \(n\)-by-\(m\) genotype matrix for the variants in the set. Let \(W\) be the diagonal matrix of prespecified variant weights. Under the null model with no genetic-set effect, fit a logistic regression and write the fitted probabilities as \(\hat{\mu}_i\). Define \(r=Y-\hat{\mu}\), \(V=\operatorname{diag}\{\hat{\mu}_i(1-\hat{\mu}_i)\}\), and the weighted genotype kernel \(K=GWWG'\). The SKAT score statistic considered here is \(Q=r'Kr\).

The standard large-sample reference uses the covariate-adjusted score covariance \(P=V-VX(X'VX)^{-1}X'V\). When the null model is identifiable and \(X'VX\) is invertible, \(P\) accounts for estimating the covariate effects. The eigenvalues \(\lambda_1,\ldots,\lambda_n\) of \(K^{1/2}PK^{1/2}\), including zero eigenvalues, determine the reference distribution.

Here the \(\chi^2_{1,j}\) variables are independent chi-square random variables with one degree of freedom. This is an approximation, not a finite-sample identity. It relies on the score behaving approximately like a multivariate normal random variable under the null and on the fitted nuisance parameters being sufficiently well behaved. With sparse outcomes or rare genotype patterns, those approximations may be poor even if the mixture calculation itself is carried out accurately.

Why the Mixture Approximation Can Fail

The quadratic-form argument gives a useful justification for the mixture reference. Under the null, approximate the covariate-adjusted score residual by a mean-zero normal vector \(e\) with covariance \(P\). Then \(Q\) is approximated by \(e'Ke\). Write \(e=P^{1/2}z\), where \(z\) has independent standard normal components. The statistic becomes \(z'P^{1/2}KP^{1/2}z\). Diagonalizing this symmetric matrix expresses the quadratic form as a sum of eigenvalues times independent squared standard normals, which are \(\chi^2_1\) variables.

The weak point in small samples is the normal approximation for the score and, in practice, the estimation of the null model used to construct it. Binary outcomes are discrete; a few cases can make the score distribution far from normal. Rare-variant kernels can also concentrate the statistic on a small number of carriers. Resampling addresses the reference-distribution approximation by generating new outcomes under the null and recalculating the statistic. It does not repair a misspecified null model, unstable fitting, incorrect genotype data, or an unsuitable sampling scheme.

Parametric Null Resampling

For a binary trait, a parametric bootstrap generates outcomes from the fitted null model while holding the observed genotypes and covariates fixed. This estimates a null distribution conditional on those observed design variables. The model and the resampling scheme must correspond: for example, use the same covariates, link function, and case-control sampling constraints that define the test being calibrated.

In an independent Bernoulli logistic model with covariates fixed, the basic generation step draws \(Y_i^*\) independently from a Bernoulli distribution with success probability \(\hat{\mu}_i\). The null model is refit in each replicate when the test procedure estimates its nuisance parameters from the outcome; the same fitting and score calculation used for the observed data are then repeated. This makes the resampling reproduce the analysis pipeline rather than compare the observed statistic with a distribution based on a different fitting rule.

This plug-in bootstrap is not an exact test: it treats estimated null parameters as if they were the generating values. It can nevertheless be a useful finite-sample calibration check when the fitted model is stable and appropriate. If null fits repeatedly fail, or fitted probabilities are extreme because of sparse covariate patterns or separation, the bootstrap does not make those problems disappear. They require attention to model specification and estimation.

Permutation Is Not Automatically Valid

A permutation test obtains a null distribution by rearranging observed outcomes while leaving genotypes and covariates fixed. It is valid only when the null makes the rearranged outcomes exchangeable under the chosen permutation scheme. With an intercept-only model and a design that fixes the number of cases, uniform permutation of case labels can be appropriate under the null of no association, subject to the study design. With covariates that predict the outcome, arbitrary label permutation generally breaks the relationship between outcomes and covariates and is not a valid substitute for model-based resampling.

Permutation within strata can be appropriate when outcomes are exchangeable within those strata under the null. For matched sets, the permissible rearrangements must respect the matching structure. Simply permuting fitted residuals from a logistic regression is not automatically exact: binary residuals are not generally exchangeable, particularly when fitted probabilities differ. When the design imposes fixed case counts or matching, use a null generator that respects those constraints rather than assuming independent Bernoulli outcomes.

Practical Choices and Common Pitfalls

Before choosing a calibration, identify what the study design and fitted null model require. The following distinctions are central:

A common mistake is to say, “There are only a few cases, so I will permute case labels at random.” That reasoning addresses the small sample but ignores whether labels are exchangeable. If covariates predict case status or the design fixes case counts within groups, unrestricted permutation generates outcomes under the wrong null. The correction is to use resampling from the fitted null model or a conditional permutation scheme that preserves the relevant design constraints.

Another pitfall is to treat a bootstrap p-value as exact or as proof of association. A resampled p-value is evidence against the specified null model, conditional on the resampling assumptions and observed design variables. It does not establish causation, and it does not validate the model, variant data, or study design. Calibration should be selected before interpreting the result, with any instability and simulation uncertainty reported transparently.