Calibration With Few Variant Carriers

Why Few Carriers Challenge Calibration

A burden test compresses variant data into a participant-level measure, then tests whether that measure is associated with a trait. For a rare-variant set, however, many participants may have burden zero and only a handful may carry any included variant. The resulting test statistic can have a coarse, discrete null distribution. A normal or chi-square approximation may then assign tail probabilities that are too small or otherwise inaccurate.

The issue is not that a score test is automatically invalid whenever carriers are rare. Its approximation is justified by large-sample behavior, and adequacy depends on how much information the data contain. The total sample size can be large while the effective information about the burden association remains small. Sparse case-carrier cells, a few participants with high burden values, and estimated covariate effects can all make the nominal reference distribution a poor guide to the actual null distribution.

Definition: Calibration is agreement between a test's reported significance level and its actual rejection probability under the null hypothesis. A test calibrated at level \(\alpha\) rejects a true null hypothesis with probability approximately \(\alpha\), or no greater than \(\alpha\) when a conservative exact procedure is used. Poor calibration means the nominal p-value does not reliably describe that null rejection behavior.

The practical task is to diagnose whether an asymptotic approximation is credible and, if not, use a method whose assumptions fit the sampling and covariate structure. Permutation and exact procedures are not interchangeable fixes: each obtains its null distribution under particular conditions.

Score-Test Approximation and Sparse Information

For a binary trait, a logistic score test evaluates the burden coefficient under the null model, which includes any specified covariates but sets the burden effect to zero. In the simplest case-control example with an intercept only, let \(G_i\) be the participant's burden (here, a carrier indicator), and let \(Y_i\) equal 1 for a case and 0 for a control. Under the null, the fitted case probability is \(\hat p\). The unadjusted score is \(U=\sum_i G_i(Y_i-\hat p)\). With the intercept treated as a nuisance parameter, its efficient information is the burden information after accounting for the intercept.

I_{\mathrm{eff}}=\hat p(1-\hat p)\left[\sum_i G_i^2-\frac{\left(\sum_i G_i\right)^2}{n}\right]

For a binary carrier indicator with \(k\) carriers, this becomes \(\hat p(1-\hat p)k(1-k/n)\). The standardized score is \(Z=U/\sqrt{I_{\mathrm{eff}}}\), and the usual two-sided score-test approximation compares \(Z^2\) with a \(\chi^2_1\) distribution. This reference distribution is an approximation, not the exact finite-sample distribution of the score.

The approximation is supported when the score aggregates enough independent information for its standardized null distribution to be close to normal. A very small carrier count can leave only a few possible score values, so that the null distribution remains markedly discrete. In logistic regression, sparse carrier-by-outcome patterns can also make fitted probabilities and covariate adjustment influential. A large \(n\) by itself does not guarantee a good approximation.

Worked Example: A Logistic Score With Three Carriers

Suppose a hypothetical unadjusted case-control sample has \(n=20\) participants, of whom 5 are cases. Three participants carry at least one included variant, and all three are cases. Test the null hypothesis of no association between carrier status and case status using the intercept-only logistic score test, then compare it with a conditional exact test.

Under the null model, \(\hat p=5/20=0.25\). The carrier score is the sum of \(Y_i-\hat p\) over the three carriers:

$$ U=3(1-0.25)=2.25 $$

For \(k=3\), the efficient information is

I_{\mathrm{eff}}=0.25(0.75)(3)\left(1-\frac{3}{20}\right) =0.1875(3)(0.85)=0.478125

Thus \(Z=2.25/\sqrt{0.478125}\approx 3.254\), and \(Z^2\approx 10.5882\). Comparing this statistic with \(\chi^2_1\) gives an approximate two-sided p-value of about 0.00114. That calculation uses the asymptotic reference distribution, despite only three carriers.

For the conditional exact calculation, hold fixed the 5 cases and the 3 carriers. Under the null of exchangeability of case labels with respect to carrier status, the number \(X\) of carrier cases has a hypergeometric distribution. Its observed value is 3, and

P(X=3)=\frac{\binom{5}{3}\binom{15}{0}}{\binom{20}{3}} =\frac{10}{1140}\approx 0.008772

For a two-sided test based on the absolute score departure from its null expectation, \(E(X)=3(5/20)=0.75\). The observed departure is \(3-0.75=2.25\), the largest possible absolute departure among \(X=0,1,2,3\). Only \(X=3\) is at least this extreme, so the exact p-value is \(10/1140\). The exact p-value is larger than the asymptotic one here; with so few carriers, the discrepancy is practically important. The exact calculation assumes that, under the null, case labels are exchangeable across these participants and that the margins are fixed as described.

Permutation and Exact Alternatives

A permutation procedure constructs a null distribution by rearranging the observed outcomes relative to the fixed genotypes and recalculating the test statistic. If all outcome labels are exchangeable under the null, every permitted rearrangement is equally likely. Enumerating all such rearrangements gives an exact permutation test under that exchangeability assumption. If only a random sample of rearrangements is drawn, the resulting p-value is a Monte Carlo estimate; a common estimate is \((b+1)/(B+1)\), where \(b\) of \(B\) sampled rearrangements are at least as extreme as observed.

Exchangeability is a substantive condition. In a case-control study with no covariates, conditioning on the total number of cases and permuting case labels can be valid under the null of no association when participants are otherwise exchangeable. If ancestry, relatedness, matching, or other covariates affect the outcome, unrestricted label permutation can destroy the structure of the study and produce an invalid reference distribution. Permuting residuals from a fitted model is not automatically exact, especially for binary outcomes. Use a method designed for the actual covariate or sampling structure, such as permutation within valid exchangeability blocks, or a suitable conditional exact procedure.

An exact test obtains its null distribution from the finite-sample design or a specified conditional model, rather than a large-sample reference curve. For a simple binary carrier indicator and binary outcome, conditioning on the case and carrier totals yields the hypergeometric distribution used above. With discrete data, attainable p-values occur in jumps; an exact test may therefore be conservative at a chosen \(\alpha\). “Exact” means exact under its stated null model and conditioning scheme, not assumption-free.

Worked Example: Exhaustive Permutation for a Quantitative Trait

Consider a hypothetical sample of 10 unrelated participants with quantitative trait values \(0,1,2,3,4,5,6,7,8,9\). Exactly two participants are carriers, and the observed carriers have trait values 8 and 9. Assume no covariates or strata are needed and that, under the null, the trait values are exchangeable across participants independently of carrier status.

Use the centered carrier score \(T=\sum_{i:G_i=1}(Y_i-\bar{x})\). Here \(\bar{x}=45/10=4.5\), so the observed statistic is \(T_{\mathrm{obs}}=(8-4.5)+(9-4.5)=8\). Under exhaustive permutation, the carrier pair can be any of \(\binom{10}{2}=45\) pairs, each equally likely. The largest possible pair sum is \(8+9=17\), giving \(T=17-9=8\); the smallest is \(0+1=1\), giving \(T=1-9=-8\). Only these two pairs satisfy \(|T|\geq 8\).

p_{\mathrm{perm}}=\frac{2}{45}\approx 0.04444

This is a two-sided exact permutation p-value under the stated exchangeability assumption. The calculation accounts for the discreteness of the statistic and does not require a normal trait distribution. It would not justify unrestricted permutation if, for example, participants belonged to outcome-relevant strata that needed to be preserved.

Worked Example: Exact Inference Within Covariate Strata

Suppose a hypothetical case-control study has two strata defined by a relevant categorical covariate. Each stratum contains 6 participants. In stratum A there are 3 cases and 2 carriers; both carriers are cases. In stratum B there is 1 case and 1 carrier; that carrier is a control. Test whether the total number \(S\) of carrier cases is unusually large or small, conditional on the case and carrier counts within each stratum. Assume case labels are exchangeable with respect to carrier status within each stratum under the null.

Let \(X_A\) and \(X_B\) count carrier cases in strata A and B. In stratum A, the probability of 0, 1, or 2 carrier cases is respectively \(\binom{3}{0}\binom{3}{2}/\binom{6}{2}=3/15\), \(\binom{3}{1}\binom{3}{1}/\binom{6}{2}=9/15\), and \(\binom{3}{2}\binom{3}{0}/\binom{6}{2}=3/15\). In stratum B, \(P(X_B=1)=1/6\) and \(P(X_B=0)=5/6\). Under the conditional null, these stratum-specific counts are independent, so the distribution of \(S=X_A+X_B\) is their convolution:

P(S=0)=\frac{1}{6},\quad P(S=1)=\frac{8}{15},\quad P(S=2)=\frac{4}{15},\quad P(S=3)=\frac{1}{30}

The observed total is \(S=2\). Its conditional expectation is \(E(S)=2(3/6)+1(1/6)=7/6\), so its absolute departure is \(2-7/6=5/6\). For a two-sided test using absolute departure from this expectation, outcomes at least as extreme are \(S=0,2,3\). Therefore,

p_{\mathrm{exact}}=P(S=0)+P(S=2)+P(S=3) =\frac{1}{6}+\frac{4}{15}+\frac{1}{30} =\frac{7}{15}\approx 0.4667

The calculation respects the stratum-specific case and carrier margins; an unrestricted permutation across all 12 participants would not preserve this design. For more complex covariate structures, exact conditional inference may require conditioning on appropriate nuisance-parameter sufficient statistics or using a method specifically developed for that model. It may also be computationally burdensome.

Choosing and Checking a Calibration Method

The method should follow the data-generating and sampling structure, not a preference for a smaller p-value. A score approximation is attractive and efficient when its large-sample reference distribution is credible. Permutation is attractive when the null makes the relevant outcomes exchangeable under the permitted rearrangements. Exact conditional methods are useful when a finite-sample conditional distribution can be derived or enumerated under defensible assumptions. A parametric bootstrap is another option when an appropriate null model is specified: simulate outcomes under that fitted null, refit as required, and recalculate the statistic. This can assess finite-sample behavior, but it is model-based and is not an exact conditional test.

Describe the information supporting the burden score.
Count carriers overall and by outcome group or relevant stratum. Inspect whether a small number of participants account for most of the burden variation.

State the null model and its nuisance structure.
Specify the outcome model, covariates, sampling or matching structure, and the assumptions behind the proposed reference distribution.

Choose a compatible calibration method.
Use an asymptotic score reference when approximation is credible; otherwise consider valid restricted permutations, an exact conditional distribution, or a clearly identified model-based simulation.

Report what the p-value conditions on.
State whether it is asymptotic, permutation-based, exact conditional, or bootstrap-based, and name the assumptions that make that reference distribution appropriate.

When feasible, comparing an asymptotic result with a conditional or simulation-based calibration is informative. Agreement does not prove that assumptions are correct, and disagreement does not by itself identify which method is valid. The comparison instead signals whether the conclusion depends materially on the reference distribution.

Common Pitfalls and Why Calibration Matters

Treating total sample size as the relevant sample size. A study may have thousands of participants but only a few carriers. The score's information about the burden effect can still be sparse.
Calling every permutation test exact. Permutation is exact only when the rearrangements are equally likely under the null and preserve the required design structure.
Using a pooled exact test despite covariate structure. Conditioning only on the total number of cases does not generally account for ancestry, matching, or other outcome-related covariates. Appropriate strata or a suitable conditional model are needed.
Assuming a smaller p-value is better calibrated. In the three-carrier example, the asymptotic p-value was much smaller than the exact p-value. The difference reflects the reference distributions, not evidence that the smaller value is preferable.
Calling a model-based simulation exact. A parametric bootstrap can be useful, but its accuracy depends on the fitted null model and its assumptions.

Calibration matters because a burden test's p-value is used to compare the observed statistic with what the null model would produce. When only a few participants carry included variants, the tail of that null distribution may be determined by a small number of attainable configurations. Choosing and reporting a defensible reference distribution is therefore part of the inferential result, not a technical afterthought.

Check Your Understanding

Use the assumptions and calculations in this tutorial to answer the following.

Why can a large total sample size coexist with poor calibration of a burden score test?
In the three-carrier example, what assumptions justify conditioning on the total numbers of cases and carriers to obtain a hypergeometric null distribution?
When would unrestricted phenotype-label permutation be inappropriate, and what feature would a valid restricted permutation need to preserve?
What is the difference between an exact conditional test and a parametric bootstrap under a fitted null?
For the two-stratum example, why is it important to condition on the case and carrier counts separately within each stratum?