Why Rare-Variant Set Tests Can Fail
A rare-variant set test combines information across variants, often because individual variants are too uncommon for stable association estimates. That combination creates opportunities for mistakes: the analyst can change which variants are combined, how much each contributes, or which test is reported after seeing the outcome data. A small nominal p-value is not reliable evidence if the analysis that produced it was selected or calibrated incorrectly.
This tutorial focuses on four related risks: data-driven changes to the variant set, unjustified weights, effect-direction heterogeneity, and choosing among tests without accounting for that choice. These are not reasons to avoid burden, variance-component, or omnibus tests. They are reasons to define the analysis carefully and ensure that the reported null distribution matches what was actually done.
For the examples below, assume that the phenotype model and covariate adjustment are correctly specified, the null score vector is adequately approximated by a mean-zero multivariate normal distribution with covariance \(V\), and the variant set is defined before examining association results. These are simplifying conditions, not universal guarantees. In small samples, with very sparse carriers, or under model misspecification, a nominal asymptotic calibration may not be adequate. Score covariance must also reflect dependence between variants, including linkage disequilibrium.
Changing the Set After Looking
A set definition may use genomic boundaries, functional annotations, allele-frequency rules, or quality-control criteria. It is legitimate to compare scientifically motivated definitions as sensitivity analyses. The problem arises when the analyst inspects association results and then drops variants, adds annotations, or changes frequency cutoffs specifically to obtain a smaller p-value, but reports the selected test as if it had been the only prespecified analysis.
Worked Example: Selecting the More Significant Set
Suppose an analyst tests two alternative definitions of a gene’s variant set. Under the global null, assume for illustration that the two p-values are independent and each test is correctly calibrated. The analyst reports whichever p-value is smaller. What is the probability that at least one p-value is at most 0.040?
For one test, the probability of a p-value greater than 0.040 is \(1-0.040=0.960\). Under the independence assumption, the probability that both p-values exceed 0.040 is \(0.960^2=0.9216\). Therefore,
Thus, selecting the smaller p-value gives a 7.84% chance of a result at or below 0.040 under this two-test null scenario, not 4.00%. A Bonferroni threshold for two tests at familywise level 0.05 is \(0.05/2=0.025\); equivalently, the Bonferroni-adjusted p-value for a minimum observed p-value of 0.040 is \(2(0.040)=0.080\), capped at 1. The independence calculation is exact only under the stated assumption. If the tests are dependent, their joint probability must be handled appropriately; the Bonferroni bound still controls the familywise error rate at 0.05 when each individual test is valid.
The same logic applies when the analyst tries many boundaries, annotation filters, or allele-frequency cutoffs. Merely describing the final set does not undo selection. A defensible analysis prespecifies a primary set definition or accounts for the search across definitions using a valid joint procedure. Sensitivity analyses should be labelled as such, with the range of choices and their results made clear.
Weights Must Be Justified and Calibrated
Weights express how variant contributions enter a test. They may reflect a prespecified scientific rationale or a defined modeling strategy; they are not automatically justified because they are common in software or because they produce a compelling result. Choosing weights after inspecting score signs, effect estimates, or p-values is another form of selection. In addition, the variance calculation must use the same weights as the statistic and account for the covariance among scores.
For a burden score, the standardization is based on \(w^\mathsf{T}Vw\), not on the sum of squared weights alone unless the score covariance matrix has the required form. For two scores with variance 1 and correlation \(\rho\), the variance of their unweighted sum is \(2+2\rho\). This follows by expanding the variance:
Worked Example: Ignoring Correlation Understates Uncertainty
Consider two null score contributions with variance 1 each and correlation \(\rho=0.90\). Suppose their observed values are \(U_1=2\) and \(U_2=2\). The unweighted burden score is \(T_B=2+2=4\). If the covariance is correctly included, its variance is \(1+1+2(0.90)=3.8\), so its standard error is \(\sqrt{3.8}=1.949\) and its standardized statistic is \(4/1.949=2.052\). The corresponding two-sided standard-normal p-value is approximately 0.0402.
If the analyst incorrectly assumes independence, the estimated variance is \(1+1=2\), the standard error is \(\sqrt{2}=1.414\), and the statistic becomes \(4/1.414=2.828\), with a two-sided standard-normal p-value of approximately 0.00468. That calculation makes the evidence appear much stronger because it treats highly correlated scores as if they supplied independent information. The numerical comparison assumes the stated normal score approximation; an actual analysis should use the covariance and null calibration appropriate to its model and data.
A useful diagnostic is to inspect how the statistic and its calibration change when score covariance is included, and to check whether the implementation uses the same variant ordering, weights, and set definition throughout. Extreme sensitivity to small weight changes can signal that a result depends on a few variants or on an unstable modeling choice. The remedy is not to choose whichever weights yield the smallest p-value; it is to justify the weights in advance and report prespecified sensitivity analyses.
Effect-Direction Heterogeneity and Test Choice
A burden test is most naturally aligned with a model in which included variants have effects in a common direction, subject to the coding and weighting scheme. If some effects are positive and others negative, their score contributions can cancel. A variance-component test instead uses squared or otherwise non-directional contributions, so opposite signs do not cancel in the same way. These approaches answer different questions and can have different power under different genetic architectures; neither is universally best.
Worked Example: Opposite Directions Cancel in a Burden Score
For a simplified illustration, suppose two variant score contributions have null covariance equal to the identity matrix, so each has null variance 1 and they are independent. The observed scores are \(U_1=3\) and \(U_2=-3\). With equal burden weights, \(T_B=U_1+U_2=0\), and its null standard error is \(\sqrt{1+1}=\sqrt{2}\). Thus the standardized burden statistic is \(0/\sqrt{2}=0\), which gives a two-sided standard-normal p-value of 1.00.
For the unweighted quadratic statistic, \(Q=U_1^2+U_2^2=3^2+(-3)^2=18\). Under the simplified null assumptions, \(Q\) has a chi-square distribution with 2 degrees of freedom. Its upper-tail p-value is \(P(\chi^2_2\geq18)=e^{-18/2}=e^{-9}\approx0.0001234\). In this constructed example, the burden score cancels while the quadratic statistic registers large contributions in both directions. The example does not establish that either score pattern is a real association; it shows why test choice must reflect the plausible effect-direction structure and why the null reference distribution matters.
A common mistake is to interpret a null burden result as evidence that no variant in the set is associated. Cancellation is one possible explanation, as are low power, poor variant inclusion, or model and calibration problems. Conversely, a small variance-component p-value does not identify which variant is causal or establish a shared direction of effect. Variant-level follow-up and biological interpretation require separate evidence.
Omnibus methods can combine or adapt across burden-like and variance-component components, but their p-values must account for the adaptation. Reporting the smallest component p-value as if that test had been chosen in advance is not valid. Calibrated omnibus procedures address a specified search across components; they do not automatically account for additional analyst decisions, such as trying many sets or weights outside that procedure.
Selection, Calibration, and Practical Diagnostics
A test is calibrated when, under its null model and stated assumptions, its reported p-values have the intended distribution (or controlled error rate). A correctly computed statistic can still have an invalid p-value if the reference distribution ignores estimation, dependence, sparsity, or the search used to select the final test. A small p-value is evidence against the specified null only to the extent that the model and calibration are credible.
Worked Example: Searching Across Several Tests
Suppose five independent, valid tests are run under a global null and the analyst reports the smallest p-value. The chance that all five p-values exceed 0.01 is \(0.99^5=0.950990\). Therefore the chance that at least one is at most 0.01 is \(1-0.950990=0.04901\), approximately 4.90%. A nominal 1% cutoff has become an approximately 4.90% chance of at least one such result in this particular independent search.
If the five tests are dependent, that independence calculation need not hold. A Bonferroni threshold of \(0.05/5=0.010\) controls the familywise error rate at no more than 0.05 when each test is valid, regardless of the dependence structure. A valid joint calibration can use the known dependence structure more directly. The important point is that the reported p-value must correspond to the selection process actually used, not merely to the final test statistic viewed in isolation.
In practice, check whether the null model and score covariance are appropriate for the phenotype and sampling design, whether sparse data make the asymptotic reference questionable, and whether any variants, weights, or tests were selected after viewing results. Compare observed null behavior with suitable calibration diagnostics when available. A failure of calibration is not fixed by changing the biological interpretation of a p-value; it calls for correcting the analysis or qualifying the inference.
| Analysis choice | Potential failure | Practical check |
|---|---|---|
| Set membership | Variants added or removed after looking at association results | Document the prespecified rule; account for searches across alternative definitions |
| Weights | Weights selected from observed phenotype or score patterns | State the rationale and timing; use matching covariance and calibration |
| Effect architecture | Opposite directions make a burden score cancel | Consider whether common-direction effects are plausible; compare appropriately calibrated test types |
| Test selection | The smallest of several nominal p-values is reported unadjusted | Use a calibrated omnibus procedure or a valid multiplicity adjustment for the actual search |
These safeguards have limits. Prespecification cannot repair a misspecified phenotype model, an incorrect variant annotation, or poor data quality. A multiplicity correction for a documented set of tests does not cover unrecorded exploratory choices. Nor does a robust omnibus p-value remove the need to assess whether the variant set and scientific question are meaningful. The goal is a transparent, calibrated test of a defined hypothesis—not a guarantee that every assumption is true.
Common Mistakes
- “I only changed the set a little.” Even a small change can be consequential if it was chosen because it improved the result. Treat result-driven alternatives as a search, not as a single prespecified test.
- “The weights are standard, so they need no justification.” A familiar weighting scheme still needs to be specified and applied consistently. Data-dependent weights require a procedure that accounts for their selection.
- “A null burden test rules out association in the set.” Opposing score contributions may cancel. A burden result concerns its tested combination, not every possible pattern of variant effects.
- “The smallest p-value is the result.” If several tests or definitions were examined, the minimum has a selection distribution. Report a calibrated omnibus or multiplicity-adjusted result for the search performed.
- “Rare variants are independent because each is uncommon.” Rarity does not imply zero covariance. Use the score covariance appropriate to the genotypes and model.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Two independent valid tests are each evaluated at 0.05. What is the probability that at least one has a p-value at most 0.05 under the global null?
- Two unit-variance scores have correlation 0.50. What is the variance of their unweighted sum?
- Why can a burden score be near zero when both variant-level scores are large in magnitude?
- An analyst tries four set definitions and reports the smallest p-value without adjustment. What is the central inferential problem, and name one valid general remedy.
Solutions. (1) The probability is \(1-(1-0.05)^2=1-0.95^2=1-0.9025=0.0975\), or 9.75%. (2) The variance is \(1+1+2(0.50)=3\). (3) With equal burden weights, a positive and a similarly sized negative score contribute in opposite directions and may cancel in the sum. (4) The minimum p-value was selected from several tests, so its nominal single-test calibration does not apply to the search. One remedy is a valid multiplicity adjustment, such as a Bonferroni correction; a suitable joint calibration is another possibility.