Why Set-Level Results Need More Than a P-Value
A rare-variant association result depends on which variants were grouped together, which samples contributed, how variants were weighted, and how the test statistic was calibrated. A statement such as “gene X was associated with the phenotype, \(p=0.004\)” does not identify the tested set or explain what the statistic measures. Another analyst could apply a different transcript, frequency threshold, or annotation rule and obtain a different result while using the same gene name.
A reproducible report therefore gives enough information to reconstruct the analysis and interpret its evidence without implying that a set-level p-value is a single-variant effect estimate. The introductory material on rare-variant set testing distinguishes tests with different genetic-architecture assumptions; here the emphasis is on recording the choices that produced the reported result.
What Must Be Specified
Begin with an unambiguous description of the set. State the genomic reference build and coordinates, gene or region definition, transcript or annotation source when relevant, and whether the set is defined by coding consequence, predicted function, or another criterion. Give the exact allele-frequency or minor-allele-count rule, the population or sample in which frequency was estimated, and any quality-control filters. Explain how multiallelic sites, missing genotypes, and variants that fail a filter were handled. A gene symbol alone is not a set definition.
Describe the analyzed participants: total sample size, case and control counts for a binary outcome, ancestry or analysis strata, and exclusions that affect the sample. State the outcome definition and measurement scale. Report the genotype representation, such as allele count dosage, and whether imputation or hard calls were used. For a burden or variance-component test, document the variant weights and their inputs, including any functional annotation or frequency-dependent weighting formula. If weights are normalized, give the normalization.
Then specify the association model. For example, a logistic model for a binary outcome may include an intercept, the variant-set contribution, and prespecified covariates such as age, sex, and ancestry principal components. State the covariate coding and the missing-data approach. When the result is based on a score test, identify the null model used to obtain the score and its covariance. If a method handles separation or uses a specialized correction, name that implementation and its settings. These details matter because they can change both the statistic and its null distribution.
Finally, distinguish the observed statistic from its calibration. Report whether the p-value came from a reference distribution, a numerical mixture-distribution method, resampling, or another procedure. For resampling, give the number of replicates, the resampling scheme, and the empirical p-value formula. Report software and version, convergence or numerical warnings, and the multiple-testing family and adjustment when a corrected result is presented.
Statistics and Their Interpretation
For a score-based set test, let \(U\) be the vector of variant score contributions under the fitted null model, and let \(V\) be its null covariance matrix. With a diagonal matrix \(W\) of variant weights, a common quadratic-form statistic is \(Q=U^\mathsf{T}WU\). A burden test instead aggregates variant scores in a specified direction; for a weight vector \(a\), its standardized score is \(Z=(a^\mathsf{T}U)/\sqrt{a^\mathsf{T}Va}\), when the denominator is positive. State the exact statistic used: names such as “burden” or “kernel” are not enough to recover all implementation choices.
A set-level p-value quantifies how unusual a statistic at least as extreme as the observed one would be under the specified null model and calibration. It is not the probability that the null hypothesis is true, and a small value does not establish that every included variant has an effect. A burden-model coefficient, when estimated and reported, has a different interpretation: its scale depends on how the burden was constructed and on the link function.
Worked Example: Defining a Tested Set Precisely
Suppose a hypothetical analysis tests coding variants in one gene for a binary disease outcome among 8,200 unrelated participants, including 1,600 cases and 6,600 controls. The analysis uses a specified genome build and transcript, includes exonic protein-altering variants with minor-allele frequency below 1% in the analysis ancestry, and excludes variants failing call-rate or allele-balance thresholds. It uses allele-count dosages and a logistic null model adjusted for age, sex, and ten ancestry principal components.
A reproducible report would identify the gene and exact transcript, build, annotation source and version, frequency reference, threshold, quality-control rules, sample counts, dosage representation, covariates, and the weighting formula. For example, if the test assigns weight \(w_j\) to variant \(j\) using a stated frequency-dependent formula and multiplies it by a functional-annotation weight, the formula and all inputs should be reported—not merely “functional weights were used.”
The set contains only variants that pass every stated inclusion rule. If 14 variants pass and 2 fail the call-rate threshold, report that 14 were tested and describe the exclusions. This count allows readers to check that the defined set and the reported analysis agree; it does not by itself establish that the analysis is well calibrated.
Worked Example: Reporting a Burden-Model Effect
In a hypothetical logistic burden analysis, define each participant’s burden as the sum of dosage counts over included variants, each multiplied by its prespecified weight. Suppose the fitted coefficient for one burden unit is \(\hat{\beta}=0.12\), with \(SE=0.05\). The estimated odds ratio per burden unit is
Under the stated large-sample normal approximation, the Wald statistic is \(Z=0.12/0.05=2.40\), giving a two-sided p-value of approximately 0.0164. A 95% confidence interval on the log-odds scale is \(0.12\pm1.96(0.05)=(0.022,0.218)\). Exponentiating the endpoints gives an odds-ratio interval of approximately \((1.022,1.244)\).
A context-appropriate report is: “In this hypothetical model, one unit of the defined weighted burden was associated with estimated odds 1.13 times as high; the 95% confidence interval was 1.02 to 1.24, and the Wald p-value was 0.016.” This estimate pertains to the burden scale and model. It is not an estimated effect for every variant in the set, and it should not be presented as causal evidence.
Worked Example: Reporting a Quadratic-Form Calibration
Suppose, hypothetically, a two-variant score vector has observed values \(U=(\sqrt{1.75},1)^\mathsf{T}\), covariance \(V=I\), and weight matrix \(W=\operatorname{diag}(1,0.25)\). The quadratic-form statistic is
If the null score vector is assumed to be multivariate normal with mean zero and covariance \(I\), the eigenvalues of \(V^{1/2}WV^{1/2}\) are 1 and 0.25. Thus the null distribution is \(\chi^2_1+0.25\chi^2_1\), with independent chi-square variables. A numerical mixture-distribution calibration, such as a Davies calculation, gives an upper-tail p-value of approximately 0.196 for \(Q=2\). The report should give the statistic, weights or eigenvalues, calibration method, and p-value, and should state that the mixture calibration relies on the specified normal null approximation.
Sensitivity Analyses and Diagnostics
A main result is easier to assess when the report shows whether it depends on defensible analysis choices. Sensitivity analyses might vary a prespecified frequency threshold, use a neighboring but biologically justified set boundary, compare alternative functional weights, or assess a different supported calibration. For every alternative, identify what changed and report the resulting set size, statistic, and p-value. Do not present a menu of many unreported analyses and then highlight only the smallest p-value.
Diagnostics should address both data construction and inference. Check that the variant count matches the inclusion rules, that allele frequencies and missingness are plausible, that weights are finite and aligned with the correct variants, and that the sample and phenotype counts are internally consistent. For model-based tests, report convergence problems and any remedial procedure. For resampling or numerical calibration, report adequacy checks and warnings. A result that changes sharply when a small number of variants are removed deserves particular attention; variant-level score contributions and influence diagnostics can help identify that dependence.
Worked Example: Summarizing a Boundary Sensitivity Analysis
Consider a hypothetical primary analysis with 14 qualifying variants in a gene and a calibrated set-level p-value of 0.004. Investigators repeat the analysis using a prespecified neighboring boundary that yields 17 variants and \(p=0.012\), and a stricter frequency threshold that yields 9 variants and \(p=0.021\). Assume the same participants, outcome model, and calibration procedure were used in all three analyses.
A transparent summary gives all three set definitions, variant counts, and p-values, and identifies the 14-variant analysis as primary if that choice was prespecified. The alternatives show that evidence weakened under both changes but remained below 0.05 in these hypothetical analyses. They do not prove that the association is robust to every reasonable choice; nor should the three p-values be described as independent replications. If the alternatives were selected after examining results, that exploratory selection should be disclosed, and the multiplicity implications should be considered.
Common Reporting Mistakes
- Reporting only the gene name and p-value. This leaves the tested variants and analysis unrecoverable. Give the build, boundary, annotation, frequency rule, filters, and set size.
- Calling a set-level p-value an effect size. The p-value summarizes evidence against a specified null; it does not provide a common effect estimate for all variants. Report a coefficient only when its model and scale are defined.
- Describing weights vaguely. “Weighted by frequency” is insufficient without the formula, frequency source, and any annotation multiplier.
- Assuming covariance alone justifies a chi-square mixture. A weighted chi-square result requires an appropriate normal null distribution or a justified approximation, not merely a score vector with mean zero and covariance \(V\).
- Hiding sensitivity results. Omitting alternative reasonable set definitions can make a fragile result appear definitive. Identify primary and exploratory analyses and report what changed.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- Why is “gene A, \(p=0.01\)” not a reproducible set-level result?
- A burden-model coefficient is reported without defining the burden weights. What interpretation is missing?
- What distributional condition is needed to derive a weighted chi-square calibration from a score vector and covariance matrix?
- A primary set has 12 variants and an alternative boundary has 15. Name three other pieces of information needed to interpret the sensitivity comparison.
Solutions. (1) The gene label and p-value do not specify the build, boundaries, transcript or annotation, inclusion filters, sample, model, statistic, or calibration. (2) The reader cannot determine what one unit of burden represents, so the coefficient’s scale and odds-ratio interpretation are unclear. (3) The null score vector must be multivariate normal, or be justified as approximately multivariate normal, with the stated mean and covariance; covariance alone is insufficient. (4) Report the exact boundary or inclusion-rule change, the statistics and calibrated p-values, and whether sample, model, weights, and calibration were held fixed; also identify which analysis was primary and whether alternatives were prespecified.