Why Sensitivity Analysis Matters
A rare-variant burden test depends on how variants are grouped and combined. The mask determines which variants enter the burden; a frequency cutoff determines which variants qualify; and weights determine how each included genotype contributes. These choices can be scientifically reasonable yet produce different burdens and different association results. Sensitivity analysis asks whether the substantive conclusion is stable across a planned set of plausible choices.
A sensitivity analysis is not a search for whichever configuration gives the smallest p-value. Its purpose is to examine how conclusions depend on defensible analysis choices, while keeping other parts of the analysis fixed where possible. This distinction matters because several analyses of the same gene and participants are related, not independent replications. Similar results across them can indicate stability to those choices, but do not by themselves provide several independent confirmations.
Plan Comparisons That Answer a Clear Question
Start by stating the primary configuration and why it is scientifically appropriate. Then list a limited set of alternatives that address specific uncertainties. For example, a researcher may ask whether the result depends on including variants up to a slightly less restrictive frequency cutoff, on using a broader functional mask, or on assigning frequency-based rather than equal weights. The alternatives should be plausible before the association results are inspected, not invented to improve a p-value.
Whenever possible, change one feature at a time. Comparing the primary analysis with a configuration that changes the mask, cutoff, and weights simultaneously cannot identify which choice accounts for a difference. A small, structured set of comparisons makes interpretation clearer. A full factorial grid is sometimes appropriate, but its size and the resulting multiplicity should be recognized in advance.
Keep the phenotype definition, sample, covariates, genotype quality control, and testing method fixed across configurations unless one of those is itself the subject of a separate sensitivity question. Otherwise, differences in results have multiple possible explanations. Also record the number of variants and carriers contributing to each burden. A mask or cutoff that retains very few carriers can make a test less stable even if its p-value is small.
Name the mask, frequency cutoff, weights, trait model, and covariate adjustment before examining association results.
For each alternative, state the uncertainty it addresses and change as few features as practical.
Show the configuration, variant and carrier counts, effect estimate where meaningful, test statistic, and p-value for every configuration.
Describe what is consistent across choices, and account for the set of tests if inferential claims depend on which result is strongest.
Worked Example: Defining a Sensitivity Set
Suppose a hypothetical study tests whether rare coding variation in a gene is associated with a quantitative trait. The team has chosen a damaging-variant mask, a minor-allele-frequency cutoff of 1%, and equal weights as the primary configuration. It wants to assess sensitivity to the cutoff and to the mask, without changing the trait model or covariates.
For illustration, define a participant’s burden under configuration \(c\) as \(B_i^{(c)}=\sum_j I_{ij}^{(c)}w_j^{(c)}G_{ij}\), where \(G_{ij}\) is the participant’s allele count at variant \(j\), \(I_{ij}^{(c)}\) indicates that variant \(j\) is included under the configuration, and \(w_j^{(c)}\) is its weight. This definition makes clear that changing the mask, cutoff, or weights can change the burden itself.
The team plans three configurations: (1) the primary damaging mask, 1% cutoff, and equal weights; (2) the same mask and weights with a 0.5% cutoff; and (3) a broader, prespecified functional mask with the original cutoff and equal weights. These comparisons address two particular choices separately. A fourth analysis changing all three features at once would be harder to interpret, so it is not included as a substitute for these one-feature comparisons.
As a check on the burden calculation, suppose that under the primary configuration one hypothetical participant carries one included variant with genotype count 1 and another with count 2. With equal weights of 1, the burden is \(B_i^{(1)}=1(1)+1(2)=3\). If the second variant is excluded by the stricter cutoff in configuration 2, the burden becomes \(B_i^{(2)}=1(1)=1\). This arithmetic shows why configurations are different tests of differently defined burdens, even though they use the same participants.
The plan should also retain the number of variants and carriers in each configuration. If the broader mask doubles the number of included variants but adds almost no carriers, a changed result may reflect a different burden definition and sparse information, not evidence that one biological interpretation is correct.
What Counts as a Robust Signal?
Robustness is a contextual judgment, not a universal threshold. It is more convincing when the primary result and plausible alternatives tell a similar scientific story: for example, estimated associations point in the same direction, their magnitudes are not sharply contradictory on a comparable scale, and the evidence is not confined to a single narrowly defined configuration. It is less convincing when only one choice produces a small p-value and nearby reasonable choices show little evidence or a substantially different estimated effect.
For burden regressions, coefficients require special care. If two configurations use differently weighted burdens, a one-unit increase in one burden need not represent the same change in variant burden as a one-unit increase in another. Therefore, their raw regression coefficients are not automatically comparable. Report the burden definition and units, and compare estimates only with an explicit understanding of their scales. A change in p-value alone does not indicate whether an estimated association changed direction or magnitude.
Likewise, a p-value above a conventional threshold in one sensitivity analysis does not prove that the association is absent there. Differences in the number of included carriers, burden variability, or standard error can change test evidence. Report results across the set rather than reducing them to a binary label of “significant” or “not significant.”
Worked Example: Reading a Planned Set of Results
Consider six prespecified configurations for one hypothetical gene and quantitative trait. The same participants, covariates, and score-test procedure are used throughout. The table gives illustrative signed score statistics \(Z\); two-sided p-values are calculated as \(2\Phi(-|Z|)\), where \(\Phi\) is the standard normal cumulative distribution function.
| Configuration | Choice varied from primary | \(Z\) | Two-sided \(p\) |
|---|---|---|---|
| Primary | None | 2.58 | 0.0099 |
| A | Stricter frequency cutoff | 2.35 | 0.0188 |
| B | Broader frequency cutoff | 1.96 | 0.0500 |
| C | Broader functional mask | 2.05 | 0.0404 |
| D | Frequency-based weights | 1.72 | 0.0854 |
| E | Alternative annotation-based weights | 2.30 | 0.0214 |
The primary p-value follows from \(2\Phi(-|2.58|)\approx 2(0.00494)=0.0099\). For configuration D, \(2\Phi(-|1.72|)\approx 2(0.0427)=0.0854\). These are illustrative values; an actual analysis should calculate the test statistic and p-value using the chosen test’s justified null calibration.
All six \(Z\) statistics are positive, so the score direction is consistent in these hypothetical results. The strength of evidence varies: configuration D has a larger p-value than the primary analysis, and configuration B is near 0.05. The pattern may support the limited statement that the positive signal is not confined to the primary configuration, while also showing that its p-value is sensitive to some choices. It does not establish that the association is true, nor does it make the six analyses independent confirmations.
If the inferential claim is based on the smallest p-value among these six, a simple Bonferroni adjustment gives \(6(0.0099)=0.0594\). Thus the smallest nominal p-value does not meet a 0.05 family-wise threshold under this conservative adjustment for these six choices. This calculation addresses selection among the six tests; if the same search also covered many genes, that broader selection must also be considered.
Why the Strongest Result Is Not the Same as Robustness
Suppose each of \(m\) tests is conducted at level \(\alpha\), and, for a simple illustration, their null p-values are independent and uniform. The probability that all \(m\) p-values exceed \(\alpha\) is \((1-\alpha)^m\). Therefore, the probability of at least one p-value at or below \(\alpha\) is
For eight independent tests at \(\alpha=0.05\), this probability is \(1-0.95^8\approx0.3366\), rather than 0.05. Burden configurations are usually correlated because they reuse participants and overlapping variants, so the independence calculation is not an exact description of their joint behavior. It nevertheless shows why searching configurations and reporting only the best result changes the interpretation of its nominal p-value. A valid adjustment or prespecified testing strategy must match the actual scope of the search.
There is also a difference between evidence robust to choices and evidence selected after searching. Robustness is assessed by showing the planned results and noting the consistency or sensitivity of their scientific interpretation. Selection occurs when the analyst chooses the mask, cutoff, or weights because they gave the strongest result, then reports that result as if that choice had been made in advance. A sensitivity analysis cannot erase that selection; its role is to make the dependence visible and support appropriately qualified inference.
Worked Example: A Post Hoc Minimum P-Value
A hypothetical analyst tries eight configurations for one gene after seeing the data and reports only the configuration with the smallest p-value, \(p=0.004\). The analyst describes this as a prespecified primary test and compares 0.004 directly with 0.05. That reasoning ignores the search across eight configurations.
If the eight tests are treated as the relevant family, the Bonferroni-adjusted p-value is \(p_{\mathrm{adj}}=\min(8\times0.004,1)=0.032\). Equivalently, the Bonferroni per-test threshold for a family-wise level of 0.05 is \(0.05/8=0.00625\), and 0.004 is below that threshold. Under this stated adjustment, the result remains below 0.05 for this eight-test family. This does not make it a prespecified result: it is still necessary to disclose the search, and a broader search across genes or other analysis decisions may require a broader multiplicity strategy.
To evaluate robustness, the analyst should report the results for all eight configurations, identify which configuration was selected after inspection, and describe how the estimates and evidence vary. The adjusted result addresses a defined testing family; it does not turn the selected configuration into independent confirmation.
Common Mistakes and Reporting
- Calling the smallest p-value “robust.” A single strong result after a search is evidence selected by that search. Report the full planned set and distinguish the primary analysis from alternatives.
- Changing several choices at once. If the result changes, it is unclear which choice mattered. Prefer comparisons that isolate a choice, or explicitly label a combined alternative as exploratory.
- Comparing coefficients without checking their scales. Different weights or masks can define burdens with different units. State the burden definition before interpreting or comparing coefficients.
- Treating overlapping analyses as replications. Shared participants and variants make configurations dependent. Consistency across them is useful sensitivity information, not a count of independent studies.
- Reporting only whether each p-value crosses 0.05. Include estimates or signed statistics, p-values, and variant or carrier counts so readers can see the pattern and its limits.
A concise report should name the primary configuration, give the rationale for the alternatives, identify what was held fixed, and present results for every planned configuration. If a choice was made after examining results, say so. State the multiplicity procedure and the family of tests it covers when a claim depends on selecting among configurations. These details let readers judge whether the conclusion is stable to plausible analysis choices without confusing stability with post hoc optimization.
Check Your Understanding
Try these questions without looking back.
- A team changes the mask and frequency cutoff simultaneously and finds a different p-value. What can and cannot be concluded about the source of the change?
- Why might two burden regression coefficients be difficult to compare when their analyses use different variant weights?
- In a planned set of four configurations, the primary p-value is 0.02 and an alternative p-value is 0.07. Explain why the latter does not by itself prove the association is absent under that alternative.
- An analyst tries ten configurations and reports only the smallest nominal p-value. What information should be disclosed, and why is the result not equivalent to a prespecified primary test?
- Explain the difference between consistent evidence across overlapping configurations and independent replication.