Why Sensitivity Analysis Matters
In a linear model for expression data, empirical Bayes (EB) moderation uses information across genes to estimate or stabilize gene-wise variances. That can improve inference when each gene has few residual degrees of freedom, but the resulting conclusions depend in part on how the across-gene variance distribution is represented. A constant prior variance and a variance trend across average expression can produce different moderated standard errors. Robust prior estimation can also alter how strongly unusual genes are moderated.
Sensitivity analysis asks whether the gene-level conclusions of interest persist under reasonable, explicitly stated alternatives for these settings. It is not a search for whichever fit produces the most discoveries. The design, contrast, gene set, and gene-wise model should be held fixed while the variance-prior choices are compared. The introductory material on empirical Bayes assumptions establishes why the across-gene model matters; here the focus is on measuring the consequences of plausible choices.
What Changes, and What Should Stay Fixed?
For gene \(g\), suppose the fitted linear model is \(y_g=X\beta_g+\varepsilon_g\), where \(X\) is the design matrix, \(\beta_g\) is the vector of gene-specific coefficients, and \(\varepsilon_g\) is the error vector. The usual model assumes \(E(\varepsilon_g)=0\) and \(\operatorname{Var}(\varepsilon_g)=\sigma_g^2 I\), with independent errors within gene and an appropriate residual distribution for the fitted inference. The design and contrast must encode the intended comparison. Sensitivity to a variance prior does not correct a misspecified design, an incorrect contrast, or confounding.
Let \(\hat\beta_g\) be the estimated coefficient vector, \(s_g^2\) its residual variance estimate, and \(\nu_g\) its residual degrees of freedom. For a contrast \(c\), the estimated effect is \(\hat\theta_g=c^\mathsf{T}\hat\beta_g\). Its ordinary estimated variance has the form \(v_gs_g^2\), where \(v_g\) is the design-dependent contrast variance factor. When only the EB variance settings change, the fitted design and contrast are held fixed; consequently, the coefficient estimate \(\hat\theta_g\) and \(v_g\) ordinarily stay fixed, while the moderated variance, test statistic, and p-value may change.
In a conjugate EB model, a gene's variance estimate is combined with a prior variance \(s_{0g}^2\) and prior degrees of freedom \(d_{0g}\). A constant-prior analysis uses a common prior variance across genes; a trend analysis lets the prior variance depend on average expression \(A_g\). Robust estimation can reduce the prior degrees of freedom applying to genes whose variances are unusually inconsistent with the fitted across-gene distribution.
Under the usual moderated-test reference model, \(t_g\) is compared with a \(t\) distribution whose degrees of freedom reflect both the residual and prior information; in the simple conjugate case these are \(\nu_g+d_{0g}\). Robust implementations can use gene-specific effective prior degrees of freedom. A comparison must use the degrees of freedom and p-values actually reported for each fit, not assume they are identical when they are not.
Why the Settings Can Change a Result
The posterior variance is a weighted average of the gene's observed variance and the prior variance. Rearranging its formula gives a useful exact comparison:
Thus, when a gene's observed variance exceeds its prior variance, moderation pulls its variance estimate downward; when its observed variance is below the prior, moderation pulls it upward. The amount of pull depends on the relative information in the prior and the gene. With larger \(\nu_g\), the weight \(d_{0g}/(d_{0g}+\nu_g)\) is smaller, so the prior generally has less influence.
A trend prior can change \(s_{0g}^2\) for genes at different expression levels. Robust estimation can change \(d_{0g}\) for genes that are outliers relative to the prior model, reducing the shrinkage they receive. These changes affect the uncertainty attached to an effect, not ordinarily the effect estimate itself. A gene can therefore keep the same estimated direction and magnitude while crossing a testing threshold in one setting but not another.
Worked Example: Comparing a Constant Prior with a Trend Prior
Suppose a hypothetical gene has an estimated contrast of \(0.50\) log2-expression units. Two plausible EB fits, using the same design and contrast, report moderated variance estimates of \(0.0235\) under a constant prior and \(0.0295\) under a trend prior. For this illustration, both fits report 10 degrees of freedom. The variance factor is \(v_g=1\).
Under the constant-prior fit, the moderated standard error and statistic are
The two-sided p-value is \(2P(T_{10}\geq 3.261)\), approximately 0.00855. Under the trend-prior fit,
The two-sided p-value is \(2P(T_{10}\geq 2.911)\), approximately 0.01554. The trend-prior standard error is larger, so the test statistic is smaller and the p-value is larger. Both unadjusted p-values are below 0.05, but their exact values differ. For a genome-wide conclusion, these raw p-values alone are not enough: multiplicity adjustment and the resulting gene-level decision must also be compared across fits.
A Practical Comparison
A useful comparison treats the prior choices as controlled alternatives rather than as a sequence of opportunities to optimize the result. Possible fits include a constant prior with ordinary estimation, a trend prior with ordinary estimation, and robust versions of either. The alternatives should be defensible from the data and model diagnostics, and the contrast and tested gene family should remain the same.
Decide whether stability concerns effect direction, approximate effect size, rank, moderated test evidence, or adjusted-threshold membership. These are different claims.
Use diagnostics and the scientific context to motivate constant or trend variance modeling and robust or ordinary prior estimation. Do not choose settings based on which yields the desired gene list.
Keep the gene-wise model, contrast, filtering, and tested family fixed. Record the effect estimate, moderated standard error, degrees of freedom, p-value, and adjusted p-value for each gene.
Compare signs and magnitudes as well as testing calls. Identify genes whose conclusions change, and describe which settings account for those changes.
Worked Example: A Hypervariable Gene Under Robust Moderation
Consider a hypothetical gene with \(s_g^2=0.16\), \(\nu_g=4\), contrast estimate \(0.60\) log2-expression units, and \(v_g=1\). Suppose the ordinary fit uses prior variance \(s_{0g}^2=0.04\) and \(d_{0g}=8\). Its moderated variance is
The standard error is \(\sqrt{0.080}=0.2828\), and the moderated statistic is \(0.60/0.2828=2.121\), with 12 degrees of freedom in this simple conjugate illustration.
Now suppose robust estimation identifies this variance as unusually high relative to the fitted prior and assigns the gene an effective prior degrees of freedom of 2, while retaining the illustrative prior variance \(0.04\). Then
The corresponding statistic is \(0.60/0.3464=1.732\), with 6 degrees of freedom in this illustration. The gene's effect remains \(0.60\) log2-expression units in both fits; the robust fit gives less prior weight to its unusually large variance and therefore more uncertainty. Both statistics are below the two-sided 0.05 critical values for their stated degrees of freedom, so this example's unadjusted threshold decision is unchanged. The example illustrates a possible variance effect; actual robust fits may also estimate other prior quantities differently.
Summarizing Stable and Setting-Sensitive Findings
Stability has more than one dimension. A gene may have the same effect direction and similar magnitude under every setting but fail an adjusted significance threshold in one fit. Another gene may remain below the chosen adjusted threshold in every fit even though its p-values vary appreciably. State which kind of stability is being assessed. For gene discovery, a transparent summary can distinguish genes passing the prespecified adjusted threshold in every fit, genes passing in some but not all fits, and genes passing in none.
Worked Example: Finding the Conclusions Shared Across Four Fits
Suppose four plausible fits produce the following adjusted p-values for five hypothetical genes. The same gene family is tested in each fit, and the decision rule is adjusted p-value below 0.05.
| Gene | Constant, ordinary | Trend, ordinary | Constant, robust | Trend, robust |
|---|---|---|---|---|
| A | 0.001 | 0.003 | 0.002 | 0.004 |
| B | 0.032 | 0.041 | 0.028 | 0.044 |
| C | 0.018 | 0.061 | 0.037 | 0.072 |
| D | 0.080 | 0.047 | 0.110 | 0.060 |
| E | 0.200 | 0.240 | 0.190 | 0.260 |
Gene A and gene B are below 0.05 in all four columns, so the adjusted-threshold conclusion is stable for those two genes across these settings. Gene C passes in two fits but not the other two; gene D passes only under the trend, ordinary fit. They are setting-sensitive under this decision rule. Gene E does not pass in any fit. The union of genes passing in at least one fit is \(\{A,B,C,D\}\), while the intersection passing in every fit is \(\{A,B\}\). These set summaries do not replace examining effect sizes or the reasons the fits differ.
Diagnostics, Alternatives, and Limitations
Before interpreting sensitivity results, check that the alternatives are credible and that each fit behaved as intended. Inspect whether a fitted variance trend plausibly follows the observed relationship with average expression, and whether it is being driven by a small number of influential genes. For robust fitting, inspect whether the genes receiving reduced prior weight are genuinely unusual under the fitted variance model. Check for failed or boundary hyperparameter estimates, unexpected changes in the tested gene family, and discrepancies in degrees of freedom or missing results. A setting that fails diagnostics should not be treated as an equally reasonable sensitivity alternative.
The principal comparison is between a constant prior and an expression-dependent trend, and between ordinary and robust prior estimation. The constant prior is simpler and may be adequate when variance does not meaningfully depend on average expression. A trend can represent systematic dependence, but an overly flexible or poorly estimated trend can add instability. Robust estimation can limit the influence of exceptional variances, but it does not repair a wrong design, poor normalization, unmodeled dependence, or broad misspecification. These choices are complementary: a trend concerns systematic dependence on expression, whereas robustness concerns observations that depart from the prior model.
Multiplicity adjustment should be applied consistently to the same tested family in each setting. Comparing a raw p-value in one fit with an adjusted p-value in another is not a valid stability assessment. Likewise, if filtering or the tested family changes between settings, adjusted values may no longer answer the same question. A sensitivity analysis also cannot guarantee that all plausible settings have been considered or establish that a stable association is causal. Stability is evidence that a conclusion is not driven by the particular variance-prior choices examined; it is not proof of model correctness.
Common Mistakes
- Choosing the fit with the most discoveries. This selects a setting based on its outcome. Instead, justify a small set of plausible alternatives using diagnostics and compare them all.
- Calling a gene unstable because its p-value changes. A numerical change alone may not change the decision or interpretation. Compare effect direction, magnitude, uncertainty, and adjusted-threshold membership separately.
- Assuming robust means universally reliable. Robust prior estimation addresses influential variance outliers under its model; it does not protect against every source of bias or dependence.
- Attributing a changed effect estimate to variance moderation. With the gene-wise model and contrast held fixed, variance-prior changes ordinarily affect uncertainty and tests, not the fitted coefficient. If the effect changes, check whether some other part of the analysis changed.
- Interpreting a stable call as certainty. Agreement across selected fits says only that the conclusion persisted under those settings and the chosen rule.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- If a variance-prior change leaves a gene's contrast estimate fixed but increases its moderated standard error, what happens to the absolute test statistic, all else equal?
- Why should filtering and the tested gene family be held fixed when comparing adjusted p-values across EB settings?
- A gene passes an adjusted threshold under two of four plausible fits. Is its threshold conclusion stable across all examined settings?
- What does robust prior estimation seek to address, and what important problems does it not automatically fix?
Solutions. (1) Since the statistic is the estimated contrast divided by its standard error, increasing the standard error reduces the absolute statistic when the estimate is unchanged. (2) Adjusted p-values depend on the tested family; changing that family makes the comparisons answer different multiplicity questions. (3) No. The conclusion is setting-sensitive under that threshold because it does not persist across all four fits. (4) Robust estimation reduces the influence of genes whose variances are unusual relative to the prior model. It does not automatically correct a misspecified design, confounding, normalization problems, or unmodeled dependence.