Why Network Structure Needs Sensitivity Analysis
An estimated omics network is the result of analytic choices as well as observed data. Transforming counts, setting a penalty value, or filtering low-information features can each change which edges are selected. A single fitted network therefore does not show whether its key edges are robust to reasonable alternatives. Sensitivity analysis makes that dependence visible by comparing networks under a prespecified set of plausible analytic choices.
The goal is not to find the one network that looks most convincing, nor to treat different pipelines as independent datasets. It is to distinguish structure that persists across reasonable choices from structure that appears only under particular choices. This is a different question from sampling uncertainty: resampling assesses variation across hypothetical repeated samples under a chosen pipeline, while sensitivity analysis changes the pipeline itself.
Designing a Fair Comparison
Begin with a reference pipeline and identify the analytic choices whose consequences matter scientifically. Vary those choices over ranges that are plausible before looking at which network is preferred. For example, a count-data analysis might compare two defensible transformations, several penalty values in a prespecified range, and a primary feature filter against a less restrictive one. A scenario should be documented well enough that another analyst could reproduce it.
For a useful comparison, hold the data and the parts of the analysis not under investigation fixed. If the purpose is to assess transformation sensitivity, keep the sample set, feature set, model family, covariate adjustment, and edge-selection rule fixed. If the purpose is to assess filtering, record that the node set changes and compare only edges whose endpoints are eligible in both networks. When a procedure’s edge-selection calibration depends on a choice being varied, apply the appropriate calibration separately for that scenario; do not silently reuse a cutoff whose meaning has changed.
The model still needs to be appropriate in each scenario. For example, a Gaussian graphical model uses conditional dependence represented through a precision matrix and relies on an approximately suitable continuous-data model; applying it to highly skewed, zero-heavy counts without a defensible transformation can make the comparison uninformative. A different transformation may improve model fit, but it does not establish that the model is correct. Check model diagnostics and the plausibility of each pipeline before interpreting differences as sensitivity to reasonable choices. The earlier material on Goodness-of-Fit Diagnostics for Network Models gives relevant model checks.
State which network features matter, such as presence, direction, or sign of selected edges, and which analytic choices will vary.
Set plausible alternatives in advance and keep other choices fixed where possible. Avoid selecting scenarios because they produce a preferred result.
Use the same stated model and selection procedure, with any choice-dependent calibration handled appropriately. Record node sets, edge sets, edge weights, and diagnostics.
Summarize edge persistence and whole-network similarity, distinguish common-node comparisons from changes caused by filtering, and state which conclusions depend on analytic choices.
Measures of Edge and Network Persistence
Suppose a set of \(K\) scenarios has the same eligible nodes. For a candidate edge \(e\), define its scenario persistence proportion as the fraction of scenarios in which it is selected:
Here \(I(\cdot)\) equals 1 when its statement is true and 0 otherwise. The quantity \(\hat{\pi}_e\) is a descriptive summary of the specified scenarios. Unless scenarios were generated from a probability model designed for that purpose, it is not a probability that the edge is true, a p-value, or a measure of sampling uncertainty. Its value can change if the analyst adds, removes, or gives different weights to scenarios.
To compare two edge sets on the same node universe, a common summary is Jaccard similarity: the number of shared edges divided by the number of edges present in at least one network.
This measure ranges from 0 (no shared edges) to 1 (identical edge sets). Its denominator is the union because it counts every edge that appears in either network. If one edge set is empty and the other is nonempty, the similarity is 0; if both are empty, the ratio is undefined, so report that both networks selected no edges rather than assigning a value without explanation.
For weighted networks, also compare edge weights and signs, not just whether an edge passes a selection threshold. A selected edge whose estimated strength changes greatly may be less robust than its binary presence suggests. Conversely, a small change near a threshold can flip a binary edge indicator even when the underlying estimate changes little. For directed networks, compare direction separately from adjacency: an edge appearing in both networks with opposite orientations is not a directionally persistent result.
Worked Example: Sensitivity to Transformation
Suppose a hypothetical study measures the same 80 samples and the same 20 features under three defensible transformations of count profiles. It fits the same Gaussian graphical model and uses the same prespecified edge-selection rule in each case. Diagnostics are judged acceptable for all three fitted models. For a particular undirected pair, the estimated partial correlations are 0.18 after a log transformation, 0.14 after a square-root transformation, and 0.12 after a rank-based transformation. The selection rule retains an edge when the absolute estimated partial correlation is at least 0.10.
Log transformation: \(|0.18| \geq 0.10\), so the edge is selected. Square-root transformation: \(|0.14| \geq 0.10\), so it is selected. Rank-based transformation: \(|0.12| \geq 0.10\), so it is selected. The persistence proportion is therefore \( \hat{\pi}_e = 3/3 = 1.00 \).
The edge is present in every transformation scenario considered, and its estimated partial correlation is positive in all three. This supports describing it as transformation-robust within this scenario set. It does not establish that the edge is a true biological interaction or that it would persist under other models, samples, or filters.
Interpreting Penalty and Filtering Sensitivity
A penalty controls the degree of sparsity in many network estimators. A larger penalty often produces fewer selected edges, but the exact behavior depends on the estimator and selection rule. Compare a prespecified range of scientifically reasonable penalty values, and report the range rather than choosing a value because it preserves a favored edge. If a data-driven tuning procedure is part of the analysis, sensitivity scenarios can vary its defensible settings, but each fitted result should still follow the stated procedure.
Filtering creates an additional complication: an edge cannot be selected if one of its endpoints has been removed. For comparisons involving different feature sets, first report node retention. Then compare the edge sets on the shared node universe, or explicitly describe edges lost because an endpoint was filtered out. A lost edge is not necessarily evidence that the remaining network estimator changed its view of that association; the candidate edge may simply no longer exist in the filtered analysis.
Worked Example: Sensitivity to Penalty Value
Consider three hypothetical fitted networks with the same 30 features and model specification, but with three prespecified penalty values. Their selected edge sets are: \(E^{(1)} = \{AB, AC, BD, CE\}\) at the lower penalty, \(E^{(2)} = \{AB, AC, BD, CE, DE, AF\}\) at the middle penalty, and \(E^{(3)} = \{AB, BD\}\) at the higher penalty.
The edge \(AB\) appears in all three networks, so its persistence proportion is \(3/3=1.00\). The edge \(AC\) appears in the first two, so its proportion is \(2/3=0.667\). The edges \(DE\) and \(AF\) each appear only in the middle-penalty network, giving \(1/3=0.333\) for each.
For the lower- and middle-penalty networks, the shared edges are \(AB, AC, BD,\) and \(CE\), so the intersection has size 4. Their union contains those four edges plus \(DE\) and \(AF\), for size 6. Thus \(J(E^{(1)},E^{(2)})=4/6=0.667\). For the lower- and higher-penalty networks, the intersection is \(\{AB,BD\}\), of size 2, and the union is \(\{AB,AC,BD,CE\}\), of size 4. Their Jaccard similarity is \(2/4=0.500\).
The edge \(AB\) is stable across these penalty choices, while the two edges found only at the middle penalty are choice-dependent. The scenario proportions are summaries of these three chosen settings, not probabilities or evidence that the middle-penalty edges are less biologically real.
Worked Example: Sensitivity to Feature Filtering
Suppose a hypothetical six-feature analysis has features \(A\) through \(F\). Under a less restrictive filter, the selected edges are \(E^{(0)}=\{AB, AC, CE, DF\}\). A stricter detection filter removes \(E\) and \(F\); the refitted network on \(A,B,C,D\) selects \(E^{(1)}=\{AB, AD\}\).
The features eligible in both networks are \(A,B,C,D\). On this shared node set, the less-restrictive network has edges \(\{AB,AC\}\): \(CE\) and \(DF\) are omitted because each includes a feature excluded by the stricter filter. The stricter network has \(\{AB,AD\}\). Their common edge set is \(\{AB\}\), and their union is \(\{AB,AC,AD\}\). Therefore the common-node Jaccard similarity is \(1/3=0.333\).
The edge \(AB\) persists, while \(AC\) and \(AD\) differ between fits on the shared node set. Edges \(CE\) and \(DF\) cannot be called absent under the stricter filter: those pairs were ineligible. Reporting both the node removals and the common-node comparison prevents a change in the candidate edge universe from being mistaken for an ordinary edge-selection disagreement.
Diagnostics, Alternatives, and Limitations
Sensitivity analysis is informative only when scenarios are defensible and fitted networks are interpretable. Signs that the analysis may be going wrong include a scenario that violates the model’s assumptions, a transformation chosen after seeing which edges it preserves, unexplained changes to multiple pipeline components at once, or a reported persistence proportion without the scenario set that produced it. If many choices change together, an observed difference cannot be attributed to any one of them. Use one-at-a-time comparisons to isolate individual choices, then include selected joint scenarios to check whether combinations produce additional changes.
The main alternatives answer related but distinct questions. Resampling and stability selection assess how selections vary across resampled datasets under a stated analysis; they do not by themselves show sensitivity to transformations or feature filters. A single prespecified model offers a clear primary result but does not show how dependent it is on other reasonable choices. Comparing several modeling families can broaden the assessment, but model-family differences may be harder to interpret than varying one pipeline choice at a time. These approaches can complement one another, but their summaries should not be conflated.
No sensitivity analysis covers every possible transformation, penalty, filter, or model. Results depend on which scenarios are included, and an edge that persists across a narrow or implausible set may still be fragile. Sensitivity also does not remove confounding, establish causality, or substitute for sampling uncertainty and model checking. Report the choices, the full or well-summarized scenario results, the edge or weight patterns, and any conclusions that change. Use labels such as “persistent across the evaluated scenarios” or “sensitive to the filter,” rather than asserting that an edge is definitively true or false.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- An edge appears in four of five prespecified transformation scenarios. What is its scenario persistence proportion, and what does that value not mean?
- Two networks have edge sets \(\{AB, AC\}\) and \(\{AB, AD\}\) on the same nodes. Calculate their Jaccard similarity.
- A stricter filter removes feature \(C\), and an edge \(BC\) consequently cannot appear in the refitted network. What should be reported before concluding that the edge is filter-sensitive?
- Why is changing the transformation and penalty simultaneously a weak design for identifying the cause of a network change?
Solutions. 1. The proportion is \(4/5=0.800\). It describes the edge’s presence in the specified scenarios; it is not the probability that the edge is true or a measure of sampling uncertainty. 2. The intersection is \(\{AB\}\), of size 1; the union is \(\{AB,AC,AD\}\), of size 3. Thus \(J=1/3=0.333\). 3. Report that \(C\) was removed and that \(BC\) was therefore ineligible; compare networks on their shared node set to assess changes among eligible edges. 4. Because both choices changed, the observed difference cannot be attributed to either one separately. Varying one choice at a time helps isolate its effect.