Why Edge Selection Needs Calibration
A sparse network is often estimated by fitting a regularized model: a penalty encourages many edge estimates to be exactly zero, leaving a manageable set of candidate connections. But a selected edge is not automatically a statistically established edge. The selection procedure has searched across many possibilities and preferentially retained estimates that look large or stable in the observed data. Consequently, ordinary uncertainty calculations applied after selection can be too optimistic.
Calibration asks what error guarantee accompanies a reported network. Is the aim to keep the chance of even one false edge small, to limit the expected proportion of false edges among reported ones, or to control the expected number of false selections? These are different targets. They require valid edge-level evidence and a procedure that accounts for the number and dependence of the hypotheses considered.
In an undirected network with \(p\) nodes, the usual family contains \(m=p(p-1)/2\) possible edges, with one hypothesis per unordered pair. In a directed graph, the candidate family and hypotheses must instead reflect which ordered pairs are allowed. An edge in a graphical model usually represents conditional association under that model; it does not, by itself, establish a causal relationship.
Edgewise Uncertainty After Regularization
Suppose \(\theta_e\) is the model parameter defining edge \(e\), and the scientific null for that edge is \(H_0:\theta_e=0\). An edgewise confidence interval or p-value is meaningful only if its sampling procedure has the claimed coverage or null distribution for the estimator being used. For example, a Wald calculation typically assumes an approximately normal estimator with a consistently estimated standard error. A regularized estimate obtained by shrinking coefficients and setting some exactly to zero does not generally satisfy the ordinary unpenalized Wald assumptions.
The central difficulty is data reuse. If the same observations both choose edges and estimate their uncertainty, the chosen estimates tend to be unusually large in magnitude even when their true parameters are zero. Treating the selected edge as if it had been specified before seeing the data ignores that search. Standard errors and p-values computed as though the final selected model had been fixed in advance therefore need not have nominal coverage or type-I error.
There are several ways to obtain more defensible edgewise uncertainty. A prespecified confirmatory sample can estimate edges selected in a separate discovery sample. Sample splitting makes the selection independent of the confirmatory observations, although it reduces the data available at each stage. Model-specific selective-inference methods can condition on the selection event, but their validity depends on correctly representing the selection rule and the model. Debiased or desparsified estimators can support approximate inference under conditions such as suitable sparsity, design regularity, and valid variance estimation; these conditions are not automatic for every multi-omics network.
Bootstrap or subsampling results also require careful interpretation. An edge's selection frequency estimates how often the specified procedure selects it under the resampling scheme. It is a measure of selection stability, not a p-value, a probability that the edge is real, or a confidence level. A bootstrap interval for a fixed, well-defined edge estimator can describe sampling uncertainty, but an interval computed only among resamples in which that edge was selected is conditional on selection and does not generally have nominal unconditional coverage.
Worked Example: A Confirmatory Edge Estimate
Suppose a discovery sample is used to choose a standardized edge between two molecular features. In a separate validation sample, an estimator for that prespecified edge is \(0.180\), with estimated standard error \(0.070\). Assume the validation estimator is approximately normal, its standard error is valid, and the discovery and validation samples are independent. The edge parameter is measured on a standardized, unitless scale.
The Wald statistic for \(H_0:\theta_e=0\) is \(z=0.180/0.070=2.571\). The corresponding two-sided normal-approximation p-value is approximately \(0.0101\). A 95% Wald confidence interval is \(0.180\pm1.96(0.070)=0.180\pm0.1372\), or \((0.0428,0.3172)\).
Under the stated approximation, the interval procedure has 95% repeated-sampling coverage for this prespecified validation parameter. It excludes zero, and the p-value is below \(0.05\), giving evidence against the edge null at that unadjusted level. The result does not establish causation. If several edges are tested in the validation sample, the p-value also needs to be interpreted in light of the testing procedure for that family.
Controlling Errors Across Many Edges
Once valid edgewise p-values \(P_1,\ldots,P_m\) are available, multiplicity procedures address the fact that many null hypotheses are tested. They do not repair invalid p-values caused by unaccounted-for selection. A defensible analysis therefore specifies the family and obtains valid p-values first, then applies an error-control procedure suited to the scientific objective.
The Bonferroni procedure tests each hypothesis at level \(\alpha/m\), equivalently rejecting when \(P_i\leq\alpha/m\). Its FWER control follows directly from the union bound. If \(\mathcal{H}_0\) is the set of true nulls, then
The argument requires each true-null p-value to be valid, meaning \(P(P_i\leq t)\leq t\) for \(0\leq t\leq1\); it does not require independence. Bonferroni can be conservative, especially for a large family, but it gives a simple bound on the chance of any false edge.
The Benjamini–Hochberg (BH) procedure instead targets FDR. Sort the p-values as \(P_{(1)}\leq\cdots\leq P_{(m)}\), and find the largest rank \(k\) for which \(P_{(k)}\leq kq/m\), where \(q\) is the target FDR level. Reject the hypotheses with ranks \(1,\ldots,k\). Under independent p-values, or common positive-dependence conditions such as positive regression dependence on a subset (PRDS), BH controls FDR at no more than \(q m_0/m\), and hence no more than \(q\), where \(m_0\) is the number of true nulls. Network edge tests are often dependent, so the independence or PRDS condition should not be presumed without justification.
For arbitrary dependence among valid p-values, the Benjamini–Yekutieli adjustment replaces \(q\) by \(q/c_m\), where \(c_m=\sum_{j=1}^m1/j\). This retains FDR control under arbitrary dependence but is more conservative. Thus BH, Benjamini–Yekutieli, and Bonferroni offer different trade-offs: FDR versus FWER, and power versus robustness to dependence.
Worked Example: Applying Benjamini–Hochberg to Edge Tests
Assume an independent validation sample provides valid two-sided p-values for six prespecified edge hypotheses. The sorted values are \(0.001, 0.008, 0.018, 0.031, 0.120,\) and \(0.400\). Apply BH at \(q=0.05\). The conditions for the usual BH guarantee are assumed to hold for these tests.
Here \(m=6\), so the ordered thresholds \(kq/m\) are \(0.00833, 0.0167, 0.0250, 0.0333, 0.0417,\) and \(0.0500\). Comparing in rank order, \(0.001\leq0.00833\), \(0.008\leq0.0167\), \(0.018\leq0.0250\), and \(0.031\leq0.0333\). The fifth comparison fails because \(0.120>0.0417\), as does the sixth.
The largest passing rank is \(k=4\), so BH rejects the first four hypotheses. The conclusion is that these four edges are discoveries under a procedure targeting FDR \(q=0.05\), subject to the stated validity and dependence assumptions. It is not a claim that each rejected edge has a 5% chance of being false, nor that exactly 5% of these four edges are false.
Selection-Aware Designs and Stability Guarantees
When a regularized discovery step is essential, a clean option is to separate discovery from confirmation. Use one data portion to select candidate edges, then test only those candidates in independent data. Conditional on the discovery data, the selected set is fixed; valid validation-sample p-values can then be adjusted over that set. This does not remove the need to define the tested family or account for multiplicity. It also does not make a poorly specified validation model valid.
Another option is stability selection, which repeats a base selection procedure on subsamples and retains variables or edges selected frequently. A standard stability-selection result bounds the expected number of falsely selected edges (PFER) under assumptions that include exchangeability of null candidates, a base procedure with expected number selected at most \(q_{\mathrm{sel}}\), and a selection-frequency threshold \(\pi_{\mathrm{thr}}>1/2\). In the usual bound, with \(m\) candidate edges,
where \(V\) is the number of selected null edges. This is an expected-count bound, not an FDR guarantee or an edgewise confidence statement. The exchangeability assumption can be implausible when null edges differ substantially in variability or network position, so the bound should not be advertised without discussing its assumptions. Stability selection is useful for assessing reproducibility of selection, but it does not replace confirmatory inference when the goal is calibrated edge-specific evidence.
Worked Example: Interpreting a Stability-Selection Bound
Consider a hypothetical network with \(m=45\) candidate edges. A subsampling analysis uses a base selector whose expected number of selected edges is at most \(q_{\mathrm{sel}}=6\), and retains edges selected at least \(\pi_{\mathrm{thr}}=0.80\) of the time. Assume the stability-selection conditions, including null-edge exchangeability, are reasonable.
Substitution into the bound gives \(E[V]\leq 6^2/\{(2(0.80)-1)45\}=36/(0.60\times45)=36/27=1.333\). Thus the result bounds the expected number of falsely selected edges by about \(1.33\) under the assumptions. It does not say there will be at most one false edge in this particular network, and it does not imply that the false-discovery proportion is at most 1.33 or 5%.
Diagnostics, Comparisons, and Common Pitfalls
Calibration should be checked as part of the network analysis, not inferred from the apparent sharpness or sparsity of the final graph. Useful checks include verifying that the null p-value procedure matches the estimator and data design; examining null calibration in simulations that reproduce the dependence structure and selection pipeline; and checking whether results change materially under a reasonable alternative testing family or multiplicity method. A null p-value histogram should be approximately uniform when hypotheses are true under a correctly calibrated procedure, although a mixture of true and false nulls can produce excess small values.
A large gap between discovery and validation estimates, or substantially weaker validation evidence, can indicate winner's curse, model misspecification, or distribution shift. Such a gap is not proof of a specific failure, but it warrants investigation. Similarly, bootstrap frequencies that vary sharply with subsample size, tuning choices, or preprocessing indicate selection instability. A network with many selected edges and no reported family definition, uncertainty method, or error target is difficult to interpret statistically.
| Approach | Main quantity controlled or estimated | Key limitation |
|---|---|---|
| Bonferroni | FWER over a specified family | Can be conservative as the number of edges grows |
| BH | FDR under independence or suitable positive dependence | Dependence assumptions may not fit edge tests |
| Benjamini–Yekutieli | FDR under arbitrary dependence with valid p-values | Often more conservative than BH |
| Independent validation | Edgewise inference after separate discovery, with adjustment as needed | Requires independent data and can lose power |
| Stability selection | Selection reproducibility; under conditions, a PFER bound | Not an edgewise p-value or general FDR guarantee |
- Pitfall: treating a selected coefficient as prespecified. A naive Wald p-value from the same regularized fit is not automatically valid because selection favors large observed estimates. Use a valid selection-aware method, independent validation, or another justified inferential design.
- Pitfall: treating a selection frequency as a probability of truth. A frequency of 0.80 means the algorithm selected that edge in 80% of the resamples under the specified procedure; it does not mean there is an 80% probability the edge exists.
- Pitfall: reporting “5% false edges” after BH. FDR control concerns an expectation across repeated applications of the procedure, under its assumptions. It does not guarantee the realized fraction of false discoveries in one network.
Check Your Understanding
Work through each question, then compare your reasoning with the solutions.
- For 20 valid edge tests, what per-test threshold does Bonferroni use to control FWER at \(\alpha=0.05\)?
- Under the usual BH assumptions, what does an FDR target of \(q=0.05\) mean, and what does it not mean?
- A regularized edge is selected in 90 of 100 bootstrap subsamples. What does this frequency describe?
- Why can a naive p-value computed from a selected, regularized fit on the same data be anti-conservative?
- Give one reason to choose Benjamini–Yekutieli rather than BH for network edge tests.
Solutions. (1) The threshold is \(\alpha/m=0.05/20=0.00250\); reject an edge null only when its valid p-value is at most this value. (2) FDR control at \(0.05\) bounds the expected false-discovery proportion by \(0.05\) under the procedure's assumptions. It does not say that each edge has a 5% chance of being false or that the realized network must contain at most 5% false edges. (3) It estimates the selection frequency under that resampling and selection procedure; it is not a probability that the edge is true. (4) The same observations selected the edge and estimated its coefficient, so selected estimates tend to be unusually large; ordinary fixed-model standard errors ignore this search and can understate uncertainty. (5) Benjamini–Yekutieli provides FDR control under arbitrary dependence among valid p-values, whereas ordinary BH requires independence or a suitable positive-dependence condition.