Evaluating Network Recovery

What Network Recovery Measures

In a simulation study, the generating network is known, so an estimated network can be compared directly with its truth. The question is not simply whether an algorithm returns a graph. It is whether the returned edges, or the scores used to rank candidate edges, agree with the simulated edge structure. That comparison must account for which pairs were eligible to be edges and how sparse the true network was.

For a multi-omics network, node blocks may correspond to modalities or other prespecified feature groups. An edge is within-block if both endpoints belong to the same block, and cross-block if its endpoints belong to different blocks. These categories can have different scientific roles and different numbers of candidate pairs, so reporting only one overall score can hide important failures.

This definition assumes the truth and estimate use the same nodes, edge convention, and eligibility rules. For an undirected network with no self-edges, the number of possible within-block pairs for blocks of sizes \(p_1,\ldots,p_B\) is \(\sum_b {p_b\choose 2}\); the number of possible cross-block pairs is \(\sum_{b<c}p_bp_c\), if every such pair is eligible. Directed networks require a different pair universe and should not be scored as undirected graphs. If a pair cannot be estimated because a modality is unavailable, state whether it is excluded from the eligible universe; do not silently count it as a missed edge.

Threshold-Aware Edge Selection

A threshold-aware metric evaluates a selected graph at a specified decision rule. For example, an algorithm might return edges whose absolute association score exceeds a fixed cutoff, or it might select a prespecified number of edges. Within any one stratum, compare selected edges with truth as follows: true positives (TP) are true edges selected; false positives (FP) are non-edges selected; false negatives (FN) are true edges not selected; and true negatives (TN) are non-edges not selected.

Precision is the fraction of selected edges that are true, and recall (also called sensitivity) is the fraction of true edges recovered. Specificity is the fraction of true non-edges correctly left unselected. The F1 score is the harmonic mean of precision and recall:

The second expression follows by substituting the definitions and simplifying. F1 gives equal emphasis to precision and recall, but it ignores TN; it is therefore not a general summary of all aspects of classification. When no edges are selected, precision has a zero denominator, and when the truth contains no edges, recall has a zero denominator. Specify a convention for such cases rather than allowing software defaults to determine the report invisibly.

Threshold choice is part of the evaluation. Use a rule defined without access to the replicate’s true edges, such as a prespecified significance or score cutoff, or a selection budget fixed in advance. Choosing the cutoff that maximizes F1 against the known simulated truth is an oracle evaluation: it uses information unavailable when analyzing real data and will generally make performance look better. If a method naturally returns a ranking rather than a single graph, evaluate its ranking separately instead of selecting a favorable cutoff after seeing the answer.

Threshold-Free Ranking Metrics

When an algorithm assigns scores to candidate edges, threshold-free metrics assess how well the scores rank true edges above non-edges across possible cutoffs. ROC AUC is the area under the receiver operating characteristic curve, which plots recall against the false-positive rate \(1-\text{specificity}\). Equivalently, with appropriate handling of tied scores, it is the probability that a randomly chosen true edge receives a higher score than a randomly chosen non-edge, with ties contributing one-half. A value of 0.5 corresponds to chance ordering in this pairwise sense; a value of 1 indicates perfect ranking.

The precision-recall curve plots precision against recall as the score cutoff varies. Average precision (AP) summarizes this curve by averaging precision at the ranks where true edges occur. If the ranked list has \(M\) candidate edges and \(y_{(k)}=1\) indicates that the edge at rank \(k\) is true, one common non-interpolated definition is:

AP is defined when the evaluated stratum contains at least one true edge. Software packages use some different interpolation conventions, so report which definition or implementation was used. ROC AUC and AP require scores whose direction is consistent: higher values must mean stronger evidence for an edge. A signed estimate may need a prespecified transformation, such as its absolute value, if the evaluation concerns edge presence regardless of sign. If the sign is part of the target, score sign recovery separately rather than discarding it.

Sparsity Changes What a Score Means

Network edge sets are often sparse: most eligible pairs are non-edges. This makes specificity and ROC AUC potentially reassuring even when selected edges are not very trustworthy. For example, a method that labels nearly every pair as a non-edge can achieve high specificity simply because non-edges are abundant, while recovering few true edges. Precision and AP are more directly affected by prevalence, the fraction of eligible pairs that are true edges. A random ranking has expected precision near this prevalence at a given cutoff, and its precision-recall baseline is commonly shown as the prevalence; finite-sample AP can differ slightly from that value.

Consequently, scores should be reported with the generating sparsity level and the number of eligible and true edges. Results at different sparsity levels are not automatically comparable as if they described the same task. A high AP in a denser network may partly reflect a higher positive-edge prevalence; a low precision in a very sparse network may coexist with useful ranking if the method substantially exceeds the sparse baseline.

Summarizing Replicates and Diagnosing Failure

For each replicate, calculate metrics within the relevant edge strata using that replicate’s own truth and eligible universe. Then summarize the replicate-level values for each method and scenario, for example with the mean and standard deviation or the median and interquartile range. This keeps the unit of replication visible. Pooling all edge decisions across replicates instead gives more influence to replicates with larger edge universes or more positive edges, and can answer a different question. State explicitly if pooled counts are also reported.

A clear report should identify, for each scenario, the block definitions, the eligible edge counts, the true within-block and cross-block edge counts, the selection rule or ranking metric, and how replicates were summarized. Include a range of realistic sparsity levels, not just a single convenient edge density. If positive-edge counts vary substantially among replicates, report that variation; a metric may be undefined in replicates with no positives, and excluding those replicates without disclosure changes the summary.

The main alternatives complement rather than replace one another. Threshold-aware precision, recall, and F1 answer how a method performs at a particular operating point; threshold-free ROC AUC and AP assess ranking over cutoffs. ROC AUC is less sensitive to prevalence than precision-based measures, but can look strong when the number of negatives overwhelms the positives. AP focuses attention on positive-edge retrieval, but changes with prevalence and is not a prevalence-free score. No single metric establishes that a network is scientifically correct, and edge recovery evaluates agreement with the simulated truth, not causal validity or biological usefulness.

Common Pitfall

A frequent mistake is to report high specificity or ROC AUC as proof that a method recovers a sparse network well. The wrong reasoning is that correctly classifying most candidate pairs must mean that the true edges were found. The correction is to inspect the true-positive count and recall, report precision or AP alongside specificity or ROC AUC, and show the true-edge prevalence. In a sparse setting, a method can classify nearly all non-edges correctly while missing most of the edges that define the network.