Why Compare Feature Rankings?
Two cohorts can agree on which molecular features appear most prominent even when their estimates are not identical. Comparing rankings gives a compact view of this agreement: it asks whether features near the top in one cohort also tend to be near the top in another. This can be useful when the scientific goal is to prioritize features for follow-up rather than to compare every effect estimate individually.
A ranking is not self-defining. It depends on the feature set being ranked and on the quantity used to order features. A list sorted by signed effect estimates answers a different question from one sorted by absolute effects or by evidence against a null hypothesis. The ranking rule should therefore be specified before calculating concordance. In particular, rankings by \(p\)-value describe ordering of statistical evidence, not ordering of biological effect sizes.
The comparison assumes that the features represent the same entities in both cohorts, the ranking rule is applied consistently, and both rankings refer to the same feature universe. If features are not measured or mapped comparably, apparent rank differences can reflect measurement coverage rather than replication. Cross-platform measurement comparability and cohort-specific analysis choices should be considered before interpreting any concordance statistic.
Rank Correlations
Spearman’s rank correlation is the ordinary correlation between the two vectors of ranks. It ranges from \(-1\) to \(1\): a value near 1 indicates similar ordering, a value near \(-1\) indicates largely reversed ordering, and a value near 0 indicates little overall monotone ordering. It measures ordering, not equality of effect estimates.
When there are no ties, Spearman’s coefficient can also be calculated from rank differences \(d_i=R_{i1}-R_{i2}\):
If ranks are tied, assign tied features their average rank and calculate the correlation of the resulting rank vectors. The shortcut formula based only on squared rank differences is not generally valid with ties.
Kendall’s tau describes pairwise ordering. A pair of features is concordant if both cohorts place the same feature higher, and discordant if they place different features higher. With no ties, Kendall’s \(\tau\) is the number of concordant pairs minus the number of discordant pairs, divided by the total number of pairs, \(m(m-1)/2\). With ties, Kendall’s \(\tau_b\) adjusts the denominator for pairs tied in either ranking. Both measures are rank-based, but their numerical values need not be identical; report which one is used.
Worked Example: Calculating Spearman Concordance
Suppose six hypothetical features have the following ranks, with 1 denoting the highest rank in each cohort.
| Feature | Cohort 1 rank | Cohort 2 rank | Difference \(d_i\) | \(d_i^2\) |
|---|---|---|---|---|
| F1 | 1 | 1 | 0 | 0 |
| F2 | 2 | 3 | -1 | 1 |
| F3 | 3 | 2 | 1 | 1 |
| F4 | 4 | 5 | -1 | 1 |
| F5 | 5 | 4 | 1 | 1 |
| F6 | 6 | 6 | 0 | 0 |
There are no ties, and both cohorts rank the same six features. Thus the no-ties formula applies, with \(m=6\) and \(\sum d_i^2=4\):
The rankings show fairly strong positive concordance: most features occupy similar positions, with two pairs switching adjacent ranks. This does not imply that the cohorts have equal effect estimates, or that the high-ranked features have been confirmed as causal.
Top-Feature Overlap
A rank correlation uses the full ordering, whereas a top-feature overlap measure focuses on a selected leading subset. Let \(A_k\) and \(B_k\) be the sets of the \(k\) highest-ranked features in cohorts 1 and 2. Their overlap count is \(|A_k\cap B_k|\). The overlap fraction relative to one list is \(|A_k\cap B_k|/k\), and the Jaccard index is the intersection size divided by the size of the union:
These summaries answer slightly different questions. The overlap fraction asks what proportion of either top-\(k\) list is shared; the Jaccard index penalizes both nonshared parts of the union. State \(k\) and the ranking rule, since changing either can change the result. Top-list overlap is descriptive unless paired with a valid inferential procedure that accounts for how the lists were selected and for dependence among molecular features.
Worked Example: Measuring Top-Three Overlap
In a hypothetical analysis, cohort 1’s top three features are \(\{F1,F2,F3\}\), and cohort 2’s are \(\{F2,F3,F5\}\). Their shared features are \(F2\) and \(F3\), so the overlap count is 2.
The overlap fraction is \(2/3=0.6667\), meaning two-thirds of either three-feature list is shared. The union is \(\{F1,F2,F3,F5\}\), which has four features. Therefore:
Thus the lists share two of their three entries, while the Jaccard index is 0.5 because four distinct features appear across the two lists. Neither summary shows how similarly the features are ordered within the top three, and neither alone establishes that the overlap is more than would occur by chance.
Precision and Rank Stability
A feature’s observed rank is calculated from estimated quantities, which vary across samples. When two estimated effects are close relative to their standard errors, small sampling fluctuations can reverse their order. Conversely, a high rank correlation may be driven by a few stable extremes even if the ordering of many mid-ranked features is uncertain. Precision is therefore essential context for interpreting concordance.
For each cohort, report or inspect the estimates and their standard errors or confidence intervals for features that drive the comparison. A confidence interval quantifies uncertainty about an effect under its model; it is not a confidence interval for the feature’s rank. Overlapping effect intervals do not, by themselves, calculate the probability that one feature ranks above another. Rank uncertainty can be examined more directly by resampling individuals within each cohort and recomputing estimates and rankings, while respecting the study design. Such resampling describes stability under the sampling scheme; it does not repair biased measurements or cohort differences.
Worked Example: Similar Rankings With Unequal Precision
Consider four hypothetical features ranked by absolute effect estimate in two cohorts. The estimates are in arbitrary effect units, and their standard errors are in the same units.
| Feature | Cohort 1 estimate | Cohort 1 SE | Cohort 2 estimate | Cohort 2 SE |
|---|---|---|---|---|
| F1 | 0.80 | 0.30 | 0.60 | 0.08 |
| F2 | 0.70 | 0.10 | 0.65 | 0.25 |
| F3 | 0.40 | 0.08 | 0.35 | 0.07 |
| F4 | 0.20 | 0.05 | 0.18 | 0.06 |
The absolute estimates give ranks \(1,2,3,4\) for F1 through F4 in cohort 1 and \(2,1,3,4\) in cohort 2. The squared rank differences sum to \(1+1=2\). With \(m=4\):
The point rankings are concordant overall, but the ordering of F1 and F2 reverses. To see why precision matters, use a normal-approximation 95% interval for F1 in cohort 1: \(0.80\pm1.96(0.30)=[0.212,1.388]\) effect units. For F2 in that cohort, the corresponding interval is \(0.70\pm1.96(0.10)=[0.504,0.896]\). F1’s estimate is substantially less precise there. In cohort 2, F2’s interval is \(0.65\pm1.96(0.25)=[0.160,1.140]\), while F1’s is \(0.60\pm1.96(0.08)=[0.443,0.757]\). These intervals make the reversal plausible in the presence of estimation uncertainty, but interval overlap is not a formal probability statement about their rank order. The rank correlation describes the observed point estimates, not the stability of the ranking across repeated samples.
Diagnostics, Alternatives, and Limitations
Before interpreting a concordance summary, check that the same feature identities, feature universe, direction convention, and ranking rule were used in both cohorts. Inspect a rank-versus-rank scatterplot or a table of the highest-ranked features. A single coefficient can hide a pattern in which most features agree but a few high-impact reversals dominate, or one in which only the extremes agree. Check for ties, particularly when ranks are based on rounded values or discretized scores, and use a tie-aware calculation where needed.
- Effect-size correlation preserves numerical differences and units, but can be dominated by scale differences or outliers. Rank correlation focuses on ordering and is less sensitive to magnitudes.
- Top-\(k\) overlap is easy to interpret for a prioritization task, but discards ordering within each list and depends on the chosen cutoff.
- Spearman correlation summarizes monotone agreement across the full list, while Kendall’s tau has a direct pairwise-ordering interpretation. Neither reveals which features disagree without inspecting the rankings.
A common mistake is to interpret a high rank correlation as evidence that effect sizes are replicated. For example, estimates of \(0.2\) and \(2.0\) can have the same rank relative to other features while differing greatly in magnitude. The correction is to describe rank agreement as ordering agreement and assess effect directions, magnitudes, uncertainty, and between-cohort differences separately. A second pitfall is to call a feature “validated” because it appears in both top lists. Shared selection criteria, noisy estimates, and correlated molecular features can all produce overlap; replication claims require context beyond list membership.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- Two cohorts rank the same five features without ties. Their rank differences have squared sum 4. Calculate Spearman’s rank correlation.
- Two top-four lists share three features. Calculate the overlap fraction and Jaccard index.
- Why can two cohorts have a high Spearman correlation but materially different effect estimates?
- What does a tie-aware rank calculation address, and what does it not address?
Solutions. (1) With \(m=5\), \(\rho_s=1-6(4)/[5(25-1)]=1-24/120=0.8000\). (2) The overlap fraction is \(3/4=0.7500\). The union has \(4+4-3=5\) features, so the Jaccard index is \(3/5=0.6000\). (3) Rank correlation depends on order, not the distances between estimates or their units; proportional or otherwise large magnitude differences can preserve the same ordering. (4) Assigning average ranks and correlating them handles tied values in the ranking. It does not account for sampling uncertainty, cohort bias, or whether the underlying measurements are comparable.