What a Known Transition Can Benchmark
A velocity estimate is a prediction about the local direction of changing expression. Benchmarking asks whether that prediction agrees with an independent reference, such as longitudinal measurements, known state ordering, or a controlled perturbation with recorded timing. The benchmark does not make the reference perfect: a perturbation time is not necessarily the time at which every cell responds, and an ordered set of states may not specify the exact path taken by each cell.
The benchmark unit and reference must be stated before choosing a metric. A cell-level displacement can support a directional comparison; ordered labels can support a test of whether predicted transitions point toward later states; and repeated sampling around a known event can support an estimate of a transition landmark. These answer different questions, so a strong result on one is not automatically evidence for the others.
Choose the Reference Before the Metric
Let \(v_i\) be the estimated velocity vector for cell \(i\), expressed in a specified feature space. When an independent measurement supplies a reference displacement \(d_i\) in the same space, the angle between \(v_i\) and \(d_i\) measures directional agreement. The cosine is
This comparison requires nonzero vectors and compatible features, units, and transformations. It also requires that the reference displacement represent the same cell or a defensible matched unit over a stated interval. Cosine is insensitive to positive rescaling of either vector: it evaluates direction, not speed. A value of 1 indicates alignment, 0 indicates perpendicular directions, and -1 indicates opposite directions. Angular error, \(\theta_i=\arccos(c_i)\), ranges from 0 to \(\pi\) radians, with smaller values indicating better alignment.
When exact displacements are unavailable but cell states have a defensible order, use an order-based target rather than pretending that the labels provide a reference vector. For example, a velocity projection or graph-based transition score can be evaluated for its ability to rank cells or candidate destination states by whether they are later in the known order. This tests progression consistency, not recovery of a unique trajectory.
A perturbation benchmark has a different reference. The intervention time is observed, but it is not automatically the transition time. An independently defined molecular landmark, a pre-specified response threshold, or a known sequence of sampled states can provide an evaluable target. The time origin, sampling schedule, landmark definition, and any lag between intervention and expected response should be reported explicitly.
Directional Agreement: Definition and Justification
A cosine is a normalized inner product. By the Cauchy–Schwarz inequality, \(\lvert v_i^\mathsf{T}d_i\rvert\leq \lVert v_i\rVert\lVert d_i\rVert\), so \(c_i\) lies between -1 and 1. If the vectors point in the same direction, their normalized inner product is 1; if they are orthogonal it is 0; and if one is a negative multiple of the other it is -1. This gives a scale-free measure of orientation, but a small predicted vector can have a high cosine despite being poorly determined. If uncertainty estimates are available, use them to describe that reliability rather than treating all directions as equally certain.
Worked Example: Comparing Reference and Predicted Directions
Suppose three hypothetical cells have reference displacements \(d_1=(1,0)\), \(d_2=(0,1)\), and \(d_3=(1,1)\). Their estimated velocity vectors are \(v_1=(2,1)\), \(v_2=(-1,2)\), and \(v_3=(1,0)\), respectively. Assume both vectors for each cell are nonzero and recorded in the same two-feature space.
For cell 1, \(v_1^\mathsf{T}d_1=2\), \(\lVert v_1\rVert=\sqrt{5}\), and \(\lVert d_1\rVert=1\). Thus \(c_1=2/\sqrt{5}=0.8944\), giving \(\theta_1=\arccos(0.8944)=26.57^\circ\).
For cell 2, \(v_2^\mathsf{T}d_2=2\), \(\lVert v_2\rVert=\sqrt{5}\), and \(\lVert d_2\rVert=1\). Therefore \(c_2=2/\sqrt{5}=0.8944\), and \(\theta_2=26.57^\circ\).
For cell 3, \(v_3^\mathsf{T}d_3=1\), \(\lVert v_3\rVert=1\), and \(\lVert d_3\rVert=\sqrt{2}\). Thus \(c_3=1/\sqrt{2}=0.7071\), and \(\theta_3=45.00^\circ\).
The mean cosine is \((0.8944+0.8944+0.7071)/3=0.8320\). The mean angular error is \((26.57+26.57+45.00)/3=32.71^\circ\). These summaries indicate overall positive directional agreement in this hypothetical example, not agreement in speed or proof that each estimated vector is biologically accurate.
Recovering Ordered Transitions
Suppose independent evidence assigns cells to an ordered sequence of states, and a method gives each candidate transition a score \(s_i\) that is higher when the method predicts movement toward a later state. Define a binary benchmark outcome \(Y_i=1\) for a cell in a pre-specified transition window and \(Y_i=0\) for a comparison cell outside that window. The area under the receiver operating characteristic curve (AUC) is the probability that a randomly selected positive cell has a higher score than a randomly selected negative cell, with ties counted as one half. It measures ranking, not calibration.
If the task is to recover an edge between two ordered states, the candidate edges and the relevant later-state labels must be defined in advance. One can report the fraction of top-ranked predicted edges that are known transitions (precision), and the fraction of known transitions recovered among those selected (recall). These depend on the selection rule, such as choosing the top \(k\) edges or applying a threshold. AUC avoids choosing one threshold, but can obscure performance in highly imbalanced settings; precision and recall make the selected operating point explicit.
Worked Example: Ranking Cells in a Transition Window
In a hypothetical benchmark, four cells are independently labeled as being in a transition window and four as comparison cells. Their velocity-derived scores are:
| Group | Scores |
|---|---|
| Transition-window cells (\(Y=1\)) | 0.90, 0.80, 0.55, 0.30 |
| Comparison cells (\(Y=0\)) | 0.70, 0.40, 0.20, 0.10 |
There are \(4\times4=16\) positive-negative pairs. The score 0.90 exceeds all four comparison scores, and 0.80 also exceeds all four. The score 0.55 exceeds 0.40, 0.20, and 0.10, giving three wins. The score 0.30 exceeds 0.20 and 0.10, giving two wins. Hence the AUC is \((4+4+3+2)/16=13/16=0.8125\).
If the rule is to select the top four cells, their scores are 0.90, 0.80, 0.70, and 0.55. Three are transition-window cells, so precision is \(3/4=0.75\). There are four positive cells in total, so recall is \(3/4=0.75\). These values describe this hypothetical benchmark and this selection rule; they do not establish performance in a different cell population or at a different transition prevalence.
Recovering a Transition Time
A timing metric requires an explicit rule that converts velocity-based output into an estimated landmark. For example, a study might define a score, average it within each sampling time, and estimate when that average first crosses a threshold fixed using independent information. Given an independently specified reference time \(\tau\) and estimated time \(\hat{\tau}\), the signed timing error is \(\hat{\tau}-\tau\), and the absolute timing error is \(\lvert\hat{\tau}-\tau\rvert\). Signed error shows whether the estimate is early or late; absolute error measures the size of the discrepancy.
Worked Example: Estimating a Landmark by Interpolation
Consider a hypothetical time-course experiment with an intervention at 2 hours and samples at 0, 2, 4, 6, and 8 hours. Suppose an independently calibrated velocity-derived score has time-point means 0.12, 0.18, 0.35, 0.68, and 0.82, respectively. The pre-specified transition landmark is the first crossing of 0.50, and independent reference data place that landmark at \(\tau=5.5\) hours. The scores increase across the sampled interval, so linear interpolation between the 4- and 6-hour means is applicable under the benchmark’s stated interpolation rule.
The interpolated estimate is
The signed error is \(4.909-5.5=-0.591\) hours, and the absolute error is \(0.591\) hours. The estimate is therefore about 0.591 hours earlier than the reference landmark. It is not an estimate of the intervention time: the intervention occurred at 2 hours. The conclusion depends on the threshold, reference landmark, and interpolation rule, all of which must be set independently of the result being evaluated.
With biological replicates, summarize timing error across replicates and show its variability. Cells sampled from the same replicate are not independent experimental replicates; treating each cell as an independent replicate can make uncertainty appear much smaller than it is. A timing estimate should also reflect the sampling interval: interpolation produces a numerical estimate, but does not create temporal information absent from the experiment.
Benchmark Design, Diagnostics, and Alternatives
A defensible evaluation makes the reference and analysis choices explicit. In particular, decide which genes, features, cells, state labels, and transition windows are eligible before inspecting benchmark performance. If the same labels or features are used both to fit velocity and to define the reference, apparent agreement may partly reflect information reuse. Separate data, held-out replicates, or an independently obtained reference can reduce this leakage. When a reference displacement is noisy, document how it was estimated and whether uncertainty in that displacement is represented.
State whether the benchmark concerns vector orientation, progression toward later states, recovery of a transition edge, or a timing landmark.
Confirm that the reference was not constructed from the same velocity score in a way that guarantees agreement.
Use cosine or angular error for reference directions, ranking metrics for ordered outcomes, and timing error for a defined landmark.
Inspect results across replicates, relevant cell groups, and reasonable pre-specified analysis choices; report failures and uncertainty as well as averages.
A comparison with an alternative should preserve the benchmark and target while changing the prediction method. For directional performance, compare cosine distributions; for transition ranking, compare AUC and, where relevant, precision-recall summaries; for timing, compare signed and absolute errors. A baseline can be a simple rule that always points toward the next known state or uses a state-order score without velocity. If a complex method does not improve on a relevant baseline, its extra modeling complexity has not demonstrated benchmark benefit for that target.
Diagnostics include checking whether estimated directions are near zero, whether scores are tied or nearly constant, whether transition rankings change markedly across replicates, and whether timing estimates move substantially under small threshold changes. For ordered states, inspect each transition separately: strong average performance can hide a failure at one boundary. For time courses, check for non-monotone score patterns; a single interpolated crossing may be misleading if the score crosses the threshold multiple times.
Common limitations include incomplete cell sampling, asynchronous responses to perturbation, uncertain state order, batch effects, and changes in cell composition. A benchmark from one perturbation or lineage may not generalize to another. A global mean cosine can also conceal a mixture of well-aligned and reversed cells. Report the units and aggregation rule, retain replicate-level summaries, and avoid interpreting high agreement as causal evidence or as proof that predicted speeds are calibrated.
Common Mistake
Wrong reasoning: “The intervention happened at 2 hours, and the estimated landmark is 4.9 hours, so the velocity method is 2.9 hours late.” This treats intervention time and transition time as the same target.
Correction: Compare an estimated transition landmark with an independently defined reference transition landmark. In the worked example, the reference transition landmark is 5.5 hours, so the estimate is 0.591 hours early. The intervention time is useful context, but it is not a substitute for the transition reference.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A velocity vector is twice the length of its reference displacement and points in exactly the same direction. What is its cosine, and does that establish correct speed?
- There are 20 positive-negative cell pairs, with 15 positive cells ranked above the negative cell in each pair on average. What is the AUC if there are no ties?
- A benchmark estimates a landmark at 7 hours; the independent reference landmark is 8 hours. Give the signed and absolute timing errors.
- Why is a perturbation’s recorded time not automatically a reference transition time?
- Name one diagnostic for a transition score and one limitation of using ordered state labels as ground truth.
Solutions. (1) The cosine is 1 because the vectors are aligned. This says nothing about speed calibration; the predicted vector is twice the reference length. (2) The AUC is \(15/20=0.75\), because 15 of the 20 positive-negative comparisons are correctly ordered. (3) The signed error is \(7-8=-1\) hour, and the absolute error is 1 hour; the estimate is early by one hour. (4) Cells can respond after different delays, and intervention records when treatment began, not when a specified cellular transition occurred. (5) Check whether scores are tied or whether rankings vary across replicates. Ordered labels may not identify exact paths or transition times, even when their ordering is reliable.