Why Principal-Curve Endpoints Need Diagnosis
A fitted principal curve summarizes a continuous path through a cell representation. Its endpoints can look like natural beginnings or endings of a biological process, but a curve must also stop somewhere in the observed data. If few cells were sampled near an extreme, the apparent endpoint may reflect a coverage gap, an outlier, or a fitting choice rather than a well-supported terminal state. Conversely, a genuine rare extreme may have low cell density and should not be discarded just for being rare.
The earlier material on principal curves establishes how cells are projected onto a fitted path and how arc length gives a relative pseudotime ordering. Here the question is narrower: how much evidence supports the location and interpretation of a fitted endpoint? Projection residuals, local density, and endpoint perturbations provide complementary evidence. None alone proves that an endpoint is biologically real.
Distances should be computed in the same representation and scaling used for fitting the curve. A residual measures how far a cell lies from the fitted path, not how far it lies along the path. Thus small residuals indicate that cells are close to the curve, while large residuals can flag cells that the path does not represent well. Neither result, by itself, tells us whether the endpoint is a biological extreme.
Three Complementary Diagnostics
First, examine residuals near each endpoint and compare them with residuals in a nearby interior band. A cluster of endpoint cells with modest residuals is geometrically compatible with the fitted path. A few endpoint cells with large residuals may be pulling a flexible curve toward an unsupported location. Summaries such as the median and upper quantile are more informative than a single residual, and comparisons should use bands of similar width in pseudotime.
Second, assess local cell density. One practical proxy is the distance from each cell to its \(k\)th nearest neighbour in the chosen representation: a larger \(k\)-neighbour distance generally indicates sparser local coverage. Alternatively, count cells in a fixed-radius neighbourhood. Compare endpoint regions with nearby interior regions using the same \(k\), radius, representation, and preprocessing. These are relative diagnostics: an endpoint may be sparse compared with the interior but still contain a coherent, repeatedly observed cell population.
A simple justification clarifies what a neighbour distance measures. Suppose, locally, cells are sampled at approximately constant density \(f\) in an effective dimension \(d\). The expected number of cells within radius \(r\) is approximately \(n f V_d r^d\), where \(n\) is the number of cells and \(V_d\) is the volume of the unit ball in \(d\) dimensions. Setting this count to \(k\) gives the rough scale
So, all else equal, a larger \(k\)-neighbour radius corresponds to lower local density. This approximation is not a calibrated density test: real cell distributions can be curved, anisotropic, or concentrated near a lower-dimensional structure. It motivates comparing like with like, not interpreting a particular radius as a universal threshold.
Third, perturb the endpoint observations and refit. For example, repeatedly subsample cells from a prespecified endpoint band, retain the interior cells, refit the curve using the same fitting choices, and align each refit to the same orientation. Compare endpoint locations and the nearby curve across refits. If removing a small fraction of endpoint observations causes large changes, the estimated endpoint may depend heavily on a few cells. If the endpoint is stable, that supports geometric robustness to this particular perturbation, but does not establish biological meaning.
Worked Endpoint Assessments
Worked Example: Comparing Endpoint Residuals
Hypothetical data. Suppose a fitted curve has an early endpoint band with 12 cells and a nearby interior band with 20 cells. Residuals are measured in standardized embedding units. The endpoint residuals have median \(0.18\) and 90th percentile \(0.31\); the interior residuals have median \(0.20\) and 90th percentile \(0.34\). The single largest endpoint residual is \(0.92\).
The endpoint median is \(0.18\), compared with \(0.20\) in the interior, a difference of \(0.18-0.20=-0.02\) units. The endpoint 90th percentile is \(0.31\), compared with \(0.34\), a difference of \(-0.03\) units. Thus the typical endpoint cell is no farther from the curve than a typical nearby interior cell. The maximum residual of \(0.92\) is much larger than either 90th percentile, so one cell merits inspection; it should not replace the band summaries.
These numbers support the claim that most cells in this endpoint band fit the curve about as closely as nearby interior cells. They do not establish that the endpoint is a biological terminal state. The comparison is most useful if the bands were chosen before inspecting the residuals and the same distance scale and fitting representation were used for both.
Worked Example: Reading Local Density at an Endpoint
Hypothetical data. For each cell, compute the distance to its fifth-nearest neighbour in the fitted representation. In an early endpoint band, the median distance is \(0.84\) units; in an adjacent interior band, it is \(0.42\) units. The endpoint-to-interior ratio is
The typical endpoint cell must reach twice as far to find its fifth neighbour. This indicates sparser local coverage under this particular representation and choice of \(k=5\). Suppose further inspection shows that the 12 endpoint cells form one compact group in the embedding, rather than one cell separated from the other 11. That pattern is consistent with a sparsely sampled but coherent group; it is less consistent with an endpoint determined by a solitary outlier.
The conclusion remains limited. A twofold neighbour-distance ratio does not mean the endpoint density is exactly half: the relationship depends on effective dimension and local geometry. Nor does sparse coverage rule out a true rare biological extreme. The finding motivates checking stability and, where available, independent evidence such as sample composition or measured cell-state features.
Worked Example: Perturbing Endpoint Cells
Hypothetical data. A baseline fit has an endpoint at \(g(b)=(2.00,0.00)\) in two standardized coordinates. In 20 refits, investigators randomly retain 80% of cells in the endpoint band, keep the interior cells, and apply the same fitting settings. After aligning orientation, the median refitted endpoint is \((1.92,0.06)\). Its displacement from the baseline endpoint is
standardized units. Suppose the median distance from an endpoint-band cell to its fifth-nearest neighbour is \(0.50\) units. The endpoint displacement is \(0.100/0.50=0.200\), or 20% of that local spacing. Also suppose the median displacement across the 20 refits is \(0.11\) units and the largest is \(0.24\) units. Relative to local cell spacing, these shifts are modest in this example, so the estimated endpoint appears reasonably stable to this subsampling perturbation.
This conclusion is conditional on the perturbation design and the fitting choices. It does not show that the endpoint would be stable to a different normalization, representation, or curve-fitting method. The comparison also requires consistent orientation; otherwise a refit may reverse its parameter direction and make corresponding ends appear to disagree.
Combining Evidence and Checking Failure Modes
The diagnostics answer different questions. Residuals ask whether endpoint cells lie near the path. Neighbourhood measures ask whether the region is well covered relative to its surroundings. Perturbations ask whether the fitted endpoint depends sensitively on particular observations. Their agreement strengthens a geometric assessment, but they are not independent tests with a combined p-value. In particular, a coherent low-density endpoint with low residuals and stable refits can be a credible rare extreme; a dense endpoint with high residuals may instead indicate that a single smooth path is a poor description of the local data.
For a disciplined assessment, define endpoint bands before examining the diagnostics, compare each endpoint with a nearby interior band, and record the representation, distance metric, \(k\) or radius, subsampling fraction, and fitting settings. Examine both ends separately: their sampling and stability need not match. Inspect the cells driving a large residual or a large perturbation shift rather than relying only on aggregate summaries.
Endpoint perturbations must be designed so they answer a clear question. Removing the outermost cells and then reporting that the fitted curve no longer reaches those cells is not informative: the data defining that extreme were deliberately removed. Instead, use repeated partial subsampling within a prespecified endpoint band, retain enough endpoint observations to fit the curve, and compare the resulting endpoint locations with the baseline. If the curve is parameterized by arc length, compare locations in the shared part of the curves or compare endpoint coordinates after alignment; do not assume that identical numerical parameter values refer to identical locations after refitting.
A principal curve can also be a misleading summary when the cell structure is not well represented by one path. A high-residual endpoint, especially alongside another nearby group, may indicate a geometry problem rather than merely a sparse endpoint. Do not force an endpoint diagnosis to settle whether there are multiple paths or branches; that requires a separate assessment of the trajectory structure.
| Pattern of evidence | Reasonable geometric interpretation | What it does not establish |
|---|---|---|
| Low residuals, moderate coverage, stable perturbations | Endpoint is supported by the sampled geometry | That it is a biological terminal state |
| Low residuals, sparse coverage, stable perturbations | Coherent but sparsely sampled endpoint is plausible | That the rare state is representative or complete |
| High residuals and strong endpoint shifts | Endpoint may depend on a few poorly represented cells | Which alternative curve or structure is correct |
| High density but high residuals | The fitted path may not represent local cell geometry well | That the endpoint should simply be extended |
Common Mistake and Limitations
Wrong reasoning: “The endpoint has fewer cells than the interior, so it is an artifact and should be removed.” A rare biological state can genuinely have fewer sampled cells, and the number observed also depends on study design and capture. Correction: describe the endpoint as sparsely covered, then assess residuals, coherence, perturbation stability, and relevant independent evidence. Geometry alone cannot decide whether a low-density extreme is biologically meaningful.
Likewise, small residuals do not certify an endpoint: a flexible curve can pass close to a few observations even when coverage beyond them is absent. Stability is also relative to the perturbations tried. These procedures diagnose sensitivity and support in the observed representation; they do not quantify the probability that an endpoint is true, correct batch effects, or establish a temporal direction. Biological interpretation needs evidence beyond the curve geometry.
Set endpoint and nearby interior bands on the oriented pseudotime scale, and use the same representation and distance metric throughout.
Compare residual summaries and local-neighbour measures for each endpoint against its nearby interior.
Repeatedly subsample endpoint-band cells, refit with consistent settings, align orientation, and summarize endpoint displacements.
State whether the endpoint is geometrically supported, sparse, or sensitive, and distinguish those observations from a claim about biological terminal status.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- What does a projection residual measure, and what does it not measure?
- An endpoint's median fifth-neighbour distance is \(0.60\) units and the adjacent interior's is \(0.30\) units. Calculate the ratio and state the warranted interpretation.
- Why is a single large endpoint residual less informative than a residual distribution for the endpoint band?
- In repeated endpoint subsampling, why must the refitted curves be aligned before comparing their endpoints?
- Endpoint cells have low residuals, sparse local coverage, and stable perturbation fits. What is a careful conclusion?
Solutions. (1) It is the distance from a cell to its projected point on the curve; it is not distance along the curve or a direct measure of biological abnormality. (2) \(0.60/0.30=2.00\). The endpoint is more sparsely covered by this neighbour-distance diagnostic, not necessarily half as dense or biologically spurious. (3) One unusual cell can dominate a maximum, whereas a median and upper quantile describe how the band as a whole fits. (4) Orientation can reverse, so unaligned endpoints may represent opposite ends rather than instability. (5) The endpoint is geometrically coherent and stable under the tested perturbation but sparsely sampled; its biological interpretation remains uncertain.