Why Two Integration Goals Must Be Read Together
An integration method is often assessed along at least two dimensions: how well it reduces batch-associated differences and how well it preserves biological structure. The earlier material on metrics for batch mixing and biological conservation introduced measures for these goals. Here the central issue is how to interpret them together. A method can improve mixing while also erasing real cell-state differences; another can preserve those differences while leaving more batch structure. A single number cannot express this trade-off without making an additional choice about the relative importance of the goals.
The scores used for comparison must first have consistent directions. In this tutorial, \(B\) denotes a batch-removal score and \(P\) a biology-preservation score, with larger values favorable for both. These symbols can represent transformed benchmark metrics, not necessarily the raw metrics originally calculated. A score of \(0.8\) is not automatically “80% successful”; its meaning depends on how the metric was defined and scaled.
Orienting and Scaling the Scores
A raw metric may increase when performance worsens. For example, a batch-separation measure may be smaller when batches are less distinguishable, whereas a mixing measure may be larger when batches are more evenly represented. Before combining or visually comparing such metrics, state which direction is favorable and transform them if needed. One simple reversal is \(S=1-R\) when \(R\) is already a bounded score on \([0,1]\) for which smaller is better. This reversal is valid only if the original scale and endpoints support that interpretation.
If the two measures have different units or ranges, put them on a common scale using meaningful reference points. For a metric \(X\) with a pre-specified lower reference \(L\) and upper reference \(U\), a linear transformation is \(S=(X-L)/(U-L)\) when larger \(X\) is favorable, and \(S=(U-X)/(U-L)\) when smaller \(X\) is favorable. This maps the reference values to 0 and 1. It does not guarantee that values outside the reference range fall between 0 and 1, nor does it make different metrics scientifically interchangeable.
The anchors should be chosen before inspecting which method wins—for example, using a defined null benchmark and a defined ideal benchmark when those are defensible. Scaling each metric by the minimum and maximum among only the methods under comparison can make the result depend on which methods happen to be included. Adding a poor method can shift everyone else’s scaled values. A common scale helps arithmetic; it does not remove the need to justify what the scores measure.
Worked Example: Orienting a Preservation Metric
Suppose a hypothetical benchmark reports a normalized biological distortion score \(D\), where smaller values mean less distortion. The score is bounded between 0 and 1, with 0 representing no measured distortion and 1 representing the defined worst reference. The benchmark compares two methods: Method A has \(D=0.18\), and Method B has \(D=0.31\). Convert the values to a preservation score \(P\) for which larger is better.
Because the scale is bounded and the endpoints have the stated interpretation, use \(P=1-D\). For Method A, \(P_A=1-0.18=0.82\). For Method B, \(P_B=1-0.31=0.69\). On this defined scale, Method A has the better preservation score by \(0.82-0.69=0.13\) score units.
This conclusion concerns this preservation benchmark only. It does not say which method removes more batch structure, and \(P_A=0.82\) should not be described as preserving “82% of biology” unless the metric definition specifically licenses that interpretation.
Composite Scores Encode Priorities
A common way to summarize two favorable scores is a weighted arithmetic mean:
Here \(w\) is the weight assigned to batch removal, and \(1-w\) is the weight assigned to biological preservation. The formula assumes that the scores are on scales where arithmetic differences are meaningful enough for this purpose, that the chosen directions are correct, and that the weights express a defensible priority. Equal weights, \(w=0.5\), are not a neutral scientific fact: they are one particular value judgment and can be misleading if the scales are not comparable.
A weighted mean is compensatory. A loss in one score can be offset by a gain in the other. That may suit some benchmark questions, but it is not appropriate when failing a minimum standard on either dimension makes a method unacceptable. Nor does the composite score show the size or direction of each component’s contribution unless the component scores are reported alongside it.
Worked Example: A Ranking Changes With the Weight
Consider three hypothetical methods with scores on comparable scales, where larger is better:
| Method | Batch removal \(B\) | Biology preservation \(P\) |
|---|---|---|
| A | 0.90 | 0.60 |
| B | 0.75 | 0.82 |
| C | 0.68 | 0.92 |
With equal weights, \(C(0.5)=(B+P)/2\). Method A scores \((0.90+0.60)/2=0.750\); Method B scores \((0.75+0.82)/2=0.785\); and Method C scores \((0.68+0.92)/2=0.800\). The equal-weight ranking is C, then B, then A.
Now give batch removal weight \(w=0.70\), so preservation receives weight \(0.30\). The scores are \(C_A=0.70(0.90)+0.30(0.60)=0.810\), \(C_B=0.70(0.75)+0.30(0.82)=0.771\), and \(C_C=0.70(0.68)+0.30(0.92)=0.752\). The ranking reverses: A, then B, then C. The data did not change; the priority encoded in the composite changed.
The exact A-versus-C switch point follows by setting their weighted scores equal: \(0.90w+0.60(1-w)=0.68w+0.92(1-w)\). Simplifying gives \(0.60+0.30w=0.92-0.24w\), so \(0.54w=0.32\) and \(w=0.5926\). For weights above approximately 0.593, A outranks C; below that value, C outranks A. Reporting the weight and the component scores makes this dependence visible.
Dominance and Trade-Offs
A comparison can be informative without collapsing both goals to one number. Method \(i\) Pareto-dominates Method \(j\) if \(i\) is at least as good on both scores and strictly better on at least one. If neither method dominates the other, each has an advantage on at least one dimension, so the pair presents a genuine trade-off under these scores. Methods not dominated by any other method form the Pareto set. This is a description of the observed profiles, not a final selection rule.
Pareto comparisons are useful because they do not require choosing a weight. They also cannot choose between methods on the Pareto set: a further priority or constraint is needed. A threshold rule is one alternative when minimum performance on each goal is substantively required. Rank aggregation is another alternative, but it emphasizes order rather than the size of score differences.
Worked Example: Finding Dominated and Non-Dominated Methods
A hypothetical benchmark reports the following batch-removal and biology-preservation scores:
| Method | \(B\) | \(P\) |
|---|---|---|
| R | 0.80 | 0.75 |
| S | 0.78 | 0.82 |
| T | 0.65 | 0.70 |
| U | 0.90 | 0.55 |
R dominates T because \(0.80>0.65\) for batch removal and \(0.75>0.70\) for preservation. S also dominates T because \(0.78>0.65\) and \(0.82>0.70\). Thus T is dominated and is worse than at least one alternative on both reported dimensions.
R and S do not dominate one another: R has higher \(B\), while S has higher \(P\). U also is not dominated by R or S because its batch-removal score \(0.90\) exceeds theirs, although its preservation score is lower. Therefore R, S, and U are non-dominated among these four methods. The profiles alone do not establish which of those three should be preferred.
Suppose, as a separate hypothetical requirement, that acceptable methods must satisfy \(P\geq0.70\). U fails that requirement because \(0.55<0.70\), while R and S meet it. This threshold changes the eligible set rather than trading one goal against the other. The threshold must have a substantive justification and should not be chosen merely to obtain a preferred ranking.
Why Aggregate Rankings Can Mislead
A composite score compresses a two-dimensional profile into one value. That compression can hide several important features:
- Priority dependence. As the worked comparison showed, changing the weight can reverse the ranking even when every component score stays fixed.
- Compensation. A very high batch-removal score can offset poor biology preservation in an arithmetic average, even when that level of biological loss would be unacceptable.
- Scale dependence. Rescaling either component can change the composite unless the weights are adjusted to preserve the intended priorities.
- Magnitude loss in rank aggregation. A method that barely beats another and a method that beats it by a large margin each receive the same one-place rank difference.
- Metric dependence. If several highly similar metrics measure the same target, including all of them as if they were independent goals can give that target extra influence.
A basic justification for checking dominance is direct. If method \(i\) has \(B_i\geq B_j\) and \(P_i\geq P_j\), then for any \(0\leq w\leq1\), \(wB_i+(1-w)P_i\geq wB_j+(1-w)P_j\). This follows by multiplying each nonnegative score difference by its nonnegative weight and adding:
If at least one score difference is strictly positive and its corresponding weight is positive, the composite is strictly higher. Thus a method dominated on both dimensions cannot outrank its dominator under this weighted-mean rule, except that a zero weight can erase a strict advantage on the ignored dimension and produce a tie. This result depends on a common, favorable direction and nonnegative weights; it does not hold for arbitrary transformations or negative weights.
Diagnostics, Alternatives, and Limits
Before interpreting a combined comparison, check the score construction and the benchmark design. Confirm that both components use the same cell population and evaluation conditions, that each direction is correctly oriented, and that scaling anchors were specified consistently. Inspect component scores and the joint scatterplot, not just the composite ranking. If repeated data sets, resampling, or other uncertainty estimates are available, show how rankings vary across them; a small difference in point estimates need not be a stable distinction. A single aggregate number does not account for uncertainty unless the construction explicitly does so.
The main alternatives answer different questions. A weighted composite gives a continuous summary under explicit, compensatory priorities. A threshold rule asks whether both goals meet minimum standards, without permitting one to compensate for failure on the other. Pareto comparison identifies domination and leaves unresolved trade-offs visible. Rank aggregation combines relative order across metrics but discards the magnitude of differences and may be sensitive to which methods are included. None is universally preferable: the appropriate summary depends on the benchmark question and should be stated before interpreting the result.
These comparisons remain limited by the measures themselves. A high score is evidence about the benchmark target as operationalized, not a guarantee that all relevant biology is preserved or all batch effects are removed. Results may differ across cell types, datasets, or benchmark choices. If evaluation conditions are not comparable, a numerical ranking can be precise-looking without supporting a fair comparison. Presenting \(B\) and \(P\), their definitions, their scales, and any aggregation rule is therefore essential.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- Two methods have profiles \((B,P)=(0.70,0.90)\) and \((0.80,0.75)\). Does either dominate the other? Explain.
- For Method V, \(B=0.60\) and \(P=0.90\). Calculate its weighted score when \(w=0.40\).
- Why can adding a third method change scores scaled using the observed minimum and maximum, even if the original two methods do not change?
- A method ranks first by an average of metric ranks but has a much lower preservation score than its competitors. What should be reported before interpreting its first-place ranking?
Solutions. (1) Neither dominates: the first method has higher \(P\), while the second has higher \(B\). (2) \(C=0.40(0.60)+0.60(0.90)=0.24+0.54=0.780\). (3) The observed minimum or maximum can change when a method is added, changing the scaling transformation and therefore the scaled values of the original methods. (4) Report the component scores and metric directions, explain the rank-aggregation rule and its loss of magnitude information, and examine whether the ranking is stable across metrics or evaluation conditions; the first-place rank alone does not show that preservation is adequate.