Selecting Metrics for Integration Comparisons

Why Metric Choice Matters

An integration benchmark can produce several reasonable summaries of prediction performance, and those summaries need not rank strategies in the same order. A method may make small errors for most individuals but a few very large errors; another may have lower average absolute error but worse extreme misses. For a binary outcome, a strategy may classify most observations correctly by favoring the common class while failing to identify the uncommon but important class. A metric is not a neutral decoration: it defines which aspects of prediction count as good.

The comparison is meaningful only when strategies are evaluated on the same held-out individuals, with the same outcome definition and compatible prediction targets. The held-out data should represent the population to which performance is meant to generalize, and all preprocessing and model fitting must be confined to the training data. A metric cannot repair an evaluation design that does not support the intended comparison.

Continuous Outcomes: Error Size and Error Shape

Let \(y_i\) be the observed continuous outcome and \(\hat{y}_i\) its prediction for evaluation individual \(i\), for \(i=1,\ldots,n\). The prediction error is \(e_i=\hat{y}_i-y_i\). Two common summaries are the mean absolute error (MAE) and root mean squared error (RMSE):

Both are expressed in the outcome's units. MAE gives the average absolute prediction error and is less dominated by a small number of large errors. RMSE gives greater weight to large errors because errors are squared before averaging. Neither is universally preferable: if unusually large errors have especially serious consequences, RMSE may reflect the objective better; if the desired summary is a typical absolute miss, MAE may be easier to interpret.

A related summary is the test-set coefficient of determination:

This compares squared prediction error with the squared error from predicting the evaluation-set mean \(\bar{y}\) for everyone. It is dimensionless; higher values indicate less squared error relative to that reference. On held-out data, \(R^2_{\mathrm{test}}\) can be negative when predictions are worse than the reference. It is not a percentage of individual outcomes correctly explained, and its value depends on the outcome variation in the evaluation sample.

The relationship between MAE and RMSE helps explain why they can disagree in comparisons. Squaring makes a large error count disproportionately: an error of 6 contributes 36 to the squared-error sum, whereas an error of 2 contributes 4. Consequently, a strategy with one extreme miss can have a competitive MAE but a poor RMSE. If all strategies are evaluated on the same observations, minimizing RMSE and minimizing mean squared error give identical rankings because the square root is increasing. MAE need not give that ranking.

Binary Outcomes: Thresholds, Probabilities, and Rankings

For a binary outcome, write \(y_i=1\) for the event and \(y_i=0\) otherwise. A model may provide a predicted probability \(\hat{p}_i\), or a predicted class obtained by applying a threshold \(c\): predict 1 when \(\hat{p}_i\ge c\), and 0 otherwise. The threshold is part of a threshold-based metric's definition. The resulting counts are true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN).

Sensitivity is the proportion of actual events detected; specificity is the proportion of non-events correctly ruled out; precision is the proportion of predicted events that are actual events. Balanced accuracy is the mean of sensitivity and specificity. These proportions are undefined if their denominators are zero, so report the relevant counts and handle such cases explicitly rather than silently assigning a value.

Accuracy uses one threshold and gives equal weight to each correctly classified observation. When the event is uncommon, predicting the non-event for everyone can yield high accuracy while sensitivity is zero. In contrast, sensitivity and specificity show performance for each actual class, and precision speaks directly to how often a positive prediction is correct. Precision depends on event prevalence in the evaluation population, so it may change when prevalence changes, even if sensitivity and specificity do not.

When probability predictions matter, rather than only classifications at one threshold, two standard losses are the Brier score and log loss. For binary outcomes, they are

Both require probabilities between 0 and 1, and smaller values are better. Brier score is mean squared probability error. Log loss penalizes confident predictions that assign very low probability to the outcome that occurs; a probability of exactly 0 for an event or exactly 1 for a non-event gives infinite loss. In computation, probabilities may be bounded away from 0 and 1 using a prespecified rule, which should be reported.

Area under the receiver operating characteristic curve (ROC AUC) summarizes ranking: it is the probability that a randomly selected event receives a higher score than a randomly selected non-event, with ties counted as one-half. It does not assess performance at a particular threshold. Precision-recall summaries focus on precision and sensitivity across thresholds and can be informative when events are rare; the precise area calculation can depend on the implementation, so the convention should be stated. Neither kind of curve summary replaces a decision-relevant threshold analysis.

A Practical Selection Framework

Before computing metrics, state what is predicted and what decision the prediction is meant to support. Then choose a small set of complementary summaries rather than selecting whichever metric makes a preferred strategy look best. The following distinctions help make that choice explicit.

If the benchmark includes several simulated scenarios, define the primary metric before comparing strategies and report scenario-specific performance rather than relying only on a pooled value. A metric's meaning also depends on the outcome scale, class prevalence, threshold, and evaluation population. A single overall score can conceal which scenarios or types of errors drive the result.

Diagnostics, Alternatives, and Limitations

Check the ingredients of a metric before interpreting it. For continuous outcomes, inspect the distribution of errors and identify whether a few extreme errors dominate RMSE. Confirm that outcome units are consistent across strategies and scenarios; MAE and RMSE are not directly comparable across differently scaled outcomes without a deliberate standardization plan. Test-set \(R^2\) can be unstable when the observed outcomes have little variation and can be negative, so report an error metric as well.

For binary outcomes, report event and non-event counts, the probability threshold when using class metrics, and the confusion counts behind the proportions. With rare events, accuracy can be deceptively high, while precision can change substantially as prevalence changes. ROC AUC can look strong even when precision at useful thresholds is low. Brier score and log loss are defined for probabilities, not merely class labels; converting probabilities to classes discards information these losses use.

No metric identifies the best integration strategy independently of the use case. If false negatives and false positives have different consequences, sensitivity and specificity at a prespecified threshold may be more actionable than accuracy, but the costs or acceptable trade-offs must come from the application. If a strategy is better on one metric and worse on another, report that trade-off instead of treating the metrics as interchangeable or choosing a winner after seeing the results. These summaries also do not by themselves establish uncertainty in a ranking or explain which omics block contributes the performance; those require separate analyses.