Scoring Predictive Performance In Simulations

What Does a Predictive Score Measure?

A predictive score is useful only when it answers a defined question. In a simulation, the outcome-generating mechanism is specified, so the benchmark can ask how well a method predicts outcomes for new individuals drawn from a particular target population. A score calculated on the training data instead describes performance on data used to fit the method; it can be optimistic and need not estimate performance on new individuals.

Let \(Y\) denote an outcome, \(X\) the information available at prediction time, and \(f(X)\) a method's prediction. A target-population loss is the expected penalty for predicting with \(f\), averaging over the target distribution of \((X,Y)\). In practice, a simulation approximates this expectation using held-out test observations in each replicate. The choice of test population and its weighting are part of the estimand: a score for an equally weighted mixture of cohorts is not automatically a score for a population in which one cohort is more common.

These properties are distinct. A model can rank individuals correctly while systematically predicting probabilities that are too high. It can also be well calibrated on average while making large individual prediction errors. No single metric summarizes all three properties. Before scoring, specify which variables the method may use, the prediction time, the outcome horizon, and whether the target is a new individual, a new site, or another defined population. Test data must represent that target and must not have contributed to model fitting or tuning.

Continuous Outcomes: Error and Scale

For continuous outcomes, common metrics compare observed values \(Y_i\) with predictions \(\hat{Y}_i\). The mean squared error (MSE) averages squared discrepancies, the root mean squared error (RMSE) returns the result to the outcome's units, and the mean absolute error (MAE) averages absolute discrepancies. The mean prediction error, with sign defined here as prediction minus outcome, describes average over- or underprediction.

MSE and RMSE penalize large errors more heavily than MAE. MSE has squared outcome units; RMSE and MAE have the original outcome units. These scores depend on the scale and spread of the outcome, so comparing raw RMSE values across outcomes measured on different scales can be misleading. A normalized metric can help with scale comparisons, but its denominator must be specified and may be unstable when the target outcome has little variation.

A useful justification explains why squared error reflects both irreducible outcome variability and prediction quality. Suppose \(m(X)=E(Y\mid X)\), and all expectations below are finite. Write \(Y-f(X)=[Y-m(X)]+[m(X)-f(X)]\), square, and take expectations. The cross-product has expectation zero because \(E[Y-m(X)\mid X]=0\). Thus:

The first term is outcome variation that remains even with the conditional mean prediction; the second is the squared discrepancy between the fitted prediction rule and that conditional mean, averaged over the target distribution of \(X\). This identity is about squared-error risk, not a claim that observed test-set MSE separates those components exactly.

Binary Outcomes: Probabilities, Ranking, and Calibration

For a binary outcome \(Y_i\in\{0,1\}\), a method that predicts risk should generally be scored using predicted probabilities \(\hat{p}_i\), not only hard classifications. The Brier score is the mean squared error of those probabilities. Log loss penalizes assigning very small probability to an event that occurs, or to a non-event that occurs. Both are proper scoring rules: at the population level, expected score is minimized by reporting the true conditional event probability, under their usual definitions.

AUC (area under the receiver operating characteristic curve) measures discrimination: it is the probability that a randomly selected case receives a higher predicted risk than a randomly selected non-case, with half credit for a tie. It does not assess whether predicted risks are numerically accurate. Calibration asks whether predictions agree with observed frequencies, for example whether individuals assigned risks near 0.2 experience the event at about 20% frequency over a sufficiently large, relevant group. Calibration plots and calibration-in-the-large or calibration slope summaries can be informative, but estimates from small test sets are imprecise.

Accuracy and sensitivity require a probability threshold, such as classifying an observation as positive when \(\hat{p}_i\geq 0.5\). They can be useful when the decision threshold and costs of errors are specified in advance, but they discard probability information and depend on event prevalence. In simulations with rare outcomes, accuracy can be high for a method that predicts no events. Report threshold-based measures only with the threshold and relevant class-specific results; do not treat them as substitutes for probability error or calibration.

Survival Outcomes: Time, Risk, and Censoring

A survival prediction may be a risk score, a predicted survival curve \(S_i(t)=P(T_i>t\mid X_i)\), or a prediction for a prespecified time horizon. These are different prediction targets and require metrics that match them. Concordance measures ranking: among comparable pairs, it assesses how often the individual with the shorter event time receives the higher predicted risk. It does not measure the accuracy of predicted survival probabilities.

At a fixed horizon \(\tau\), a Brier score compares predicted survival probability with the indicator of surviving past \(\tau\). With complete event-time observation, this is simply a mean squared probability error. In real or simulated data with right censoring, the horizon-specific event status may be unknown for individuals censored before \(\tau\). A censoring-adjusted score, such as an inverse-probability-of-censoring-weighted (IPCW) Brier score, weights contributions by the estimated probability of remaining uncensored long enough for the relevant status to be observed. Its validity depends on an appropriate censoring model, conditional independent censoring given the variables used in that model, and positive probability of being observed over the required time range. If those conditions fail, the adjusted score can be biased or unstable.

Longitudinal Predictions and the Unit of Weighting

For repeated measurements, an observation-level score gives more weight to subjects with more observed visits. That may be appropriate when the target is a randomly selected visit from the target population. If the target is a randomly selected subject, first calculate each subject's average loss and then average those subject-level losses, giving each subject equal weight. These estimands differ when visit counts vary or errors vary within subjects. State whether the prediction concerns a new visit, a subject's trajectory, or a particular time point, and score only the time points and outcomes included in that target.

Diagnostics, Comparisons, and Limitations

A defensible scoring plan starts by matching the metric to the prediction target. For continuous outcomes, RMSE emphasizes large errors and MAE is less sensitive to them. For binary outcomes, Brier score and log loss assess probability predictions, AUC assesses ranking, and calibration summaries assess agreement between risk and frequency. For survival outcomes, concordance focuses on ranking while horizon-specific scores assess survival probabilities at stated times. For longitudinal outcomes, specify whether subjects or visits receive equal weight. None of these metrics alone proves that a method is useful for a clinical decision.

In each simulation replicate, score methods on the same independently generated test population when the comparison is intended to hold the test cases fixed. Aggregate scores only after preserving the replicate-level results and the intended target weighting. The average over test observations approximates a population score; a small test set can make that approximation noisy. The simulated outcome mechanism may also contain irreducible randomness, so even a correctly specified prediction rule will not generally predict every realized outcome exactly.

• Check the target: verify that the test distribution, cohort mix, time horizon, and visit weighting match the intended population.
• Check predictions: probabilities must lie in \([0,1]\); survival curves should be non-increasing in time and within \([0,1]\). Inspect extreme probabilities that can dominate log loss.
• Check outcome prevalence and range: a high accuracy score can conceal poor detection of rare events, and raw continuous errors can be dominated by scale.
• Check censoring and missing visits: identify who has unknown status at the evaluation horizon and whether the censoring or observation assumptions required by a weighted score are plausible.
• Check aggregation: determine whether a subject with many visits is intended to contribute more, and avoid changing the target population accidentally through unequal replicate or cohort weighting.

A frequent mistake is to call AUC a measure of calibration because it is high, or to call a low RMSE proof that predictions are well calibrated in every subgroup. AUC depends on ordering, so any strictly increasing transformation of a risk score preserves its ranking while changing its probability meaning. RMSE averages squared errors and can hide systematic errors in particular groups or time periods. The correction is to report complementary metrics tied to the stated target, inspect relevant calibration and error patterns, and interpret each score only for what it measures.