Why Compare Predictive Scores in Pairs?
When two frozen NLME models are evaluated on the same external subjects, each subject contributes information about both models. The comparison should preserve that pairing: a subject who is difficult for both models should not be treated as two unrelated pieces of evidence. The central quantity is the difference between the models’ scores on the same independent evaluation unit, followed by an uncertainty calculation across independent units.
A score must be selected before looking at which model performs better. It might be a log predictive score, a prediction-error score, or another score defined for the external prediction target. Its direction must also be made explicit. In this tutorial, a larger score means better predictive performance. If the original measure is a loss for which smaller is better, reverse its sign or define the contrast so that a positive value still favors the model identified as better.
The pairing is within an evaluation unit; it does not make all records within that unit independent. A subject with several concentration measurements might have one prespecified subject-level score, such as the average per-record score, or a score for the whole profile. The choice determines the estimand and must be stated. If subjects are clustered within sites and site-level dependence matters, the independent units for uncertainty may instead be sites, with each site contributing a prespecified contrast.
Paired Inference and Its Assumptions
Suppose there are \(n\) independent evaluation units with differences \(d_1,\ldots,d_n\). The sample mean difference and sample standard deviation are
If the independent-unit differences are drawn from a population with mean \(\mu_d\) and standard deviation \(\sigma_d\), the standard error of \(\bar{d}\) is estimated by \(SE(\bar{d})=s_d/\sqrt{n}\). The paired t interval and test are justified when the units are independent and the differences are approximately normally distributed; with normally distributed differences, the t procedure is exact. For a large number of independent units, the mean can often be approximately normal even when individual differences are not, but severe skewness or influential differences can still undermine that approximation.
The variance reduction from pairing follows directly from subtracting the scores before estimating variability. Since \(d_i=S_{Ai}-S_{Bi}\),
When the two scores tend to rise and fall together across subjects, their covariance is positive and the variability of the difference can be lower than the variability implied by treating the two model samples as unrelated. Pairing does not guarantee a narrower interval; it uses the actual within-unit relationship rather than discarding it.
Worked Example: A Paired Comparison on Five External Subjects
Suppose the prespecified score is the negative absolute prediction error, so larger scores mean smaller absolute errors. Its units are mg/L. On five hypothetical external subjects, model A and model B have the following subject-level scores.
| Subject | Model A score (mg/L) | Model B score (mg/L) | Difference \(d_i=A-B\) (mg/L) |
|---|---|---|---|
| 1 | -2 | -4 | 2 |
| 2 | -4 | -5 | 1 |
| 3 | -3 | -6 | 3 |
| 4 | -7 | -6 | -1 |
| 5 | -4 | -4 | 0 |
The independent units are subjects, and each subject contributes a paired score. Assume these subjects are independent draws from the target population and that the differences are sufficiently close to normal for a t interval to be reasonable. The mean difference is
The deviations from the mean are \(1,0,2,-2,-1\) mg/L, whose squared values sum to \(10\). Thus \(s_d=\sqrt{10/4}=1.581\) mg/L and \(SE=1.581/\sqrt{5}=0.7071\) mg/L. With \(df=4\), the 95% t critical value is 2.776, giving
The test statistic for \(H_0:\mu_d=0\) is \(t=1.00/0.7071=1.414\), with \(df=4\) and a two-sided \(p\)-value of about 0.230. These data do not provide strong evidence of a nonzero mean score difference. The interval includes both a possible advantage for B and a possible advantage for A; it does not establish that the models are equivalent.
Repeated Records and Site Clustering
A common evaluation data set has multiple measurements per person. Those records can be informative for calculating a person’s score, but they are generally not independent replications of the model comparison. If the target is the mean score difference per subject, calculate each subject’s score according to a prespecified rule and then calculate one difference per subject. The number of independent units is then the number of subjects, not the total number of records.
Worked Example: Repeated Measurements Within Subjects
For illustration, let each entry below be the paired difference in negative absolute error, \(A-B\), at one of three scheduled times for a hypothetical subject. The target is the average per-record difference within a subject, followed by the mean of those subject-level averages. Assume the four subjects are independent.
| Subject | Time 1 | Time 2 | Time 3 | Subject mean difference (mg/L) |
|---|---|---|---|---|
| 1 | 1 | 2 | 0 | 1 |
| 2 | 0 | 1 | 2 | 1 |
| 3 | -1 | 0 | 1 | 0 |
| 4 | 2 | 1 | 0 | 1 |
The four subject means are \(1,1,0,1\) mg/L. Their mean is \(\bar{d}=3/4=0.750\) mg/L. Their squared deviations from 0.750 sum to \(0.750\), so \(s_d=\sqrt{0.750/3}=0.500\) mg/L and \(SE=0.500/\sqrt{4}=0.250\) mg/L. For a 95% interval, \(t_{0.975,3}=3.182\), so the interval is \(0.750\pm3.182(0.250)=0.750\pm0.7955\), or \((-0.0455,\ 1.546)\) mg/L. It includes zero. The calculation uses four independent subjects, not twelve time-point differences as though they came from twelve independent people.
If subjects are grouped within external sites, subjects at the same site may share characteristics or data-collection processes. Treating every subject as independent can then understate uncertainty about performance across sites. One transparent approach, when the target weights sites equally, is to calculate a prespecified mean paired difference within each site and conduct inference across independent sites. This changes the target from an average over all subjects to an average over sites; the weighting must match the scientific question.
Worked Example: Comparing Models Across Six External Sites
Suppose six hypothetical independent sites contribute equal-weight site-level mean differences, in score points, of \(0.4, 0.2, 0.5, -0.1, 0.3,\) and \(0.1\), with positive values favoring A. Assume the sites can reasonably be treated as independent draws from the target site population and that the site contrasts are sufficiently well behaved for a t interval. Here \(n=6\), not the number of subjects across all sites.
The sum is \(1.4\), so \(\bar{d}=1.4/6=0.2333\) score points. The squared deviations from 0.2333 sum to approximately 0.2333; hence \(s_d=\sqrt{0.2333/5}=0.2160\), and \(SE=0.2160/\sqrt{6}=0.08819\). With \(df=5\), the 95% critical value is 2.571. The interval is
The test statistic is \(t=0.2333/0.08819=2.646\), with \(df=5\), giving a two-sided \(p\)-value of approximately 0.046. Under the stated assumptions, the interval and test provide evidence that the mean site-level contrast is positive. The interval is only just above zero, and six sites provide limited information about the distribution of site effects; the result should not be presented as a precise estimate of performance at every site.
Diagnostics, Alternatives, and Limitations
Before interpreting an interval or test, check the analysis unit, the score construction, and the pattern of differences. A useful sequence is:
Verify that the score and direction were prespecified, model predictions were produced without validation-data leakage, and both models were scored on the same eligible units.
Determine whether the target is subjects, sites, or another cluster. Repeated records within a subject and subjects within a site do not automatically create independent units.
Plot or tabulate the unit-level differences. Look for skewness, outliers, different patterns by site, and a small number of units dominating the mean. Check whether missing scores or exclusions affect the two models differently.
Use a paired t procedure only when its independence and distributional conditions are defensible. For clustered targets, calculate uncertainty across independent clusters or use a method that explicitly accounts for clustering.
An unpaired comparison of the two models’ scores discards the within-unit pairing and can misstate uncertainty. Treating every concentration record as an independent paired comparison is also generally wrong: within-subject dependence makes the nominal sample size too large. If there are many independent clusters, cluster-robust standard errors or a bootstrap that resamples whole clusters can be alternatives to a t analysis of cluster summaries. Resampling individual records within clusters does not preserve the dependence structure. With very few clusters, neither a large-sample cluster-robust calculation nor a cluster bootstrap guarantees reliable coverage; results may need to be framed as descriptive and interpreted cautiously.
The paired t interval targets the mean of the chosen unit-level score differences. It does not establish that one model is better for every subject, at every time, or at every site. A mean can hide important heterogeneity, and a few extreme differences can strongly influence it. The method also cannot repair a score chosen after inspecting results, a mismatch between validation subjects and the intended deployment population, or model adaptation using external outcomes. The score definition, target population, aggregation rule, and independent unit all shape what the estimated contrast means.
Common Pitfall
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- If \(d_i=S_{Ai}-S_{Bi}\) and larger scores are better, what does a negative mean difference indicate?
- A subject has five repeated records. If the prespecified target is mean score difference per subject, how many paired differences does that subject contribute?
- For six independent clusters, what degrees of freedom are used in a one-sample t interval for the cluster contrasts?
- Why can an unpaired analysis be inefficient or misleading when both models are scored on the same subjects?
- A confidence interval for the mean paired difference includes zero. Does that prove the two models have equal predictive performance?
Solutions. (1) A negative mean difference favors B on average under the specified score. (2) One, formed using the prespecified subject-level aggregation rule. (3) \(6-1=5\). (4) It discards the within-subject relationship; the paired difference directly captures the shared subject difficulty and its variability. (5) No. It indicates that the data do not rule out a zero mean difference at the interval’s confidence level; it does not establish equality or equivalence.