Why Validation Needs a Site-Level View
When an NLME model is evaluated at several external sites, the observations are clustered: patients treated or measured at the same site may share protocols, equipment, populations, or operational conditions. A single pooled score can conceal this structure. A model might predict well for most patients while having a persistent bias at one site, or it might show modest average error at every site but substantial variability among individual observations within each site.
The right summary depends on the prediction target. If the target is a randomly selected patient from the combined external population, sites with more patients should generally contribute more. If the target is a typical external site, each site should generally receive equal weight. Neither weighting is universally correct; the choice must match the target defined for validation. In either case, uncertainty calculations must recognize that patients within a site are not independent replicates of the site-level environment.
A site mean error measures systematic over- or underprediction at that site, while the site RMSE reflects both the site’s average bias and the spread of its errors. A useful validation report typically shows site sample sizes, site mean errors, and a measure of within-site error such as RMSE or the distribution of individual errors. Pooled summaries can then be added, with the weighting scheme stated explicitly.
Decomposing Pooled Squared Error
The pooled mean squared error can be separated into three interpretable pieces: variation among observations around their own site mean, variation among site mean errors, and overall bias. Define the patient-weighted mean error as \(\bar e_w=N^{-1}\sum_j n_j\bar e_j\), where \(N=\sum_j n_j\). Then the following identity holds:
The first term is within-site error variation around each site’s mean error. The second is between-site variation in mean error, weighted by site size. The last is squared overall bias. This is a descriptive decomposition; it does not by itself establish that site differences are caused by site characteristics.
For RMSE, take the square root of pooled mean squared error after specifying the weights. The decomposition applies to squared errors, not directly to RMSEs or absolute errors. In particular, the average of the site RMSEs is generally not the pooled RMSE: averaging square roots is not the same as taking the square root of a pooled average of squares.
Worked Example: Separating Within-Site and Between-Site Error
Suppose four hypothetical external sites each contribute two concentration prediction errors, in mg/L: Site A has \(-1,1\); Site B has \(0,2\); Site C has \(1,3\); and Site D has \(3,5\). The site mean errors are respectively \(0,1,2,4\) mg/L.
The patient-weighted mean error is \(\bar e_w=(0+1+2+4)/4=1.75\) mg/L, since all sites have the same number of observations. The pooled sum of squared errors is \(1+1+0+4+1+9+9+25=50\) \((\mathrm{mg/L})^2\), so the pooled MSE is \(50/8=6.25\) \((\mathrm{mg/L})^2\), and the pooled RMSE is \(\sqrt{6.25}=2.50\) mg/L.
Within each site, the sum of squared deviations from its site mean is 2, so the total within-site sum of squares is 8 \((\mathrm{mg/L})^2\). The squared deviations of the four site means from 1.75 are \(3.0625,0.5625,0.0625,\) and \(5.0625\), summing to 8.75 \((\mathrm{mg/L})^2\). With two observations per site, the between-site contribution to the pooled sum of squares is \(2(8.75)=17.5\) \((\mathrm{mg/L})^2\). The overall-bias contribution is \(8(1.75^2)=24.5\) \((\mathrm{mg/L})^2\).
The pieces add to \(8+17.5+24.5=50\), as required. Dividing by eight gives \(1.00+2.1875+3.0625=6.25\) \((\mathrm{mg/L})^2\). Thus this pooled squared error includes within-site scatter, differences among site mean errors, and a substantial positive overall bias. It would be misleading to describe the result only as “typical individual error” without showing the site pattern.
Choosing the Weighting for the Validation Target
The pooled mean error \(\bar e_w\) and pooled RMSE weight each observation equally, so sites with more observations have more influence. An equal-site mean error, \(J^{-1}\sum_j\bar e_j\), instead gives each site equal influence. These answer different questions. The first describes average error for the observed collection of patients; the second summarizes average site bias when the sites themselves are the units of interest. If the external sites were selected with unequal probabilities or do not represent the intended deployment population, neither unweighted summary automatically represents that population.
Worked Example: Patient-Weighted Versus Site-Weighted Bias
Consider four hypothetical sites with the following mean errors and observation counts: Site A, \(n_1=2\) and \(\bar e_1=0\) mg/L; Site B, \(n_2=2\) and \(\bar e_2=1\) mg/L; Site C, \(n_3=4\) and \(\bar e_3=2\) mg/L; Site D, \(n_4=8\) and \(\bar e_4=4\) mg/L.
The equal-site mean error is \((0+1+2+4)/4=1.75\) mg/L. The patient-weighted mean error is \((2(0)+2(1)+4(2)+8(4))/(2+2+4+8)=42/16=2.625\) mg/L. The difference occurs because the largest site has the largest positive mean error and contributes half of all observations.
If the target is the average observed patient across these sites, 2.625 mg/L is the relevant mean error. If the target is the mean bias across these four sites, 1.75 mg/L is relevant. Reporting both, with their distinct interpretations, makes the influence of site size visible. Neither number alone describes how variable errors are within each site; site-level RMSEs or the underlying error distributions are also needed.
Accounting for Clustered Uncertainty
A common way to express the site structure in mean errors is \(\bar e_j=\mu_e+u_j+\bar\epsilon_j\). Here \(\mu_e\) is the mean error across the target population of sites, \(u_j\) is a site deviation with mean zero and variance \(\tau^2\), and \(\bar\epsilon_j\) is the average within-site error component. Under a simple model with independent, mean-zero within-site errors of variance \(\sigma^2\), the variance of a site mean is \(\tau^2+\sigma^2/n_j\). This makes the key distinction explicit: increasing the number of patients at a site can reduce the within-site contribution, but it does not eliminate genuine between-site heterogeneity \(\tau^2\).
This model is a working description, not a guarantee that site effects are normally distributed or independent. It assumes the sites are independent draws from a defined target population, and the simple \(\sigma^2/n_j\) term assumes independent within-site errors with common variance. When observations have additional repeated-measure or patient-level dependence, that structure must also be represented. A site-level random-effects model, a suitable clustered variance estimator, or a bootstrap that resamples whole sites can address dependence; resampling individual observations as though they were independent sites cannot.
Worked Example: Uncertainty in the Mean Site Error
Use the four hypothetical site mean errors \(0,1,2,4\) mg/L from the first example. Suppose the inferential target is the mean error in a population of comparable sites, and assume these sites are independent, representative draws with approximately normally distributed site mean errors. For this calculation, the unit of analysis is the site, not the individual concentration observation.
The sample mean across sites is \(\bar e_{\text{site}}=1.75\) mg/L. The sample variance across site means is \([(-1.75)^2+(-0.75)^2+(0.25)^2+(2.25)^2]/(4-1)=8.75/3=2.9167\) \((\mathrm{mg/L})^2\). Thus the sample standard deviation is \(\sqrt{2.9167}=1.708\) mg/L, and the standard error of the mean across four independent sites is \(1.708/\sqrt{4}=0.8539\) mg/L.
Using a \(t\) interval with \(df=3\) and critical value \(t_{0.975,3}=3.182\), the 95% confidence interval is \(1.75\mathbin{\pm}3.182(0.8539)=1.75\mathbin{\pm}2.717\), or approximately \((-0.967,4.467)\) mg/L. Under the stated approximate-normality assumption, this interval has approximately 95% coverage for the population mean site error in repeated samples of four sites. Exact 95% coverage requires the site mean errors to be independent and identically normally distributed, as assumed for a one-sample \(t\) interval. Its width reflects having only four independent sites; having many patients within these same four sites would not provide four additional independent site replicates.
Diagnostics, Alternatives, and Limitations
A useful report should show site-specific summaries rather than rely on one pooled number. Examine whether a site mean error is persistently positive or negative, whether one site has unusually large RMSE, and whether the apparent between-site pattern is driven by one small site. Plotting site means with uncertainty intervals alongside individual errors can reveal patterns hidden by pooling. Also inspect each site’s sample size and covariate or dosing support: differences in observed site errors can reflect different patient mixes as well as genuine site-specific performance.
The main descriptive alternatives answer distinct questions:
| Summary | Weighting | Interpretation | Main caution |
|---|---|---|---|
| Pooled mean error or RMSE | Each observation equally | Performance for the combined observed patients | Large sites dominate; clustering affects uncertainty |
| Mean of site mean errors | Each site equally | Average bias for a typical site among those represented | Small and large sites have equal influence |
| Distribution of site metrics | Site is the unit | Variation in performance across sites | Few sites give an unstable estimate of heterogeneity |
With many sites, a hierarchical model can estimate the average site error and between-site variance while accounting for different precision of the site means. With a small number of sites, variance estimates and random-effects assumptions can be fragile; show the site-level data and avoid overconfident generalization. If the sites are a fixed set of intended deployment locations rather than a sample from a broader population, describe those sites directly instead of implying inference to an unspecified population of sites.
A frequent mistake is to treat every patient observation as independent when calculating uncertainty for an average site-level performance measure. That can make standard errors too small because observations within a site share conditions. The correction is to match the analysis unit and uncertainty method to the target: use site-level summaries when sites are the target, and use a method that preserves site clusters when estimating uncertainty for patient-level summaries. A second mistake is to average site RMSEs and call the result the pooled RMSE; compute the pooled squared errors with the intended weights first, then take one square root.
Check Your Understanding
Answer each question, then compare with the brief solutions below.
- Four equally sized sites have mean errors \(-2,0,1,5\) mg/L. What are the equal-site and patient-weighted mean errors?
- Why can a large patient count at one site not substitute for having more independent external sites when estimating between-site heterogeneity?
- When is an equal-site mean error more appropriate than a patient-weighted mean error?
- Why is the mean of the site RMSEs generally different from the pooled RMSE?
Solutions. (1) Both summaries are \((-2+0+1+5)/4=1.00\) mg/L because the sites are equally sized. (2) Patients at one site share site conditions; more patients can improve precision of that site’s summary but do not add independent draws from the population of sites. (3) It is appropriate when the target is the average performance across sites, giving each site equal influence, rather than the average performance across observed patients. (4) Site RMSEs are square roots of site mean squared errors; averaging these roots does not generally equal taking the square root after pooling the weighted squared errors.