Validation Across Multiple External Studies

Why Validation Across Studies Requires More Than One Pooled Number

A model may be evaluated in several independent external studies, with each study reporting a prediction-performance estimate and its uncertainty. A summary can help describe performance across settings, but a single pooled number can conceal an important fact: the model may perform differently from one study to another. For external population pharmacokinetic validation, the scientific question is therefore not only “What is the average performance?” but also “How much does performance vary across settings, and is that variation practically important?”

The estimates must first be comparable. They should refer to the same locked model, the same performance measure and direction, and sufficiently aligned definitions and units. Assay, protocol, population, and sampling-design differences can affect performance; harmonizing or explaining such differences is part of interpreting a multi-study result, not something that a pooling formula can repair. The material on Assay and Protocol Differences in External Validation is relevant when deciding whether study estimates can be placed on a common scale.

Definition: A study-level performance estimate is a numerical estimate of a specified validation measure calculated within one external study. Let \(y_i\) denote the estimate from study \(i\), and let \(v_i\) denote its estimated sampling variance. The standard error of \(y_i\) is \(\sqrt{v_i}\). A multi-study synthesis combines these estimates while retaining the fact that they arise from distinct study settings.

For example, suppose each study reports mean prediction error defined as predicted concentration minus observed concentration. A positive value indicates overprediction on average, a negative value indicates underprediction, and zero indicates no average signed error. The measure has concentration units. This interpretation does not apply to every metric: root mean squared error, coverage, and calibration slope have different meanings and scales. They should not be combined as if they were interchangeable.

Fixed-Effect and Random-Effects Summaries

A fixed-effect synthesis assumes that all studies share one underlying true performance value, while their reported estimates differ because of sampling error. Under that assumption, more precise estimates receive greater weight. A random-effects synthesis allows the underlying true performance to differ across studies. It estimates an average across a distribution of study-specific true performances and includes both within-study uncertainty and between-study variation in its weights.

Definition: In a fixed-effect model, \(y_i=\theta+\epsilon_i\), where \(\theta\) is the common true study performance and \(\epsilon_i\) is sampling error with variance \(v_i\). In a random-effects model, \(y_i=\theta_i+\epsilon_i\), with study-specific true performance \(\theta_i\) varying around a mean \(\mu\), and between-study variance \(\tau^2\). The random-effects summary estimates \(\mu\), the mean of the distribution of true study performances, not a guarantee that every study has performance near \(\mu\).

These models require independent studies for the usual formulas below. If the same participants contribute to multiple reports, or studies share substantial data, their estimates may be correlated; treating them as independent can understate uncertainty. The standard random-effects calculation also assumes that the study estimates and their variances are usable on a common scale, and that the distribution of true study effects is adequately represented by a mean and variance. With few studies, that last assumption is difficult to assess.

For the fixed-effect estimate, set \(w_i=1/v_i\), the inverse-variance weight. The pooled estimate is the weighted mean. A widely used moment estimator of \(\tau^2\), the DerSimonian–Laird estimator, adds a between-study variance estimate to each study’s sampling variance before computing random-effects weights.

\hat{\theta}_{F}=\frac{\sum_{i=1}^{k}w_i y_i}{\sum_{i=1}^{k}w_i}, \qquad Q=\sum_{i=1}^{k}w_i(y_i-\hat{\theta}_{F})^2, \qquad C=\sum_{i=1}^{k}w_i-\frac{\sum_{i=1}^{k}w_i^2}{\sum_{i=1}^{k}w_i}

\hat{\tau}_{DL}^{\,2}=\max\left(0,\frac{Q-(k-1)}{C}\right), \qquad w_i^*=\frac{1}{v_i+\hat{\tau}_{DL}^{\,2}}, \qquad \hat{\mu}_{R}=\frac{\sum_{i=1}^{k}w_i^*y_i}{\sum_{i=1}^{k}w_i^*}

Here \(k\) is the number of studies. The quantity \(Q\) measures the weighted dispersion of the study estimates around the fixed-effect estimate; under the common-effect model and usual approximations, it is compared with a chi-squared distribution with \(k-1\) degrees of freedom. The DerSimonian–Laird estimate cannot be negative, hence the maximum with zero. A conventional standard error for the random-effects mean, treating \(\hat{\tau}_{DL}^{\,2}\) as fixed, is \(\sqrt{1/\sum_i w_i^*}\). This calculation does not fully account for uncertainty in estimating \(\tau^2\), which matters especially when \(k\) is small.

The logic of the random-effects weights is direct. If the study-specific true values vary with variance \(\tau^2\), then an estimate from study \(i\) has total variance \(v_i+\tau^2\): sampling variation contributes \(v_i\), and variation among true study settings contributes \(\tau^2\). Inverse total-variance weights therefore give \(w_i^*=1/(v_i+\tau^2)\). When \(\tau^2\) is large relative to differences in \(v_i\), the weights become more similar, so a very large study no longer dominates solely because it has a small sampling variance.

Worked Calculations and Study-Level Interpretation

Worked Example: Random-Effects Synthesis of Mean Prediction Error

Hypothetical data. Three independent external studies evaluate the same locked PK model and report mean prediction error, predicted minus observed concentration, in mg/L. Their estimates are \(0.1981\), \(1.100\), and \(2.002\) mg/L, and each has estimated sampling variance \(0.2500\;(\text{mg/L})^2\). The studies are treated as independent, and their metric definitions and units are aligned.

Each fixed-effect weight is \(w_i=1/0.2500=4.000\;(\text{mg/L})^{-2}\). The weighted mean is therefore the ordinary mean in this equal-variance example:

\hat{\theta}_{F} =\frac{4(0.1981)+4(1.100)+4(2.002)}{12} \approx 1.100\ \text{mg/L}.

Using the unrounded values \(0.198127\), \(1.100\), and \(2.001873\), whose deviations from 1.100 mg/L are approximately \(-0.901873\), \(0\), and \(0.901873\), gives:

Q=\frac{(-0.901873)^2+0^2+(0.901873)^2}{0.2500} \approx 6.507.

There are \(k-1=2\) degrees of freedom. Also, \(C=12-(4^2+4^2+4^2)/12=8.000\;(\text{mg/L})^{-2}\). Thus:

\hat{\tau}_{DL}^{\,2} =\frac{6.507-2}{8.000} =0.563375\;(\text{mg/L})^2.

The random-effects weight for each study is \(1/(0.2500+0.563375)\approx1.2294\;(\text{mg/L})^{-2}\). Since all three weights are equal, the random-effects mean remains approximately 1.100 mg/L. Its conventional standard error, treating \(\hat{\tau}_{DL}^{\,2}\) as fixed, is:

SE(\hat{\mu}_{R}) =\sqrt{\frac{1}{3(1.2294)}} \approx 0.5207\ \text{mg/L}.

A conventional normal-approximation 95% confidence interval is \(1.100\pm1.96(0.5207)\), or approximately \(0.0794\) to \(2.121\) mg/L. Under the model and its approximation, this interval describes uncertainty in the mean true performance across the distribution of study settings. It is not an interval containing 95% of future study effects, and it does not establish that every setting has positive bias. The positive mean estimate indicates average overprediction, but the study estimates range from near zero to about 2 mg/L, so the pooled mean alone is not an adequate account of consistency.

Worked Example: Comparing Fixed-Effect and Random-Effects Weighting

Hypothetical data. Three independent external PK studies report mean prediction errors of \(-0.200\), \(0.400\), and \(1.000\) mg/L, with sampling variances \(0.0400\), \(0.250\), and \(0.640\;(\text{mg/L})^2\), respectively. Suppose first that a common true performance value is scientifically defensible. The fixed-effect weights are \(25.00\), \(4.000\), and \(1.5625\;(\text{mg/L})^{-2}\).

\hat{\theta}_{F} =\frac{25(-0.200)+4(0.400)+1.5625(1.000)} {25+4+1.5625} =\frac{-1.8375}{30.5625} \approx -0.06013\ \text{mg/L}.

The fixed-effect standard error is \(\sqrt{1/30.5625}\approx0.1809\) mg/L, giving a conventional 95% interval of \(-0.06013\pm1.96(0.1809)\), or approximately \(-0.415\) to \(0.294\) mg/L. This summary is dominated by the first, most precise study. That weighting is appropriate for a common-effect estimand only if true performance is in fact common across these settings.

The dispersion statistic is \(Q=\sum w_i y_i^2-(\sum w_i y_i)^2/\sum w_i\). Substitution gives \(Q=3.2025-(-1.8375)^2/30.5625\approx3.092\), with 2 degrees of freedom. This does not by itself prove that the common-effect assumption is correct: with only three studies, a test of heterogeneity has limited ability to detect variation. If meaningful setting-level variation is expected, a random-effects summary or separate study-level interpretation is more relevant than allowing the most precise study to stand in for all settings.

Worked Example: A Leave-One-Study-Out Consistency Check

Return to the first hypothetical example, with study estimates \(0.1981\), \(1.100\), and \(2.002\) mg/L and equal variances. A simple sensitivity check recomputes the arithmetic mean after omitting each study. Omitting the first gives \((1.100+2.002)/2=1.551\) mg/L; omitting the second gives \((0.1981+2.002)/2=1.100\) mg/L; omitting the third gives \((0.1981+1.100)/2=0.6491\) mg/L.

The leave-one-out means range from 0.6491 to 1.551 mg/L, despite the full-sample mean of about 1.100 mg/L. This indicates that the estimated average depends noticeably on which study is included. It is a sensitivity diagnostic, not a new confidence interval or a formal test. A full random-effects refit would also re-estimate \(\tau^2\) each time, and with so few studies those estimates can be unstable. The calculation nevertheless makes the influence of individual settings visible.

Assessing Heterogeneity and Choosing a Synthesis

A forest plot is often the clearest first display: show each study estimate with its uncertainty interval, then show the pooled estimate if pooling is justified. Inspect the direction and magnitude of study estimates, their uncertainty, and whether differences correspond to plausible setting features such as population, sampling schedule, assay, or protocol. A pooled value should not erase study-specific results.

Two common summaries of heterogeneity are \(Q\) and \(I^2\). \(Q\) compares observed dispersion with what would be expected from sampling error under a common-effect model. A large value relative to its degrees of freedom can be evidence against that model, but a nonsignificant result is not proof of homogeneity. \(I^2\) describes the proportion of observed dispersion in the study estimates that is attributed to between-study heterogeneity rather than sampling error under the calculation’s assumptions. It is not the percentage of studies that differ, nor does it measure the absolute size of the differences. Its value depends on the precision of the studies, so interpret it alongside \(\hat{\tau}^2\), study estimates, and the performance scale’s practical meaning.

In the first worked example, \(I^2=\max(0,(Q-(k-1))/Q)\times100\%\), so \(I^2=(6.507-2)/6.507\times100\%\approx69.3\%\). This indicates considerable relative heterogeneity by that measure, but it does not say whether differences of this size are clinically or operationally consequential. For that judgment, interpret \(\hat{\tau}^2\) in mg/L squared and consider the range of study-level errors in mg/L.

Approach	What it summarizes	Main caution
Fixed-effect synthesis	A common true performance value, with study weights based on sampling variance	Can give an overly narrow or misleading summary when true study performance varies
Random-effects synthesis	The mean of a distribution of study-specific true performance values	Does not imply each setting performs like the mean; estimates of heterogeneity are uncertain with few studies
Study-by-study reporting without pooling	The observed performance and uncertainty in each setting	Offers no single summary, but may be the most honest choice when studies are not comparable

Pooling is inappropriate when studies report incompatible measures or units, when one measure has been defined in different ways, or when studies are dependent and that dependence has not been handled. Converting units may solve a unit mismatch, but it does not make different estimands equivalent. For example, signed mean error and root mean squared error answer different questions and should not be averaged together. If clinically important study differences are systematic, a pooled mean can be supplemented by stratified summaries; the rationale and limits of any strata must be stated rather than inferred from a few observed differences.

Bootstrap calculations must also preserve the study structure. Resampling individual observations after combining all studies as though they came from one population can remove between-study differences and produce uncertainty for the wrong target. When quantifying uncertainty in study-level estimates, resampling should respect the original study as a unit and, where appropriate, resample observations within each study. Inference about a broader distribution of study settings requires enough independent studies; resampling a very small set of studies cannot create information about settings that were never observed.

Common Pitfalls and Practical Limits

Reporting only the pooled estimate. A mean prediction error of 1.1 mg/L does not show whether every study had similar error. Report the study-level estimates and uncertainty alongside any pooled summary.
Calling a nonsignificant heterogeneity test evidence of consistency. A test with few studies may have low power. Lack of strong evidence against a common effect is not proof that effects are identical.
Using random effects as a cure for incompatibility. A random-effects formula allows true values to vary; it cannot reconcile different metrics, units, or validation definitions.
Interpreting a confidence interval for the mean as a range for future studies. The interval around \(\hat{\mu}_{R}\) concerns uncertainty in the mean. A future study’s true performance also varies by \(\tau^2\), and with few studies its prediction is especially uncertain.
Confusing average study performance with average patient performance. Inverse-variance pooling targets a mean across study effects as specified by the model. It is not automatically the result that would be obtained by combining all patients into one large dataset.

The main limitation is often the number and diversity of independent external studies. With few studies, estimates of \(\tau^2\), confidence intervals, and assessments of the distribution of true performance can be fragile. A random-effects result should therefore be presented with its assumptions, the individual study results, and a clear description of the settings represented. It is evidence about those observed external evaluations; broader transportability requires judgment about whether they represent the settings where the model will be used.

Key takeaway: A multi-study validation should preserve two levels of information: uncertainty within each study and variation between study settings. Choose a fixed-effect or random-effects summary only after defining the estimand and assessing comparability, and never let the pooled number replace the study-level pattern.

Check Your Understanding

Answer each question, then compare with the brief solutions.

Three studies estimate the same performance measure in the same units but have very different sampling variances. What determines their fixed-effect weights?
In a random-effects model, what does the pooled mean estimate, and what does it not guarantee?
Why does a nonsignificant \(Q\) test not establish that performance is consistent across studies?
Two studies report mean prediction error, while another reports root mean squared error. Should all three be pooled directly? Why or why not?

Solutions. (1) The fixed-effect weight for study \(i\) is \(1/v_i\), so lower sampling variance receives greater weight. (2) It estimates the mean of the distribution of study-specific true performances; it does not guarantee that each setting has performance close to that mean. (3) With few studies, the test may have limited power, so failure to reject the common-effect model is not proof of homogeneity. (4) No. Signed mean error and root mean squared error are different performance measures with different interpretations; they do not estimate one common quantity and should not be averaged as if they did.