External Validation Across Sampling Designs

Why Sampling Design Changes an External Estimate

Two external evaluations can apply the same locked population pharmacokinetic model to compatible datasets and still report different average predictive performance. One dataset may concentrate observations near peak concentration; another may contain mostly late samples. One may collect a few samples from many patients, while another collects many samples from each of fewer patients. These differences matter because an average over the observed records gives weight to the times and patients that the design samples most densely.

The key distinction is between a difference in model behavior and a difference in the evaluation distribution. A design-specific summary answers a question about the observations collected under that design. Comparing model behavior across studies usually requires a common target distribution—for example, a defined mix of early, middle, and late sampling occasions—and an uncertainty calculation that respects how observations were sampled.

For illustration, let \(L_{ij}\) denote a prespecified score for patient \(i\) at sampling occasion \(j\), with larger values representing worse performance and units of mg/L. The examples use a mean of these scores only to show how design affects an average; the score itself is assumed to have been defined consistently before comparing datasets. The same weighting principles apply to other summaries, though their uncertainty calculations may differ.

Define the Target Before Comparing Summaries

A summary has meaning only in relation to the distribution it averages over. Suppose sampling occasions are grouped into prespecified time strata \(h=1,\ldots,H\). Let \(\mu_h\) be the mean score in stratum \(h\), and let \(w_h\) be the target weight for that stratum, where \(w_h\geq 0\) and \(\sum_h w_h=1\). The target-standardized mean is

The weights describe the target mix of occasions; they are not automatically the proportions observed in either external dataset. If the target is an equal mix of early and late occasions, the weights are \(0.5\) and \(0.5\), even if one dataset contains 80% early observations. Choosing weights is a scientific and design decision, not a calculation that can be settled from the observed data alone.

Another decision is whether the target unit is an observation or a patient. An observation-weighted mean gives more influence to patients with more recorded occasions. A patient-weighted mean first summarizes each patient's observations and then gives each patient equal weight. Neither is universally correct: they answer different questions. The intended target should be stated before interpreting the estimate.

Density and the Unit Being Weighted

Different sampling densities create a related issue. If every recorded concentration contributes one row to a pooled average, a patient with ten samples contributes ten times as much as a patient with one sample. That may be appropriate when the target is a randomly selected recorded occasion under a specified schedule. It is not the same as asking about the average patient.

Precision Under Stratified Sampling

Standardization changes the target weights, and its standard error must reflect those weights and the amount of information within each stratum. Under a simple stratified design, assume independent patients within and across strata, a prespecified target weight \(w_h\), and \(n_h\) independently sampled patients in stratum \(h\). Let \(s_h\) be the sample standard deviation of the patient-level scores in that stratum. The estimated standardized mean is \(\hat{\mu}_{\text{target}}=\sum_h w_h\bar{L}_h\). Its estimated variance is

The squared weights appear because variance scales with the square of a constant: for independent stratum means, the variance of their weighted sum is the sum of the squared weights times the individual variances. The formula also shows that precision depends on within-stratum variability and stratum sample sizes, not only on the total number of patients.

Repeated Samples and Dependence

When multiple occasions come from the same patient, they are generally not independent. A patient-level characteristic can make that patient's scores similar across occasions. Treating every record as an independent patient may therefore understate uncertainty, especially with many observations per patient. The preceding stratified formula applies directly when its independent units are patients. With repeated measurements, uncertainty must instead account for within-patient clustering, using an analysis or resampling scheme that preserves the patient as the sampling unit.

Sampling density can affect precision as well as weighting. Adding occasions to an already observed patient may improve knowledge about that patient's trajectory, but it does not provide the same independent information as adding another patient. A report should therefore describe both the number of patients and the distribution of observations per patient, rather than relying on the total record count alone.

Diagnostics, Alternatives, and Limitations

A useful comparison begins with a design table: list the time strata, the number of patients and observations in each, and the target weights. Then compare stratum-specific summaries before comparing overall estimates. If the stratum-specific results are similar but the raw overall means differ, a changed time mix is a plausible explanation. If they differ within strata, the overall design mix alone does not explain the discrepancy.

Three common approaches answer different questions:

Standardization is not a remedy for missing information. If a target assigns positive weight to a time stratum that one dataset does not observe, that dataset cannot provide the stratum mean from its own data. Reporting a standardized estimate in that situation requires additional assumptions or external information; it should not be presented as directly observed evidence. Even with all strata represented, small stratum sizes can make a weighted estimate imprecise.

Other failure signals include a large gap between sample-weighted and patient-weighted summaries, one patient contributing a substantial fraction of the records, highly unequal stratum sizes, and uncertainty estimates that change sharply when the patient rather than the record is treated as the resampling unit. These findings call for clarifying the target and the dependence structure, not simply adding decimal places to the result.

A common target improves interpretability, but it does not make two studies identical. Standardized estimates can still differ because of sampling variability or genuine differences in the observed populations and data collection. This tutorial's calculations address how sampling time and density define and weight the evaluation summary; they do not by themselves identify the cause of a remaining difference.