Designing an External Validation Dataset

Start With the Prediction Task

An external dataset is useful only in relation to a defined task. A collection of concentration measurements does not, by itself, establish what population or treatment setting the data represent, what predictions the data can evaluate, or whether the recorded information is sufficient to interpret those predictions. Dataset design therefore starts by specifying the intended use and the population to which the evaluation is meant to apply.

For a population PK model, a prediction task should state the drug and formulation, the population of interest, the treatment conditions, and the concentration outcomes to be evaluated. It should also say what information about a patient and their treatment will be available for making the prediction. For example, a task might concern concentrations after a specified regimen in adults with a particular disease, using baseline covariates and dose history available at the time of prediction. The intended prediction time or times should be explicit.

The task specification determines the dataset requirements. If the intended use is prediction in a new hospital, a dataset from that hospital is relevant only if its patients and treatment practices also match the target population. If the intended use concerns concentrations near the end of a dosing interval, measurements only around the expected peak may not be informative. A design should make these connections explicit before recruitment or data extraction.

Specify Patients, Sites, and Treatment Conditions

Eligibility criteria should describe the patients to whom the evaluation is intended to apply, not merely those easiest to recruit. Specify age range, disease or indication, relevant organ-function range, treatment history, and any other defining characteristics. Record clinically relevant covariates even when they are not eligibility criteria: they help characterize who was actually observed and whether the data cover the intended population. Include patients across the range of covariate values expected in use, rather than concentrating only near the average.

Sites and treatment settings matter because clinical practice can shape the data. State the number and type of sites, whether care is inpatient or outpatient, and whether dosing follows a common protocol or routine practice. Record the dose, formulation, route, dosing interval, treatment duration, and relevant dose changes. The design should include the settings to which the conclusions are intended to apply. A large patient count from one unusually specialized site may describe that site well but provide limited information about variation across ordinary practice.

Define the sampling unit and the observation unit separately. Patients are usually the independent sampling units; concentration records are repeated observations within patients. More records per patient can clarify an individual concentration-time profile, but do not create the same breadth of population information as enrolling more patients. The planned balance depends on the task: a design should have enough patients to represent between-patient variation and enough appropriately timed samples to observe the concentration patterns of interest.

Plan Sampling Times Around the Concentration-Time Question

Sampling times should be chosen in relation to dose times and the pharmacokinetic behavior the task needs to evaluate. A schedule intended to characterize early absorption should include appropriately timed early samples; one intended to assess late-interval concentrations needs observations near the end of the dosing interval. When the target concerns the full profile, measurements should cover relevant portions of the interval rather than clustering at one convenient time.

The schedule must be feasible in the clinical setting. A scientifically attractive sequence that patients cannot follow will yield missed or mistimed samples. State whether sampling is at prespecified times, within allowed windows, or opportunistic. For every sample, record the actual collection date and time, not only the nominal time. Record the exact times and amounts of doses around the sample, including deviations from the planned regimen. A concentration without a usable time relative to dosing may be difficult to interpret.

Repeated measurements within a patient are correlated. A simple calculation shows why counting records alone can overstate how much independent information a dataset contains. Suppose, for illustration, that each patient has \(m\) measurements with a common variance \(\sigma^2\), and every pair has correlation \(\rho\). For the mean of those \(m\) measurements,

The factor \(1+(m-1)\rho\) is the design effect relative to \(m\) independent measurements. Under this simplified exchangeable-correlation illustration, the equivalent number of independent observations is \(m/[1+(m-1)\rho]\). It is not a universal sample-size formula for PK profiles: correlations can vary with time, and patient effects and residual variation have different roles. Its purpose is to demonstrate why more samples from the same patient cannot substitute without limit for additional patients.

Define the Outcome and the Minimum Data Record

The principal outcome for a concentration prediction task is a measured drug concentration linked to a particular patient, dose history, and actual sample time. Specify the analyte, biological matrix, units, assay method, and any relevant handling or storage information. Use a consistent laboratory procedure where practical, and preserve the information needed to identify assay or protocol differences. These details help distinguish a genuine model-prediction discrepancy from a measurement or recording problem.

At minimum, each concentration record should be linkable to a patient identifier, sample date and time, actual dose history, and treatment regimen. The patient-level record should include eligibility information, relevant covariates, site, treatment setting, and dates of treatment and sampling. Preserve the original recorded values and units, along with any derived or transformed values used for analysis. Record whether a sample was collected, missed, unusable, or not scheduled; otherwise, the observed dataset may conceal systematic gaps.

The dataset specification should state what information is available when a prediction is made. Do not rely on a covariate or clinical measurement collected only after the prediction time if that information would not be available in intended use. This distinction prevents an evaluation from quietly using information unavailable to the real task. It also makes clear which measurements are predictors or context and which are outcomes used to evaluate the predictions.

Design Checks, Alternatives, and Limitations

Before data collection or extraction is finalized, check whether the specification supports the task. A useful review asks whether the eligibility criteria match intended use; the recruited sites and treatment settings are relevant; the covariates cover clinically important ranges; the sampling schedule addresses the requested times; and each outcome can be linked to reliable dose and time information. Confirm that the expected number of patients, not merely the total number of samples, is adequate for the planned evaluation. Any adequacy claim should be tied to a prespecified precision or decision objective rather than to a generic record-count threshold.

Failure modes include enrolling only highly adherent patients when routine-use prediction is the goal, sampling at convenient times that miss the concentration region of interest, omitting actual dose times, and recording outcomes without assay or unit information. Missingness can also be informative: for example, patients who feel unwell may miss visits or have treatment changed. Report the extent and reasons for missing samples where known. A schedule with poor completion or a narrow covariate range may leave the dataset unable to address its stated task, even when the collected measurements are accurate.

A prospective external study can specify eligibility, sampling windows, and data capture before observations occur. A retrospective external dataset may be less costly and reflect routine practice, but the available records can limit timing precision, covariate coverage, and control over measurement procedures. A dense sampling design describes within-patient profiles more fully but may burden participants and produce many correlated records. A sparse design can be more feasible and may represent routine practice, but its value depends on appropriate timing across patients and sufficient patient-level coverage. Neither design is inherently superior; the choice follows from the prediction task and practical constraints.

External data also have limits. A dataset from a different population or setting may assess performance in that setting without establishing relevance to every intended population. A single site may not reveal between-site variation. A well-documented concentration dataset can support a defined evaluation, but cannot compensate for a task that was never specified or for essential information that was never recorded.