Why External Validation Needs a Precision Plan
An external validation can produce a prediction-bias estimate, an error summary, and an interval-coverage estimate, but the numerical result alone does not show how precisely it has been measured. A small estimated bias may be compatible with both negligible bias and a practically important error if its uncertainty is large. Planning the numbers of subjects, observations, and sampling times helps determine whether the validation can distinguish those possibilities.
The independent sampling unit is usually the subject, not each concentration record. Repeated concentrations from one person share that person's pharmacokinetic characteristics and are therefore generally dependent. More sampling times can improve the characterization of a subject's concentration profile, but they do not create the same amount of new information about between-subject performance as enrolling more independent subjects.
The model is assumed to be locked for external validation: predictions are generated without re-estimating the model using the validation outcomes. The target population, prediction task, error definition, and weighting scheme must also be specified before planning. This is consistent with prediction without re-estimation and with distinguishing validation from model updating.
Choose the Estimand and Independent Unit
Let \(Y_{ij}\) be the observed concentration for subject \(i\) at sampling time \(j\), and let \(\hat{Y}_{ij}\) be the corresponding prediction from the locked model. Define the prediction error as \(e_{ij}=Y_{ij}-\hat{Y}_{ij}\), in concentration units. Under this convention, positive error means the prediction is below the observed concentration.
If the target is average performance across subjects, a natural approach is to construct a summary for each subject and then average those summaries over the \(n\) independent subjects. For example, a subject's mean error is \(B_i=m_i^{-1}\sum_{j=1}^{m_i}e_{ij}\), where \(m_i\) is that subject's number of sampled times. The estimated mean subject-level bias is \(\bar{B}=n^{-1}\sum_{i=1}^{n}B_i\). This gives each subject equal weight, even if their numbers of observations differ.
A different estimand is the average error over all observed records, which weights subjects with more records more heavily. That may be appropriate for a record-weighted prediction task, but it is not automatically equivalent to average performance for a randomly selected subject. State which estimand is intended. For coverage, the same choice arises: estimate average within-subject coverage and then average across subjects, or estimate coverage across all records. The sampling plan and uncertainty calculation must match the chosen estimand.
For subject-level summaries \(T_i\), assume the validation subjects are independently sampled from the target population and that the summaries are defined comparably across subjects. If \(T_i\) has finite variance \(\sigma_T^2\), then the variance of their sample mean is \(\sigma_T^2/n\), and its standard error is estimated by \(s_T/\sqrt{n}\). This is the basic reason that adding independent subjects improves precision.
Precision for External Bias
For the mean subject-level bias, a conventional confidence interval is \(\bar{B}\pm t_{1-\alpha/2,n-1}s_B/\sqrt{n}\), where \(s_B\) is the sample standard deviation of the \(B_i\) values. This interval assumes independent subjects and uses a normal approximation for the distribution of the subject-level mean; the \(t\) interval is exact when the subject-level summaries are normally distributed. With enough subjects, moderate departures from normality are often tolerable, but strong skewness or influential subjects can undermine the approximation.
If a desired confidence-interval half-width is \(h\), a first planning approximation, using a pilot estimate \(\sigma_B\), is \(n\approx (z_{1-\alpha/2}\sigma_B/h)^2\). Here \(z_{1-\alpha/2}\) is the corresponding standard normal quantile. Because the \(t\) critical value depends on \(n\), a final plan can check integer sample sizes using \(t_{1-\alpha/2,n-1}\sigma_B/\sqrt{n}\le h\). The calculation targets precision for the mean bias; it does not choose sample size to detect a specified nonzero bias.
Worked Example: Planning Precision for Mean Bias
Suppose a hypothetical pilot validation suggests that the standard deviation of subject-level mean errors is \(8\) mg/L. Plan a two-sided 95% confidence interval with a half-width no greater than \(2\) mg/L. Assume independent subjects, a representative sample from the target population, and a subject-level error summary defined in the same way for everyone.
Initial approximation. Using \(z_{0.975}=1.96\),
This suggests at least 62 subjects under the normal approximation. Check the \(t\)-based half-width using the pilot standard deviation. For \(n=64\), \(df=63\) and \(t_{0.975,63}\approx1.99834\), giving
For \(n=63\), \(df=62\), and the half-width is approximately \(1.9990(8/\sqrt{63})\approx2.015\) mg/L. Thus 64 subjects meet the stated \(t\)-based target using the pilot standard deviation. This calculation is conditional on that estimate being suitable for the planned population and design; uncertainty in a small pilot can make the achieved interval wider.
Precision for Prediction Error and Interval Coverage
Error metrics such as mean absolute error and root mean squared error (RMSE) are nonlinear summaries, and their uncertainty should still respect subject clustering. A useful general strategy is to define each subject's contribution to the metric, estimate the overall metric, and quantify uncertainty by resampling subjects or by a suitable variance approximation. Resampling individual records as if they were independent breaks the subject-level sampling structure.
Coverage is the proportion of observations whose prespecified prediction intervals contain their corresponding outcomes. For a subject with \(m_i\) sampling times, let \(C_i\) be the fraction of that subject's observations covered. If the target is average subject-level coverage, estimate it by \(\bar{C}=n^{-1}\sum_i C_i\). Each \(C_i\) lies between 0 and 1, and the subject—not each time point—is the independent unit for uncertainty about this average.
For independent Bernoulli coverage outcomes with common coverage \(p\), the approximate standard error of a sample proportion based on \(N\) independent outcomes is \(\sqrt{p(1-p)/N}\). A rough normal-approximation half-width is \(z_{1-\alpha/2}\sqrt{p(1-p)/N}\). Repeated outcomes within subjects are not independent Bernoulli trials. With equal cluster size \(m\) and exchangeable within-subject correlation \(\rho\), the variance of the mean across all \(nm\) outcomes is inflated relative to independence by the design effect \(D=1+(m-1)\rho\). This approximation assumes the same marginal coverage probability and the same within-subject correlation for every pair of times in a subject; it may be poor for irregular sampling or time-varying dependence.
Worked Example: Planning Coverage Precision With Repeated Times
A hypothetical study targets average coverage of 90% and wants an approximate 95% confidence-interval half-width of 5 percentage points. Each subject contributes four scheduled sampling times. Assume equal cluster size, independent subjects, and an estimated within-subject correlation of \(\rho=0.50\) among the four binary coverage indicators.
If all outcomes were independent, the approximate required number of observations would be
The repeated outcomes have design effect \(D=1+(4-1)(0.50)=2.5\). For \(n\) subjects with four times each, the approximate half-width is \(1.96\sqrt{(0.90)(0.10)D/(4n)}\). Solving for \(n\) gives
Round up to 87 subjects, or 348 scheduled observations. The independent-observation calculation alone would suggest 139 observations, but that would not account for their clustering. The result is an approximate planning value, not a guarantee that a realized interval will have a 5-point half-width. If the correlation is higher than assumed, or coverage varies substantially by time, more subjects may be needed.
How Sampling Times Affect Error Precision
More times per subject can make that subject's profile summary more stable, particularly when the intended metric averages over a time course. They can also improve coverage assessment across prespecified time regions. Their value depends on where the samples occur and on how errors vary over time; taking many samples at nearly identical times may add little information about performance in other parts of the profile.
For example, to plan precision for RMSE under equal numbers of times per subject, define \(Q_i=m^{-1}\sum_{j=1}^{m}e_{ij}^{2}\), the subject's mean squared error. The record-averaged RMSE is \(\widehat{R}=\sqrt{\bar{Q}}\), where \(\bar{Q}=n^{-1}\sum_i Q_i\). If the subject contributions are independent and \(\bar{Q}\) is sufficiently far from zero for a local approximation to be reasonable, the delta method gives
where \(s_Q\) is the sample standard deviation of the \(Q_i\). This approximation shows how precision depends on between-subject variation in squared errors as well as the number of subjects. It does not claim that increasing the number of times reduces that variation by a fixed amount: the effect depends on within-subject dependence and the sampling schedule.
Worked Example: Planning Precision for RMSE
In a hypothetical pilot with 40 subjects, suppose the mean subject-level mean squared error is \(\bar{Q}=100\ (\text{mg/L})^2\), and the standard deviation of the \(Q_i\) values is \(s_Q=80\ (\text{mg/L})^2\). Thus the estimated RMSE is \(\sqrt{100}=10\) mg/L. Use the delta-method approximation to plan a 95% interval with half-width about 1 mg/L.
The approximate standard error at \(n=40\) is
Using \(1.96\) as the planning critical value, the approximate sample size for half-width \(h=1\) mg/L is
The approximation suggests at least 62 subjects. It relies on the pilot estimate of \(s_Q\), the chosen subject-level weighting, and the delta-method approximation. A subject-level bootstrap can be used to check uncertainty for the nonlinear RMSE, especially when squared errors are skewed or a few subjects have unusually large errors.
Derivation, Diagnostics, and Alternatives
The mean-bias precision formula follows directly from independence. If the subject summaries have variance \(\sigma_B^2\), then
The middle equality uses independence, which makes the covariances between different subjects zero. Taking the square root gives \(SE(\bar{B})=\sigma_B/\sqrt{n}\); replacing the unknown \(\sigma_B\) by \(s_B\) gives the estimated standard error used in the confidence interval. The same argument does not permit treating repeated observations within a subject as independent, because their covariances need not be zero.
The principal alternatives differ in how they represent dependence and the target estimand:
| Approach | What it uses | Main caution |
|---|---|---|
| Subject-level summaries and a \(t\) interval | One contribution per independent subject | Requires a clearly defined subject-level estimand; small-sample inference can be sensitive to skewness. |
| Cluster-robust variance or a subject-level bootstrap | Record-level calculations while retaining subject clusters | Needs enough independent subjects; a bootstrap must resample whole subjects, not individual records. |
| Observation-level binomial or standard-error formula | Treats each record as independent | Can substantially understate uncertainty when measurements within a subject are correlated. |
Check the assumptions and data structure before trusting a planned precision calculation. Useful checks include the number of independent subjects, distributions of subject-level summaries, variation in observations per subject, missing or unusable scheduled samples, and whether the planned times cover the regions relevant to the intended metric. For coverage, inspect whether the target is overall or time-specific coverage; a precise overall estimate can conceal imprecision at an important time. For bias and error, influential subjects can make a normal approximation or a small pilot's variance estimate unreliable.
A common mistake is to count every concentration as a new independent participant. For example, 20 subjects with 10 samples each do not generally provide the same precision for population-level performance as 200 independent subjects. The correction is to account for within-subject dependence and make the number of independent subjects explicit in the plan. Increasing the number of sampling times may improve within-subject summaries, but it cannot substitute without limit for sampling more subjects from the target population.
Precision targets also do not resolve design limitations. If the external sample poorly represents the intended population, a narrow interval describes the wrong target. If a subgroup is important, its precision depends on the number of subjects in that subgroup, not just the total sample size. Planning from a single pilot variance can be fragile, so examine plausible larger variances, correlations, and rates of missing data as sensitivity scenarios. The final interpretation should distinguish the uncertainty of the performance estimate from the question of whether that performance is clinically acceptable.
Check Your Understanding
Answer each question, then compare with the brief solutions.
- A validation has 50 subjects with 6 observations each. What is the primary independent unit for estimating population-level performance, and why?
- A pilot estimates the standard deviation of subject-level bias as 5 mg/L. Using a normal approximation, what sample size targets a 95% half-width of 1.5 mg/L?
- For binary coverage measured once per independent subject, explain why the approximate standard error is largest when the target coverage is 0.50 rather than 0.90, for a fixed sample size.
- What should be resampled in a bootstrap for an RMSE calculated from repeated observations within subjects?
Solutions. (1) The subject is the independent unit because measurements from the same person can be correlated; 300 records do not constitute 300 independent subjects. (2) \(n\approx(1.96(5)/1.5)^2=(6.5333)^2\approx42.68\), so round up to 43 subjects. (3) The Bernoulli variance is \(p(1-p)\), which is 0.25 at \(p=0.50\) and 0.09 at \(p=0.90\); thus the standard error is larger at 0.50 for the same \(n\). (4) Resample whole subjects, carrying along all of each selected subject's observations, so the within-subject dependence is retained.