Diagnosing Overfit In Small Virtual Populations

Why a Small Virtual Population Can Match Too Well

A virtual population is a collection of mechanistic parameter vectors representing variation across patients. When the population is small and must match many targets, selection or tuning can produce an apparently excellent fit that depends on a narrow set of parameter values or one unusual virtual patient. Such a fit may reproduce the chosen summaries while representing the intended patient variability poorly.

Here, overfit means that the construction has been tuned to match the specified targets so closely that the match is fragile or achieved at the expense of plausible patient-to-patient variation. A small number of targets alone does not prove overfitting, and a large target count alone does not prove it either: targets can be redundant, and their difficulty depends on the candidate population and construction procedure. The diagnosis therefore combines target residuals, parameter diversity, and sensitivity to individual virtual patients.

The earlier material on optimization-based virtual population selection and matching population summaries supplies the context: each virtual patient has a parameter vector, and each target is a summary computed from the selected population. The diagnostics below assess whether the resulting match is dependable; they do not establish that the targets themselves are correct or that the model is clinically predictive.

Target Residuals on Comparable Scales

A raw residual has the units of its target: a clearance residual might be in L/h, whereas a response residual might be in percentage points. Scaled residuals make discrepancies easier to compare across targets, but only when their scales are chosen and justified. A scale of zero is invalid, and choosing scales after seeing the fit can make an inadequate match look acceptable.

One descriptive summary is the mean squared scaled residual:

Here \(m\) is the number of targets included. \(Q\) is a fit summary, not a p-value, a probability, or a universal pass/fail test. It can conceal a single large discrepancy, so inspect the individual \(r_j\) values as well. If the construction uses target ranges, assess each summary against its stated range rather than treating the range midpoint as an exact target without justification.

Check Whether Parameter Variation Has Collapsed

A virtual population is intended to represent a range of plausible patients, not merely to reproduce a few aggregate summaries. Compare the spread of important parameters in the selected population with the spread in the eligible candidate population or another scientifically justified reference. Use the same parameter definitions and units in both groups. For parameters with different units or scales, use dimensionless summaries such as a coefficient of variation only when the parameter is positive and a ratio-based spread is meaningful; otherwise, standardize by a defensible reference scale.

For a parameter \(x\) measured across \(n\) virtual patients, the sample standard deviation \(s\) summarizes variation around the sample mean \(\bar{x}\). It should be read alongside the parameter's units and distribution: a standard deviation alone does not describe skewness or unusual values. For multiple parameters, inspect each scientifically important parameter and the joint pattern. A population can retain variation in every marginal parameter yet have an implausibly restricted combination of parameters.

This is a diagnostic comparison, not a rule that selected spread must equal candidate spread. Selection is supposed to change the population when targets or feasibility constraints warrant it. Concern arises when fit improves dramatically while spread contracts sharply without a scientific reason, especially when many parameters were adjusted to meet a small set of summaries.

Test Sensitivity to Individual Virtual Patients

A full-population summary can be stable-looking while depending heavily on one virtual patient. A direct check is to recalculate each important target after removing one patient at a time, without refitting or replacing that patient. This is a diagnostic perturbation, not a new population-construction procedure. Record how much each target summary changes and whether it crosses its tolerance or leaves its allowed range.

For a target that is the arithmetic mean of values \(x_1,\ldots,x_n\), the change caused by removing patient \(i\) has a simple expression. Since \(n\bar{x}=\sum_{k=1}^{n}x_k\), the mean after removal is \(\bar{x}_{(-i)}=(n\bar{x}-x_i)/(n-1)\). Subtracting the original mean gives:

This derivation shows why a patient far from the mean can have greater influence, and why the same deviation matters more when \(n\) is small. For medians, quantiles, proportions, or nonlinear model outputs, recalculate the summary directly; the mean formula does not apply. Also inspect changes to more than one target, because the same patient may support several target matches.

Combine the Diagnostics and Choose a Response

The diagnostics answer different questions. Target residuals show which specified summaries are mismatched. Parameter-spread comparisons show whether the selected patients retain relevant variation. Leave-one-out checks show whether the match is sensitive to a particular member. No single diagnostic substitutes for the others. For example, residuals may be small while parameter spread collapses, or spread may look reasonable while one target depends on a single patient.

These checks complement, rather than replace, other approaches. Optimization scores efficiently rank populations but can obscure individual residuals or diversity loss. Hard acceptance ranges make requirements transparent but do not guarantee that the accepted population is stable. Where suitable independent information exists, evaluating against targets not used to tune the population can provide a useful check on generalization; however, a withheld summary is not independent evidence if it is derived from the same data or closely duplicates a tuning target.

A practical report should state the targets and scales used, show notable residuals, describe how parameter spread was assessed, and summarize the largest leave-one-out changes. There is no universal cutoff for declaring overfit. Conclusions depend on target quality, parameter meaning, candidate-population design, and the purpose of the virtual population. In particular, passing these diagnostics does not establish coverage of every plausible patient type or validate predictions outside the information used to construct the population.

Common Mistake