Why Covariate Sampling Needs Its Own Review
A simulation can implement the pharmacokinetic model as intended and still represent the wrong population. The cause may be a category sampled at the wrong rate, a continuous covariate outside its allowed range, missing dependence between covariates, or an eligibility rule applied in the wrong order. Such errors can alter simulated exposure, response, and subgroup summaries even when the structural model is correct.
The review should begin with the population the simulation is meant to represent. Specify the covariates, their target distributions, any relationships between them, their permitted support, and the rules defining inclusion. Then compare simulated records with those specifications. A sample will not reproduce every target proportion or summary exactly: random variation is expected. The task is to identify discrepancies that are too large, impossible under the target, or inconsistent with the intended sampling process.
State the Target and the Assumptions
Write down the target before inspecting the simulated values. For a categorical covariate, record each target category probability. For a continuous covariate, record its distribution or at least the required support and relevant summaries. For related covariates, specify the target joint distribution, conditional distributions, or dependence structure. For eligibility rules, specify whether the simulation draws directly from the eligible population or draws from a broader population and then filters.
Formal comparisons require assumptions as well as a target. In the categorical examples below, simulated records are independent draws, categories are mutually exclusive and exhaustive, and target probabilities are fixed in advance. If the simulation instead uses clustered draws, sampling without replacement from a finite population, or dependent records, a simple independent-draw reference distribution may not apply. A discrepancy statistic can still be useful descriptively, but its usual p-value may not be calibrated.
For a categorical covariate with \(k\) categories, let \(O_j\) be the simulated count in category \(j\), \(n\) the total number of records, and \(p_j\) the specified target probability. The expected count is \(E_j=np_j\). The Pearson discrepancy statistic compares observed and expected counts:
When the target probabilities are fixed, records are independent, and expected counts are sufficiently large for the approximation, this statistic has an approximate chi-square distribution with \(df=k-1\) under the target model. The degrees of freedom arise because the category counts sum to \(n\), so only \(k-1\) count deviations can vary freely.
Checking Category Proportions
A target probability is not a quota that every random sample must match exactly. The Pearson statistic measures the discrepancy relative to the variability expected under the target. A small upper-tail p-value indicates that a discrepancy at least this large would be unusual under the specified sampling model; it does not establish that the simulation code is faulty. Conversely, a non-small p-value does not prove correct implementation. Review the target specification, sampling assumptions, and code as well as the statistic.
Worked Example: Two Fixed Target Proportions
Suppose a hypothetical simulation generates two independent strata with fixed sizes. In stratum A, the target probability of a binary covariate being present is 0.30 and \(n=500\). In stratum B, the target probability is 0.10 and \(n=750\). The simulation produces 180 present values in A and 85 in B. Check both proportions together.
Under the target, the expected present and absent counts are 150 and 350 in A, and 75 and 675 in B. The observed counts are 180 and 320 in A, and 85 and 665 in B. The Pearson discrepancy is:
Substitution and arithmetic give:
There are two independent binomial comparisons, each with one free count deviation, so the combined statistic has \(df=2\) under the fixed target probabilities. The upper-tail p-value is approximately \(0.00656\). The expected counts are all large enough for the chi-square approximation to be reasonable. Under the stated model, this discrepancy would be unusual, so the result warrants checking the specified stratum probabilities, stratum assignment, and randomization implementation. It is not, by itself, proof of a particular coding error.
The degrees of freedom depend on the question. Here the two group sizes and both target probabilities were specified in advance, giving \(df=2\). A test of independence in a two-by-two table with probabilities estimated from the margins would be a different procedure and would have a different degrees-of-freedom calculation.
Ranges and Dependencies
A marginal check asks whether each covariate separately resembles its target. It cannot establish that the relationships between covariates are correct. For example, simulated age and body weight might each have plausible means and ranges while their association is missing or has the wrong direction. Compare joint or conditional distributions when dependence is part of the target.
For a categorical pair with a fully specified target joint distribution, the same Pearson formula can compare each joint-cell count with its target expected count. If the joint probabilities are fixed, the degrees of freedom are the number of cells minus one, assuming all cells have positive target probability and expected counts are adequate. This is not the same as testing independence while estimating margins from the observed table.
Worked Example: Checking a Target Joint Distribution
Suppose two categorical covariates have four joint categories with target probabilities 0.24, 0.56, 0.06, and 0.14. These probabilities encode the intended dependence and sum to 1. In a hypothetical sample of 1,000 independent simulated records, the observed joint counts are 260, 540, 40, and 160.
The target expected counts are \(1000(0.24)=240\), \(1000(0.56)=560\), \(1000(0.06)=60\), and \(1000(0.14)=140\). Thus:
The contributions are \(400/240=1.666667\), \(400/560=0.714286\), \(400/60=6.666667\), and \(400/140=2.857143\), giving \(X^2=11.904762\). All expected counts exceed 5. Since the four target joint probabilities are specified in advance, \(df=4-1=3\), and the upper-tail p-value is approximately 0.0077. This result flags an unusual joint-table discrepancy under the specified target; it does not tell which cell or coding step caused it.
Looking only at the separate marginal counts could miss this problem: marginal totals can agree with their targets even when joint combinations are wrong. A review should therefore compare joint counts or conditional proportions when the target includes dependence.
For continuous covariates, distinguish a support check from a distributional check. Support is the set of values that can occur under the target. If age is required to lie between 18 and 75 years, an unrounded simulated value of 17.6 years violates the support and should be investigated. If the target is an unbounded normal distribution, an extreme value is not automatically impossible; evaluate how unusual it is using the target distribution and sample size. Means, standard deviations, quantiles, histograms, and quantile plots each reveal different features, but matching a few summaries does not establish that the full distribution is correct.
Worked Example: A Hard Range Violation
A hypothetical target specifies age in years from a distribution truncated to the inclusive interval 18 to 75. Among 500 simulated records, the smallest unrounded age is 17.6 years and the largest is 74.8 years.
The upper value is within the permitted interval, but \(17.6<18\), so the minimum violates the target support. Because the target assigns zero probability to ages below 18, this is not ordinary sampling variation under the stated model. Check whether truncation was applied, whether the simulation used a different age field than the review, and whether the value was transformed or rounded in a way that obscures the source. If the target were instead an untruncated normal distribution, this deterministic conclusion would not follow; the value would need to be assessed against that distribution.
Sampling Exclusions and the Target Population
Exclusion rules change the population represented by the retained records. If a simulation first samples from a broad source population and then removes ineligible records, the retained covariates follow a conditional distribution. That distribution need not equal the original source distribution. The review should verify both the eligibility logic and whether the requested sample size refers to candidates or eligible records.
For an eligibility probability \(q\) and \(N\) independent candidate draws, the number retained has a binomial distribution with mean \(Nq\) and standard deviation \(\sqrt{Nq(1-q)}\). This gives a reference for the retained count when the assumptions hold. It does not verify the correctness of the individual eligibility decisions or the covariate distribution among retained records.
Worked Example: Checking an Eligibility Filter
A hypothetical source population has a known eligibility probability of 0.88. A program generates 500 independent candidates and retains records meeting the specified criteria. It retains 438. The intended deliverable, however, is a sample of 500 eligible individuals.
The expected number retained from 500 candidates is \(500(0.88)=440\). The standard deviation is:
The observed count is two below expectation, or \((438-440)/7.266=-0.275\) standard deviations. That count is entirely plausible under the stated candidate-sampling process. However, 438 eligible records do not meet a request for 500 eligible records. The program should continue drawing candidates until 500 eligible records are obtained, or sample directly from the conditional eligible population. Simply labeling 438 records as a 500-person eligible sample would be a design error.
Also inspect exclusion reasons and retained covariates. A plausible total count does not show that each record was filtered correctly, that eligibility was applied to the intended variable and time point, or that the retained distribution matches the target conditional population.
Diagnostics, Alternatives, and Limitations
Use checks that correspond to the target feature, and combine them rather than relying on one pass/fail statistic. A practical review can include:
- Observed and expected category counts, with the target probabilities stated explicitly.
- Minimum, maximum, and counts outside each required support or eligibility range.
- Histograms or quantile summaries for continuous covariates, compared with target values or reference simulations.
- Joint tables, conditional summaries, or correlation measures when covariates are dependent.
- Retained sample size and exclusion counts, with reasons and the intended denominator reported.
The Pearson chi-square approximation is convenient when expected counts are adequate and the independent multinomial model applies. With sparse expected counts, consider combining categories only when substantively justified, using an exact method, or obtaining a reference distribution by simulation from the stated target. For continuous distributions, graphical comparisons and target-based simulation envelopes can be more informative than a single omnibus statistic. A test focused on one feature can miss errors in another: matching a mean does not certify the tails, and matching marginal distributions does not certify dependence.
These checks also have limitations. Large samples can make negligible differences statistically detectable; small samples can fail to flag material differences. A p-value summarizes compatibility with a specified sampling model, not the probability that the model or code is correct. Repeatedly testing many covariates and subgroups increases the chance of at least one apparently unusual result. Interpret findings in context, examine the underlying records and implementation, and distinguish a deterministic violation from a stochastic discrepancy.
A Review Sequence
Record marginal and joint distributions, support, eligibility criteria, and whether the requested sample size counts candidates or eligible records.
Inspect missing values, category labels, units, ranges, and the correct covariate time point before summarizing distributions.
Use counts and expected counts for categories, suitable summaries or plots for continuous values, and joint checks for specified dependencies.
Check independence, fixed targets, expected counts, and exclusion logic before using a formal reference distribution or concluding that a result is unusual.
Investigate flagged records and relevant code paths, then report the target, simulated result, check used, and remaining uncertainty.
Check Your Understanding
Answer each question, then compare with the solutions below.
- A fixed target assigns probabilities 0.25 and 0.75 to two categories. In 200 independent records, what are the expected counts and the degrees of freedom for a Pearson goodness-of-fit check?
- Why can matching the separate margins of two covariates fail to show that their joint sampling is correct?
- A target explicitly restricts a covariate to values from 0 to 1. What does an unrounded simulated value of 1.03 imply?
- A program draws 1,000 candidates, of whom 80% are expected to meet eligibility criteria. The requested output is 1,000 eligible records. Is retaining approximately 800 records sufficient?
Solutions. (1) The expected counts are \(200(0.25)=50\) and \(200(0.75)=150\). With two fixed target probabilities summing to 1, \(df=2-1=1\). (2) Different joint distributions can have the same marginal distributions; compare joint cells or conditional distributions when dependence is specified. (3) The value is outside the target support and is a hard violation, assuming the target and recorded value are on the same scale and no transformation or rounding explains it. (4) No. About 800 eligible records is the expected yield from 1,000 candidates, but the requested output is 1,000 eligible records; the program must obtain additional eligible records or sample directly from the eligible population.