Why Diagnose a Simulated Covariate Sample?
A covariate sampling procedure can run without errors and still produce a virtual population that differs from its intended target. A small discrepancy may be ordinary finite-sample variation; a persistent or consequential one may distort a population prediction. Diagnosis therefore asks two related questions: does the sample represent the target distribution adequately, and could any mismatch matter for the quantity being simulated?
The target distribution is specified when the target covariate population is defined. The simulated records are a finite sample from that target, generated by a method such as empirical resampling, parametric sampling, joint sampling, or stratified sampling. The diagnostics in this tutorial compare those records with the target; they do not replace the choice or justification of the target itself.
The comparison should match the sampling design. If individuals are drawn independently from a target distribution, finite-sample variation provides a useful reference for interpreting observed differences. If records are sampled without replacement from a finite target dataset, or if sampling is stratified, the variability of the diagnostic can differ from that under independent sampling. In those cases, use the design that generated the sample when judging how unusual a discrepancy is.
Marginal Checks: Proportions and Continuous Summaries
For a categorical covariate with target category probabilities \(p_1,\ldots,p_K\), compare each target probability with its simulated proportion \(\hat{p}_k=n_k/n\), where \(n_k\) is the number of simulated records in category \(k\). Report the difference in percentage points as well as the proportions themselves. A category with an important clinical interpretation can merit attention even if its overall contribution to a summary appears small.
For an independently sampled categorical variable, the variance of the simulated proportion for category \(k\) is \(p_k(1-p_k)/n\). This follows by writing the category indicator for each record as a Bernoulli random variable with variance \(p_k(1-p_k)\); the sample proportion is the average of \(n\) independent indicators, so its variance is \(p_k(1-p_k)/n\). Its standard deviation is therefore \(\sqrt{p_k(1-p_k)/n}\). This is a diagnostic reference, not a guarantee that every category will lie within a fixed number of standard errors. The proportions are also dependent across categories because they sum to one.
For a continuous covariate, compare the sample mean \(\bar{x}\) with the target mean \(\mu\), and the sample standard deviation \(s\) with the target standard deviation \(\sigma\), where these target quantities are defined. Also inspect quantiles and the distribution’s shape. Similar means and standard deviations do not rule out different tails, skewness, or multimodality. When a parametric target distribution is specified, its theoretical quantiles can be compared with sample quantiles; for an empirical target, compare with the corresponding empirical distribution.
Worked Example: Checking a Categorical Covariate
Suppose a hypothetical target population has three renal-function categories with target proportions 0.50, 0.30, and 0.20. A simulated sample of \(n=400\) contains 176, 136, and 88 records in those categories. Compare the sample with the target and assess the size of the differences.
The simulated proportions are \(176/400=0.44\), \(136/400=0.34\), and \(88/400=0.22\). Subtracting target from simulated proportions gives differences of \(-0.06\), \(+0.04\), and \(+0.02\), respectively: the first category is 6 percentage points below target, and the other two are 4 and 2 percentage points above target.
Under independent sampling, the target-based standard deviations of these proportions are:
The observed differences are about \(-2.40\), \(+1.75\), and \(+1.00\) of these standard deviations. The first category’s difference deserves a closer look, but these calculations do not by themselves establish a sampling failure. The categories are dependent, and the interpretation also depends on the design and on whether a 6-percentage-point shift would affect the intended prediction.
Continuous Covariates: Look Beyond the Mean
A numerical summary is useful only if it addresses plausible ways the sample could differ from the target. At minimum, check location and spread; then compare quantiles or plots to reveal distributional features those summaries miss. A histogram, density plot, empirical cumulative distribution, or target-versus-sample quantile plot can show a shifted center, an overly narrow range, an unexpected tail, or a shape mismatch. Axes, units, and the target reference should be explicit.
Worked Example: Comparing a Continuous Covariate with a Normal Target
Suppose body weight in a hypothetical target population follows \(X\sim N(70,100)\), with weight measured in kilograms. A sample of \(n=100\) simulated individuals has \(\bar{x}=72.1\) kg, \(s=9.4\) kg, a 5th percentile of 54.8 kg, and a 95th percentile of 89.6 kg. Compare these summaries with the target.
The sample mean is 2.1 kg above the target mean: \(72.1-70=2.1\) kg. For independent draws from the stated target, the standard deviation of the sample mean is \(\sigma/\sqrt{n}=10/\sqrt{100}=1.00\) kg. Thus the mean difference is 2.10 such standard deviations. This is a useful scale comparison, not proof of a systematic error.
The sample standard deviation is 0.6 kg below the target standard deviation: \(9.4-10=-0.6\) kg. For a normal target, the 5th and 95th percentiles are approximately \(70-1.645(10)=53.55\) kg and \(70+1.645(10)=86.45\) kg. The sample’s reported percentiles are 1.25 kg above the target 5th percentile and 3.15 kg above the target 95th percentile. The upper-tail difference is more noticeable than the small difference in standard deviations suggests.
A target-versus-sample quantile plot would help determine whether the upper tail is consistently shifted or whether this is an isolated feature of a finite sample. The sample summaries alone do not establish the cause. If the simulated weight distribution is used in a prediction with strong weight dependence, evaluate that prediction directly before deciding whether the tail difference is consequential.
Joint Patterns Can Fail When Marginals Match
Marginal comparisons check one covariate at a time. They cannot establish that the simulated combinations of covariates match the target. For two categorical covariates, compare the joint cell proportions in a contingency table. For continuous covariates, a scatterplot or comparison of correlations can reveal changes in association. For mixed covariates, inspect combinations such as a continuous distribution within each category. The appropriate check depends on which covariate combinations influence the simulated model.
In particular, matching each marginal does not imply matching the joint distribution. If a prediction changes across covariate combinations, replacing the target joint distribution with a different dependence pattern can alter the population average even when each covariate’s individual proportions appear correct.
Worked Example: A Joint Mismatch Changes the Average Prediction
Consider two binary covariates in a hypothetical target population: treatment group A or B, and normal or low renal function. The target and simulated joint proportions are shown below. Both samples have 50% in group A and 50% in group B, and 50% with normal and 50% with low renal function.
| Joint covariate pattern | Target proportion | Simulated proportion | Hypothetical predicted probability |
|---|---|---|---|
| A, normal renal function | 0.30 | 0.15 | 0.20 |
| A, low renal function | 0.20 | 0.35 | 0.10 |
| B, normal renal function | 0.20 | 0.35 | 0.10 |
| B, low renal function | 0.30 | 0.15 | 0.05 |
For example, the target proportion in group A is \(0.30+0.20=0.50\), and the simulated proportion is \(0.15+0.35=0.50\). The normal-renal-function proportions are likewise \(0.30+0.20=0.50\) and \(0.15+0.35=0.50\). Yet the joint cell proportions differ substantially.
The target average predicted probability is the sum of each target cell proportion times its cell prediction:
Using the simulated joint proportions gives:
The simulated average is lower by \(0.1075-0.115=-0.0075\), or 0.75 percentage points. This difference is attributable to the changed joint weights in this calculation; the cell predictions were held fixed. Whether a 0.75-percentage-point change matters depends on the simulation’s purpose and decision context, not on a universal threshold.
A Practical Diagnostic Sequence
An effective diagnosis moves from basic checks to the prediction itself. It should identify a mismatch, assess whether the mismatch is compatible with the sampling design, and trace its possible effect rather than treating every difference as equally important.
Record the target distributions, units, subgroup definitions, and whether records were sampled independently, with replacement, without replacement, or by strata.
Compare category proportions and continuous summaries, then inspect plots and quantiles for distributional features hidden by a mean or standard deviation.
Compare relevant cell proportions, associations, or within-group distributions. Avoid assuming that matching marginals guarantees a matching target population.
Recalculate or compare the target-weighted prediction and the prediction using the simulated distribution, holding other inputs fixed where possible.
Check coding, units, missingness, category definitions, weights, and sampling implementation. State which discrepancy was found and why it was or was not consequential.
Diagnostics, Alternatives, and Limitations
Plots and numerical summaries answer complementary questions. A table of proportion differences is precise for categories but can conceal an unusual ordering or a small subgroup pattern. A histogram makes shape visible but depends on bin choices. Quantile comparisons avoid binning but can be noisy in the extremes. A joint table is interpretable for a modest number of categories, while high-dimensional covariate combinations can be sparse. In that setting, prioritize combinations with substantive relevance to predictions and inspect lower-dimensional conditional summaries as well.
A formal goodness-of-fit test is an alternative when a test statistic and reference distribution are appropriate, but a small p-value is not a measure of practical importance, and a large p-value does not prove that the distributions are equal. With a large simulated sample, minor differences can be flagged; with a small one, important differences may be hard to detect. For workflow diagnosis, effect sizes, plots, design-aware sampling variability, and consequences for the target prediction are often more informative than a single test result.
Other useful alternatives include reweighting simulated records to the target proportions or regenerating the sample using a corrected sampling design. Reweighting can be useful when target weights are known and all relevant target patterns are represented, but extreme or undefined weights can make results unstable or impossible to recover. Regeneration is preferable when the sampling implementation is wrong. Neither approach substitutes for checking the resulting covariate distribution and downstream predictions.
A discrepancy can arise from ordinary finite-sample variation, incorrect target weights, coding or unit errors, an unsuitable sampling model, or a failure to preserve dependence. Diagnostics identify evidence to investigate; they do not automatically identify the cause. Nor can they establish that the chosen target population is scientifically appropriate. That question belongs to the definition of the target and the purpose of the simulation.
Common Mistake
Check Your Understanding
Answer each question, then compare your reasoning with the solutions below.
- A target category proportion is 0.40, and 250 independently sampled records contain 115 records in that category. What is the simulated proportion and its difference from target?
- A target covariate has mean 50 units and standard deviation 8 units. For 64 independent draws, what is the standard deviation of the sample mean under the target?
- Two binary covariates each have the correct simulated marginal proportions. Does that establish that their joint distribution matches the target? Explain.
- Why should a diagnostic report prediction differences as well as covariate discrepancies?
Solutions: (1) The simulated proportion is \(115/250=0.460\); its difference from target is \(0.460-0.400=0.060\), or 6 percentage points. (2) The standard deviation of the sample mean is \(8/\sqrt{64}=1.00\) unit. (3) No. The covariates can have the correct separate proportions but a different association, producing different joint cell proportions. (4) A covariate mismatch is consequential only in relation to its effect on the quantity of interest; two discrepancies of similar size can have very different effects on a model prediction.