Where Double Counting Enters a Simulation
A simulated outcome can vary for several reasons: subjects differ, occasions may differ within a subject, and measurements contain residual error. In a fitted nonlinear mixed-effects model, these sources are represented through model parameters and random effects. A simulation design can accidentally introduce the same source twice—for example, by resampling observed subject outcomes, which already reflect subject differences, and then adding newly drawn subject effects.
The key question is not simply whether a design uses both resampling and random effects. It is what the resampled object contains. Resampling covariates and then drawing random effects can be appropriate: the covariates describe the subjects, while the newly generated effects represent their unobserved differences. Resampling outcomes is different. Those outcomes already carry the realized effects and measurement variation that contributed to them.
For orientation, a general observation from subject \(i\), occasion \(k\), and measurement \(j\) can be written as
Here \(\theta\) denotes fixed model parameters, \(\eta_i\) subject-level random effects, \(\kappa_{ik}\) occasion-level random effects when included, \(x_{ikj}\) the relevant covariates and design information, and \(\epsilon_{ikj}\) residual error. The exact structure of \(f\) depends on the model. In general, these components need not add linearly to the outcome; the additive examples below are deliberately simple models that make the accounting transparent.
A simulation should match its target. To generate new subjects from a fitted population model, one typically supplies or samples their covariates and generates random effects and residuals according to the model. To make a subject-level bootstrap replicate, one resamples observed subjects and their data as clusters; the sampled outcomes already include the realized effects and errors. Combining these designs without accounting for what is retained can duplicate variation.
Trace the Source Before Drawing Again
A useful audit labels each simulated quantity by its origin. A resampled subject record may contain covariates, observed outcomes, and the consequences of the subject’s realized random effects and residual errors. A resampled covariate row, by contrast, contains covariates but does not by itself contain a random-effect realization or an outcome residual. Do not treat these objects as interchangeable.
The following distinctions are especially important:
- Resample outcome clusters: resampling all records for a subject retains that subject’s observed outcome pattern and its within-subject dependence. New subject effects should not automatically be added to those outcomes.
- Resample covariates only: this does not retain observed random effects or residuals. Drawing new random effects and residual errors can be part of a model-based simulation of new outcomes.
- Resample fitted subject effects: empirical Bayes estimates are shrunken, data-informed estimates, not population draws. The earlier material on why empirical Bayes estimates are not population draws is relevant: resampling them does not generally reproduce the population random-effect distribution.
- Resample parameter estimates: this concerns uncertainty about fitted parameters, not the same thing as generating subject-to-subject variation. A parameter draw and a subject random-effect draw have different roles and should not be substituted for one another.
Worked Example: Resampled Occasion Records Plus a New Occasion Effect
Consider a hypothetical additive model with a subject effect \(B_i\), an occasion effect \(C_{ik}\), and residual error \(E_{ikj}\):
Assume independent, mean-zero components with variances \(\operatorname{Var}(B_i)=36\), \(\operatorname{Var}(C_{ik})=16\), and \(\operatorname{Var}(E_{ikj})=4\). Two records are from the same subject and the same occasion, so they share \(B_i\) and \(C_{ik}\), while their residual errors are independent. These assumptions imply an original record variance of \(36+16+4=56\) and a covariance of \(36+16=52\) for the pair.
Now suppose the analyst resamples these occasion records as observed outcomes, so the shared subject and occasion variation is already present, and then adds a new independent occasion draw \(C^\ast\) to each record. The new draw has variance 16 and is independent across the two records. For either record, the variance becomes
The pair still shares the original subject and occasion components, so its covariance remains
Thus the correlation is \(52/72\approx0.7222\). The extra independent occasion draws increase each record’s variance but do not add shared covariance. This is the resampling-plus-extra-draw design.
A different design would replace the original shared occasion effect with independent occasion effects, rather than retain it and add more effects. Then covariance would be only \(36\), each variance would be \(36+16+4=56\), and correlation would be \(36/56\approx0.6429\). That is not the design just calculated: the two calculations differ because one retains the shared original occasion effect and the other removes it.
A Short Variance and Covariance Justification
The calculations above follow from two elementary rules. For independent, mean-zero random variables, the variance of a sum is the sum of the variances. For two records, a component shared by both contributes its variance to their covariance; independent record-specific components contribute zero covariance. In the example, the extra occasion draws are independent of each other and of the resampled records, so they add 16 to each marginal variance and nothing to the covariance.
More generally, suppose two records share a subject effect \(B_i\) with variance \(\omega^2\), and each has an independent residual error with variance \(\sigma^2\). Their original variances are \(\omega^2+\sigma^2\), and their covariance is \(\omega^2\). If a second independent subject effect with variance \(\omega^2\) is added once per record, the result depends on the level at which it is drawn:
- If one new draw is shared across all records of that simulated subject, both marginal variances and their covariance increase by \(\omega^2\). The subject heterogeneity has effectively been included twice.
- If a new independent draw is made for each record, each marginal variance increases by \(\omega^2\), but the covariance does not. The intended within-subject dependence is then altered as well as the marginal variability.
This is why identifying the resampling unit and the draw level matters. “Add a random effect” is incomplete as a design description: specify which observations share each draw and whether the resampled object already contains that source.
Worked Example: Resampling Subjects and Redrawing Subject Effects
Suppose a hypothetical study has two measurements per subject, generated by \(Y_{ij}=\mu+B_i+E_{ij}\), where the subject effect has variance 25 and independent residual errors have variance 9. Assume, for this calculation, that the resampled subject outcomes carry subject-effect variation with variance 25 and residual variance 9. The intended marginal variance is therefore \(25+9=34\), and the covariance between two measurements on the same subject is 25.
An analyst resamples whole observed subject clusters, retaining each cluster’s outcomes, and then adds a new subject effect \(B_i^\ast\) with variance 25, using the same new draw for both records of that simulated subject. The resampled records already carry the original subject heterogeneity; the additional shared draw duplicates it. Each resulting measurement has variance
The covariance between the pair is \(25+25=50\): the original shared subject variation and the newly added shared subject variation both contribute. The resulting correlation is \(50/59\approx0.8475\), compared with the intended \(25/34\approx0.7353\). Under the stated assumptions, both the subject-level spread and the within-subject correlation have changed.
If the goal were instead to simulate new subjects from the fitted model, a suitable design would start with their covariates (or a specified covariate sample), draw one subject effect per new subject, and generate new residual errors for each measurement. It would not begin with outcome clusters that already contain realized subject effects and then add another subject effect.
The variance calculation is an idealized illustration. In an actual finite data set, the empirical variance of resampled observed subject effects and errors will not necessarily equal the assumed population variances exactly. The principle still holds: the resampled outcomes retain their observed variability, while the new draw contributes additional variability.
Resampling Residual Variation Can Also Duplicate a Source
Double counting is not limited to random effects. Suppose a resampled outcome contains residual noise, and the analyst then adds another residual error to it. The new residual increases marginal variance even though the first residual is still present in the resampled value. If residuals are record-specific, the added errors do not restore or preserve the original residual realization; they add another independent contribution.
Worked Example: Adding Residual Error to Resampled Outcomes
Take a hypothetical model \(Y_{ij}=\mu+B_i+E_{ij}\) with subject-effect variance 16 and residual variance 9. Two measurements from the same subject share \(B_i\), and their original residual errors are independent. Each observed record has variance \(16+9=25\), the pair’s covariance is 16, and their correlation is \(16/25=0.6400\).
Suppose observed outcomes are resampled and a fresh independent residual error with variance 9 is then added to every resampled record. The resampled outcomes still contain their original residual contribution, so the new marginal variance is \(16+9+9=34\). The pair continues to share only the subject effect, so the covariance is 16. The resulting correlation is \(16/34\approx0.4706\).
The added error has not made the original error disappear. If the goal is to simulate new outcomes from the model, generate them from the model using the appropriate subject effects and one residual draw per measurement. If the goal is to bootstrap observed outcomes, resample the outcomes as intended and do not append another residual draw unless the procedure explicitly removes the carried residual component first.
Choosing and Checking a Simulation Design
A model-based population simulation and an outcome bootstrap answer different questions. A model-based simulation generates new outcomes under the fitted model; it is suited to predicting new subjects or records under specified covariates and design conditions. A subject-cluster bootstrap resamples observed subjects and their records; it carries observed outcomes and their dependence into each replicate. Neither is universally preferable. The choice follows from the target and the information the resampled data retain.
| Design | What is sampled or drawn? | Double-counting risk to check |
|---|---|---|
| Model-based new-subject simulation | Covariates, then random effects and residuals as required by the model | Do not also retain outcome-level variability from observed records for those same new outcomes. |
| Subject-cluster outcome bootstrap | Whole observed subject clusters | Do not add a second subject effect or residual error to retained outcomes without removing the original source. |
| Covariate resampling with model-generated outcomes | Covariates only, followed by model-based effects and errors | Confirm that the resampled object is covariates only, not observed outcomes or effect estimates. |
Before running a simulation, write down the origin of each component and the level at which it is shared. Then check whether the intended marginal variances and dependence are consistent with the design. For a simple additive model, compare the variance and covariance implied by the data-generating equation with the target. For a nonlinear model, the outcome variance may not be a simple sum, but the same audit remains useful: identify which random inputs are present in each simulated record and which observations share them.
Useful diagnostics include inspecting simulated within-subject correlations and marginal spread, comparing them with the intended model-based or bootstrap target, and tracing the simulation code from input object to final outcome. A suspiciously broad distribution is not proof of double counting: it could also reflect the intended target, a different covariate distribution, or model misspecification. Likewise, a plausible marginal spread does not guarantee that dependence is correct. Review both what varies and how that variation is shared.
Common Pitfall and Limitations
Wrong reasoning: “The resampled subjects need random effects, so I will draw a new effect for every resampled subject.” This is appropriate only if the resampled objects do not already contain outcomes carrying their realized subject effects. If the resampled objects are observed outcome clusters, that reasoning adds a new effect on top of the effects already reflected in those outcomes.
Correction: state the target first, then identify the resampled object. For a new-population simulation, resample or specify covariates and generate one set of random effects and errors from the model. For an outcome bootstrap, resample outcome clusters and retain their observed variation. If a hybrid design is needed, explicitly state which source is preserved, removed, or replaced; do not assume that adding a draw replaces the old source.
These calculations rely on clearly specified independence and sharing assumptions. Real NLME models can have correlated random effects, nonconstant residual variance, nonlinear transformations, and covariate-dependent variability. In those settings, variance components may not add in the simple way shown here. The bookkeeping principle still applies, but the implications should be evaluated under the actual model rather than inferred from an additive approximation.
Check Your Understanding
For each question, identify the source accounting and compare it with the stated target.
- A resampled outcome contains a subject effect with variance 10 and a residual with variance 4. The analyst adds a new independent residual with variance 4. What is the new marginal variance, and has the subject covariance changed?
- In the occasion example, why is covariance 52 for resampling records and adding independent occasion draws, rather than 36?
- An analyst resamples covariate rows only, then draws one subject effect per new subject and residual errors per measurement. Is that necessarily double counting? Explain.
- Two records share a subject effect with variance 9 and each has an independent residual with variance 7. The analyst adds a new subject effect with variance 9, shared across both records. Calculate the resulting marginal variance, covariance, and correlation.
Solutions. (1) The new marginal variance is \(10+4+4=18\). The subject covariance remains 10, because the added residuals are independent across records and do not alter the shared subject effect. (2) The resampled records retain both the shared subject effect (variance 36) and the shared original occasion effect (variance 16), so covariance is \(36+16=52\). The newly added occasion draws are independent per record and add no covariance. (3) No. Covariate rows alone do not carry observed outcomes or their realized random effects and residuals. Generating effects and errors can therefore be appropriate for new outcomes, provided the design matches the intended target and the covariates are sampled as intended. (4) Each record’s variance is \(9+7+9=25\). The pair’s covariance is \(9+9=18\), because both the original and new subject effects are shared. The correlation is \(18/25=0.7200\).