Why the Replication Scheme Matters
A predictive check compares observed outcomes with outcomes that a fitted model could generate. In a generalized linear mixed model, those generated outcomes depend both on the outcome distribution within a cluster and on the cluster’s random effect. A replication can keep the fitted effects for the observed clusters fixed, or draw new effects from the fitted random-effects distribution. Those choices produce different reference distributions, so a check can appear reassuring under one scheme and concerning under the other.
The distinction is about the question being asked. Holding fitted cluster effects fixed emphasizes whether the model can reproduce outcomes within these clusters, given their fitted latent levels. Drawing new effects also asks whether the model’s assumed distribution of cluster effects can reproduce variation across clusters. Neither scheme is universally preferable; the diagnostic target determines the choice.
Model and Replication Definitions
Let \(Y_{ij}\) be the outcome for observation \(i\) in cluster \(j\), with covariates \(x_{ij}\) and random effect \(b_j\). A GLMM specifies a conditional outcome distribution and a link model such as
Here \(\mathcal{F}\) is the chosen outcome family, \(g\) is its link, \(\beta\) contains fixed-effect coefficients, \(z_{ij}\) describes the random-effect contribution, and \(D\) is the random-effects covariance matrix. The usual model assumptions include the specified conditional outcome distribution, the specified link and linear predictor, and independent random effects across clusters drawn from the stated distribution. Conditional independence of observations given the random effects is also assumed unless the model explicitly includes additional dependence.
After fitting, write the estimated parameters as \(\hat\beta\) and \(\hat D\), and let \(\hat b_j\) denote a fitted cluster effect, often an empirical Bayes or conditional mode estimate. In a basic plug-in check, covariates and design quantities are held at their observed values, and simulated outcomes are generated using these fitted quantities.
In either scheme, a discrepancy statistic \(T\) is calculated for the observed data and for each replicated data set. The earlier material on choosing discrepancy statistics is relevant to selecting \(T\); the replication scheme determines which model variation contributes to the comparison. For instance, \(T\) might be the total number of zeros, a cluster-level event rate, or a measure of between-cluster spread.
What Changes Between the Two Schemes?
In a conditional replication, the fitted cluster effect is treated as fixed for the simulation. Conditional on that effect, the outcome is redrawn, so the outcomes can vary from replication to replication even though the latent cluster level does not. This is useful for asking whether the outcome model describes repeated observations at the fitted levels of the clusters in hand.
In an unconditional replication, the cluster effects vary as well as the outcomes. New clusters can therefore have different latent levels, including more extreme effects than those represented by the fitted effects. This makes the check sensitive to the model’s assumed random-effects distribution and to the resulting population-level variability. If the same set of cluster covariates and observation counts is retained, the check generates a new set of latent cluster effects for that design.
The word “conditional” should not be mistaken for conditioning on known true effects. The fitted \(\hat b_j\) are estimated from the observed outcomes, often with shrinkage toward zero. A plug-in conditional check holds those estimates fixed; it does not account fully for uncertainty about them. Likewise, a plug-in unconditional check draws from \(\hat D\) but ordinarily treats \(\hat\beta\) and \(\hat D\) as fixed. These checks are diagnostic comparisons, not automatic tests with universal reference distributions.
Worked Examples
Worked Example: A Poisson Random-Intercept Model
Suppose a hypothetical count model has \(Y_{ij}\mid b_j\sim\text{Poisson}(\lambda_{ij})\), with \(\log(\lambda_{ij})=\log(2)+b_j\), and \(b_j\sim N(0,0.36)\). Consider an observation in a cluster whose fitted effect is \(\hat b_j=0.18\).
For a conditional replication, the fitted mean is held at \(2\exp(0.18)=2.3944\). A new count is generated from a Poisson distribution with that mean, so its conditional variance is also \(2.3944\). Repeating this simulation changes the count, but not the cluster effect used to calculate its mean.
For an unconditional replication, a new effect is drawn from \(N(0,0.36)\). At the population level, the marginal mean is \(E(\lambda)=2E\{\exp(b_j)\}\). Since \(b_j\) is normal with variance \(0.36\), \(E\{\exp(b_j)\}=\exp(0.36/2)\), giving \(2\exp(0.18)=2.3944\). The marginal mean happens to equal the mean for the particular fitted effect \(0.18\); that equality is not generally true for a particular cluster.
The marginal variance is larger than the mean because the random effect adds between-cluster variation. By the law of total variance, \(\operatorname{Var}(Y)=E\{\operatorname{Var}(Y\mid b)\}+\operatorname{Var}\{E(Y\mid b)\}\). Thus
The second term is approximately \(2.484\), using unrounded values in the displayed expression. The conditional variance for the stated fitted effect is approximately \(2.394\), while the unconditional marginal variance is approximately \(4.879\). Thus a tail-count or spread check can look adequate conditionally but show too little variation when clusters are regenerated.
Worked Example: Holding Four Fitted Cluster Effects Fixed
Consider a hypothetical Poisson random-intercept fit with conditional mean \(\lambda_j=1.5\exp(b_j)\). For four observed clusters, suppose the fitted effects are \(-0.4,\ 0,\ 0.2,\ 0.5\). The observed counts are \(0,\ 1,\ 3,\ 4\), respectively. These values are illustrative and are not from a real study.
For conditional replications, the fitted means are \(1.5\exp(-0.4)\approx1.0055\), \(1.5\), \(1.5\exp(0.2)\approx1.8321\), and \(1.5\exp(0.5)\approx2.4731\). Their sum is approximately \(6.8107\), so the expected replicated total across these four clusters is \(6.8107\). The observed total is \(0+1+3+4=8\). A single difference of about \(1.189\) is not enough by itself to establish poor fit; the useful comparison is the observed discrepancy against its distribution across many conditional replications.
For an unconditional comparison, suppose the fitted random-intercept variance is \(0.36\). The expected total for four newly drawn effects is \(4(1.5)\exp(0.18)\approx7.183\). This is a population-average expectation under the fitted model, rather than the sum of means based on the four fixed fitted effects. The contrast between \(6.811\) and \(7.183\) illustrates that the two schemes need not have the same expected statistic for the observed clusters. It does not imply that either expectation is the correct expected total for every possible set of clusters.
Worked Example: A Binary Outcome and Between-Cluster Spread
Suppose a hypothetical logistic random-intercept model has \(\operatorname{logit}\{P(Y_j=1\mid b_j)\}=b_j\), and three observed clusters have fitted effects \(-1,\ 0,\ 1\). There is one binary observation per cluster. Under conditional replication, their fitted event probabilities are \(\operatorname{logit}^{-1}(-1)\approx0.2689\), \(0.5\), and \(\operatorname{logit}^{-1}(1)\approx0.7311\). The expected total number of events is therefore \(0.2689+0.5+0.7311=1.500\).
The conditional variance of that total is the sum of the three Bernoulli variances: \(0.2689(1-0.2689)+0.5(1-0.5)+0.7311(1-0.7311)\approx0.6432\). For an unconditional replication with a symmetric, mean-zero random-intercept distribution, the marginal event probability is \(0.5\): symmetry gives \(\operatorname{logit}^{-1}(b)+\operatorname{logit}^{-1}(-b)=1\). With independently redrawn effects for the three clusters, the expected event total remains \(1.500\), but its variance is \(3(0.5)(0.5)=0.750\). The mean alone does not reveal the distinction; the variability of replicated totals does.
Choosing and Interpreting a Check
The choice should follow the target of the discrepancy. A statistic about within-cluster residual patterns, conditional event rates, or counts given the fitted cluster levels is naturally examined with conditional replications. A statistic about the distribution of cluster totals, the spread of cluster-specific rates, or the frequency of unusually extreme clusters calls for unconditional replications, because the random effects themselves must vary for those features to be represented.
| Replication scheme | Held fixed | Most directly informative about | Important limitation |
|---|---|---|---|
| Conditional | Observed design and fitted cluster effects | Outcome variation around the fitted levels of observed clusters | Can mask inadequate random-effects variation; fitted effects use the observed outcomes |
| Unconditional | Observed design; fitted parameter estimates in a plug-in check | Outcome variation plus variation generated by the random-effects distribution | Combines within-cluster and between-cluster features, so a discrepancy may not identify its source |
A useful diagnostic practice is to compare both schemes when the scientific question involves both levels. If conditional replications reproduce within-cluster patterns but unconditional replications produce too little spread across clusters, the random-effects distribution or its estimated variance deserves attention. If both schemes miss a conditional outcome feature, the outcome family, link, fixed-effects structure, or other conditional assumptions may be inadequate. These patterns guide investigation; they do not uniquely identify a cause.
Other approaches answer different questions. Residual plots examine patterns in observed residuals; they do not generate a reference distribution of replicated data. A conditional replication is closer to a model-based check of outcomes at fitted cluster levels, while an unconditional replication adds variation across generated clusters. Neither should be confused with a held-out-cluster evaluation: both schemes described here use the fitted model and, for conditional checks, the observed clusters’ fitted effects. In-sample agreement can be optimistic because those effects were estimated using the same outcomes being checked.
Diagnostics, Assumptions, and Common Pitfalls
Before interpreting a check, verify that the simulation follows the fitted model: use the observed design if that is the target, generate each outcome from the stated conditional family, and draw random effects from the fitted distribution only for the unconditional scheme. Then examine whether the discrepancy is sensitive to cluster size, covariates, or a few extreme clusters. For sparse outcomes, a small number of generated events can make a discrepancy distribution discrete and variable; this is a reason to inspect the distribution, not to overinterpret a single simulated value.
An apparent mismatch can also reflect parameter uncertainty or a poorly estimated random-effects distribution. Plug-in replications omit parameter-estimation uncertainty, and \(\hat b_j\) can be especially uncertain for clusters with little information. Conditional checks based on shrunken estimates may underrepresent genuine differences between clusters. Unconditional checks depend on the assumed distribution \(N(0,\hat D)\); they cannot verify that distribution merely by generating from it.
- Mistake: “Unconditional” means the fixed effects and all parameters are redrawn. In the plug-in definition here, \(\hat\beta\) and \(\hat D\) stay fixed. Only new random effects and outcomes are generated.
- Mistake: Conditional replication tests the random-effects distribution just as directly. Holding \(\hat b_j\) fixed removes new random-effect variation from the replication. It is therefore less direct for checking between-cluster spread.
- Mistake: A close match proves the model is correct. A check assesses the chosen discrepancy and replication scheme. Other discrepancies or assumptions may still fail.
- Mistake: Compare only the replicated means. As the binary example shows, two schemes can have the same expected total but different variances and tails.
Check Your Understanding
Answer each question, then compare your reasoning with the solutions.
- A fitted random-intercept model has three observed clusters. Which quantities are newly generated in a conditional replication, and which are newly generated in an unconditional replication?
- In a Poisson random-intercept model, why can the unconditional variance exceed the unconditional mean even when each conditional distribution is Poisson?
- A conditional check matches the observed within-cluster residual pattern, but unconditional replications show too little between-cluster spread. What model feature should be investigated?
- For the logistic example above, verify the conditional expected event total from the three fitted probabilities.
Solutions. (1) In a conditional replication, outcomes are redrawn while the observed design and fitted effects \(\hat b_j\) are held fixed. In an unconditional replication, new effects are drawn from \(N(0,\hat D)\), then outcomes are generated conditional on those effects; the design remains fixed in this setup. (2) The law of total variance gives \(\operatorname{Var}(Y)=E\{\operatorname{Var}(Y\mid b)\}+\operatorname{Var}\{E(Y\mid b)\}\). The second term is nonnegative and represents variation in conditional means across effects. (3) Investigate the random-effects distribution and estimated random-effects variance, while recognizing that other model features may also affect the discrepancy. (4) The sum is \(0.2689+0.5000+0.7311=1.5000\) expected events.