Prior Predictive Checks for Random-Effect Variability

Why Check a Variance Prior Predictively?

A prior for a random-effect standard deviation can look reasonable on its own while implying group differences that are difficult to defend in the model’s outcome scale. In a logistic GLMM, for example, a standard deviation on the log-odds scale does not directly describe a typical difference in event probabilities. The effect of that same scale depends on the fixed intercept and other predictors. Prior predictive checking makes those implications visible by simulating group effects and outcomes before using observed data to update the prior.

The check is not a test that declares a prior valid or invalid. It is a structured way to ask whether the patterns permitted by the prior—including its common and tail behavior—are compatible with substantive knowledge about plausible groups and outcomes. The introductory material on calibrating prior scales from plausible group heterogeneity motivates choosing a scale; prior predictive simulation examines the consequences of that choice in the model itself.

Definition: The prior predictive distribution of outcomes $Y$ is obtained by averaging the sampling model over the prior distribution of all unknown parameters and random effects. For a GLMM with parameters $\theta$ and group effects $b$, it is $$ p(y)=\int p(y\mid b,\theta)\,p(b\mid\theta)\,p(\theta)\,db\,d\theta. $$ A prior predictive check compares simulated values of $b$, model-implied quantities, and outcomes $Y$ with patterns that would be plausible before observing the data.

Model and Simulation Order

For clarity, consider a random-intercept logistic GLMM for binary outcomes $Y_{ij}$, where $i$ indexes units within group $j$. Conditional on the group effect, suppose outcomes are independent Bernoulli variables with

Y_{ij}\mid b_j,\beta_0 \sim \operatorname{Bernoulli}(p_{ij}),\qquad \operatorname{logit}(p_{ij})=\beta_0+b_j,\qquad b_j\mid\sigma_b\sim N(0,\sigma_b^2).

Here $\beta_0$ is the fixed intercept, $b_j$ is the group-specific deviation on the log-odds scale, and $\sigma_b$ is the random-intercept standard deviation. A prior must also be specified for $\beta_0$ and $\sigma_b$, unless either is held fixed for a particular check. The groups are conditionally independent given their parameters, and units are conditionally independent given their group effects. Other GLMMs follow the same simulation logic, with a different conditional outcome distribution and linear predictor.

A prior predictive simulation draws model quantities in dependency order: first prior parameters, then group effects conditional on the variance component, then outcomes conditional on the effects and fixed effects. If group size or predictor values affect the question, use the planned design values. A check that omits the outcome stage can still reveal implausible latent effects, but it does not show the full implications for observed data.

Draw prior parameters.
Draw $\beta_0$ and $\sigma_b$ from their specified priors, or hold them at stated values for a conditional illustration.

Draw group effects.
For each group, draw $b_j$ from $N(0,\sigma_b^2)$. A shared $\sigma_b$ draw applies to all groups in that simulated data set.

Compute model implications.
Calculate group-specific probabilities, rates, odds ratios, or other interpretable quantities from the linear predictor and link.

Draw outcomes and summarize.
Simulate outcomes using the conditional sampling model, then examine group-level summaries and event patterns relevant to the application.

From Random-Effect Scale to Group Probabilities

The logistic transformation is $\operatorname{logit}^{-1}(x)=1/(1+\exp(-x))$. A random-intercept difference of one log-odds unit multiplies the odds by $\exp(1)$, but it does not create a fixed probability difference. Probability changes are smaller near zero or one than near the middle of the probability scale. That is why checking only the numerical size of $\sigma_b$ is insufficient.

Worked Example: Interpreting a Draw on the Log-Odds Scale

Suppose a conditional prior predictive draw uses $\beta_0=-2$, $\sigma_b=0.70$, and two standardized normal draws $z_1=1.5$ and $z_2=-1.5$. Since $b_j=\sigma_b z_j$, the group effects are $1.05$ and $-1.05$.

For the first group, the linear predictor is $-2+1.05=-0.95$, so its event probability is $1/(1+\exp(0.95))\approx0.2789$. For the second group it is $-2-1.05=-3.05$, giving probability $1/(1+\exp(3.05))\approx0.0452$. With no group effect, the probability would be $1/(1+\exp(2))\approx0.1192$.

The two group odds relative to the zero-effect group are multiplied by $\exp(1.05)\approx2.858$ and $\exp(-1.05)\approx0.3499$, respectively. This is a conditional illustration at a selected $\sigma_b$, not a complete simulation from a prior on $\sigma_b$. It shows how the same scale must be interpreted alongside the intercept and the response scale.

Checking Simulated Prior Implications

Useful summaries should be chosen before simulation and tied to the scientific setting. For a binary outcome, possible summaries include the distribution of group event probabilities, the largest and smallest group probabilities, the spread in those probabilities, and the number of groups with all-zero or all-one outcomes. The last summaries depend on group size: a probability that seems moderate can still produce many all-zero groups when groups are small. Inspect simulated draws individually as well as summaries; a tail pattern can be hidden by a mean.

The following example shows the arithmetic for one hypothetical prior predictive replicate. Its few groups are for illustration, not enough to estimate tail probabilities accurately.

Worked Example: A Small Logistic Prior Predictive Replicate

Suppose $\beta_0=-2$ is fixed for this illustration and $\sigma_b$ has a half-normal prior with scale $0.5$. One possible prior draw is $\sigma_b=0.60$, followed by standardized normal draws $(-1.4,-0.5,0.3,1.2)$ for four groups. The resulting effects are $(-0.84,-0.30,0.18,0.72)$.

The group linear predictors are therefore $(-2.84,-2.30,-1.82,-1.28)$. Applying the inverse-logit gives approximate event probabilities $(0.0552,0.0911,0.1394,0.2176)$. The largest-to-smallest probability ratio is $0.2176/0.0552\approx3.942$. This ratio is a descriptive summary, not an odds ratio.

Now suppose there are 20 units in each group and the simulated binomial counts are $(1,2,3,5)$. The corresponding observed proportions are $(0.05,0.10,0.15,0.25)$. They are compatible in broad scale with the group probabilities, but one replicate does not establish that the prior is well calibrated. Across many prior predictive replicates, one would examine how often counts or probability spreads exceed contextually plausible bounds, and whether those patterns arise frequently or only in extreme tail draws.

For a formal Monte Carlo check of an event $A$, such as “at least one group has an event probability above 0.8,” simulate $M$ complete prior predictive data sets and estimate its prior predictive probability by $\hat{P}(A)=M^{-1}\sum_{m=1}^{M}I(A_m)$, where $I(A_m)$ equals 1 when replicate $m$ meets the condition and 0 otherwise. For example, if 37 of 1,000 simulated data sets meet the condition, the estimated prior predictive probability is $37/1000=0.037$, or 3.7%. Its Monte Carlo standard error is approximately $\sqrt{0.037(1-0.037)/1000}=0.00597$, about 0.60 percentage points. Whether 3.7% is acceptable depends on the context and on whether the event is genuinely implausible.

A Count-Outcome Example

The principle is not limited to binary responses. In a Poisson random-intercept model with log link, a group with effect $b_j$ has expected count $\lambda_j=\exp(\beta_0+b_j)$ per unit of exposure. The random-effect scale is on the log-rate scale, and its implication is often easiest to see by computing the rate and a simple outcome probability.

Worked Example: Zero Counts Under a Poisson Prior Predictive Draw

Consider a hypothetical Poisson GLMM with $\beta_0=\log(2)$, unit exposure, and a simulated $\sigma_b=0.8$. For two standardized effect draws $z_1=1.25$ and $z_2=-1.25$, the group effects are $b_1=1.00$ and $b_2=-1.00$.

The corresponding mean counts are $\lambda_1=2\exp(1)\approx5.437$ and $\lambda_2=2\exp(-1)\approx0.7358$. For a Poisson variable, $P(Y=0\mid\lambda)=\exp(-\lambda)$. Thus the zero-count probabilities are $\exp(-5.437)\approx0.00436$ and $\exp(-0.7358)\approx0.4792$.

A prior predictive simulation can reveal whether the variance prior often creates groups with nearly zero rates alongside groups with very high rates, and whether that pattern produces implausible counts at the planned exposure and group sizes. The check must use the relevant exposure: a rate that implies few counts per unit may still imply many counts over a long exposure.

What to Look For, and What a Check Cannot Do

A prior predictive check is most informative when its summaries have substantive meaning. For example, decide whether knowledge rules out a group probability above a specified threshold, a particular degree of between-group spread, or a certain frequency of groups with no events. Then inspect both the frequency and severity of those patterns across simulations. A single surprising draw is not automatically evidence of a bad prior; a tail event can occur under a reasonable prior. Conversely, plausible average outcomes can conceal a problematic tail.

The simulation should reflect the design to which the prior will be applied: number of groups, group sizes, covariate values, and exposure where relevant. If these are omitted or unrealistic, simulated outcomes may give a misleading impression of the prior’s consequences. Monte Carlo error also matters. For a rare event, increase the number of simulations or report that the estimated frequency is imprecise rather than treating a small simulated count as proof that the event cannot occur.

Prior predictive checks complement, rather than replace, prior-density inspection and elicitation. A density plot describes the prior on its parameter scale; predictive simulation translates it to quantities that may be easier to judge. Neither comparison with observed data nor posterior updating is part of a prior predictive check. Posterior predictive checks use the posterior distribution after observing data and address a different question: whether the fitted model can reproduce relevant features of those data.

This procedure also cannot establish that a model is correct. Implausible simulated patterns may indicate an unsuitable variance prior, but they may instead reflect an unsuitable fixed-effect prior, link, likelihood, or design assumption. Investigate which component generated the pattern before changing the variance prior. A useful diagnostic is to record the simulated $\sigma_b$, group effects, linear predictors, and outcomes together; that trace makes it possible to locate the source of an extreme result.

Common Pitfalls

Checking only $\sigma_b$. A standard deviation has no direct probability-scale meaning in a logistic model. Transform the implied linear predictors to group probabilities or odds ratios.
Simulating effects but not outcomes. Latent heterogeneity may look large or small while the observed implications depend strongly on group size, exposure, and the likelihood. Simulate outcomes when the question concerns observable patterns.
Using one replicate as a verdict. A single prior predictive data set can be unusual by chance. Estimate how often a prespecified pattern occurs over many simulations.
Confusing prior predictive and posterior predictive checks. The prior predictive distribution uses the prior without conditioning on observed outcomes; the posterior predictive distribution uses information from the data.
Calling a pattern implausible without context. Numerical extremity alone is not a criterion. State which feature conflicts with substantive knowledge and why.

Key takeaway: Simulate the variance component, the group effects, and the outcomes in the model’s dependency order. Judge the resulting group-level patterns on a meaningful response scale and against explicit contextual knowledge, not by the variance parameter alone.

Check Your Understanding

Answer each question, then compare your reasoning with the solutions.

In a random-intercept logistic GLMM, what quantities are drawn before simulating a binary outcome?
With $\beta_0=-1$ and $b_j=0.5$, calculate the group event probability to three significant figures.
In 2,000 prior predictive simulations, an event occurs 50 times. Estimate its prior predictive probability.
Why should prior predictive simulations use the intended group sizes or exposure?

Solutions. (1) Draw the parameters from their priors, including $\sigma_b$ and any uncertain fixed effects; then draw $b_j\mid\sigma_b$; then compute the conditional probability and draw the binary outcome. (2) The linear predictor is $-1+0.5=-0.5$, so $p=1/(1+\exp(0.5))\approx0.378$. (3) The Monte Carlo estimate is $50/2000=0.025$, or 2.5%; it estimates the prior predictive probability of the specified event, subject to simulation error. (4) Group size and exposure affect the distribution of observed counts even when the underlying group probability or rate is unchanged, so a check using the wrong design may misrepresent plausible outcome patterns.