Initializing MCMC Chains for Variance Components

Why Initial Values Matter

MCMC initialization is a practical design choice with statistical consequences. A chain started at a single convenient value may take a long time to explore a distant part of the posterior, particularly when a variance component is weakly identified or near zero. Conversely, implausibly extreme initial values can produce numerical failures that say more about the starting point than about the model. With multiple chains, the goal is to use starting states that are different enough to reveal dependence on initialization, but plausible enough to let the sampler operate in the region supported by the model and data.

This tutorial concerns initial states for variance components and associated GLMM parameters. It does not prescribe a warmup schedule or a formal mixing diagnostic. Those are separate questions. Here, the key tasks are to define what makes a starting state valid, select meaningful variation across chains, and check that no construction quietly suppresses variation in the parameters of interest.

For example, in a random-intercept GLMM, a model might use

Here \(Y_{ij}\) is the response for observation \(i\) in group \(j\), \(x_{ij}\) is its predictor vector, \(\beta\) is the fixed-effect coefficient vector, and \(\tau\) is the random-intercept standard deviation. A complete state may need to include \(\beta\), \(\tau\), every \(b_j\), and any extra variables introduced by the chosen sampling scheme. Initializing \(\tau\) but leaving other state variables inconsistent or undefined is not a complete initialization.

Validity, Plausibility, and Dispersion

Three checks should be kept distinct. Validity means the state obeys the model’s support and parameter constraints. For a positive standard deviation, for instance, \(\tau>0\). Plausibility means that the state is not so far from scales supported by the prior and data that the sampler encounters avoidable numerical problems. Dispersion means that separate chains begin at meaningfully different, valid points. A set of starts can satisfy any two of these properties and fail the third.

For a variance component \(\tau^2\), it is often clearer to choose a few initial values on the standard-deviation scale, because their size is easier to interpret in relation to the linear predictor or response. The corresponding variance is obtained by squaring. If the sampler uses an unconstrained parameter \(u=\log(\tau)\), transform each positive proposed value by taking its logarithm. For example, \(\tau=0.5\) corresponds to \(\tau^2=0.25\) and \(u=\log(0.5)\approx-0.6931\). These are three representations of the same starting point, not three distinct starts.

“Dispersed” does not mean selecting the largest and smallest numerically possible values. A near-zero variance may be useful if it is scientifically plausible, but an astronomically large variance can make fitted linear predictors extreme and likelihood calculations unstable. The prior, predictor scaling, response pattern, and substantive scale all inform what counts as plausible. If the posterior plausibly concentrates near zero, include a small positive start rather than setting a positive-only scale parameter to exactly zero.

Correlated Random Effects Need Valid Starts

For random intercepts and slopes, the group-level effects may be modeled jointly as \(b_j\sim N(0,G)\), where \(G\) is a covariance matrix. A valid covariance or correlation matrix must be positive semidefinite. Positive definiteness is required when the model specifically assumes a nonsingular covariance matrix or when a particular computational factorization requires it; it is not the general definition of covariance-matrix validity. A positive-semidefinite matrix may be singular, as with perfectly correlated components.

If the model represents \(G\) using standard deviations \(\tau_0,\tau_1\) and correlation \(\rho\), then

For positive \(\tau_0,\tau_1\), a two-dimensional correlation matrix is valid when \(-1\leq \rho\leq 1\). If the parameterization or sampler requires a nonsingular matrix, the start must instead use \(-1<\rho<1\). Check the actual computational requirement rather than treating positive definiteness as a universal validity condition.

Why Plausible Starts Do Not Remove Initialization Effects

A Markov chain can be started away from the posterior distribution. Let \(\pi\) denote the target posterior distribution, \(P\) the transition rule, and \(\pi_0\) the distribution induced by the chosen initial state. After \(m\) transitions the chain has distribution \(\pi_0P^m\), which need not equal \(\pi\). If the transition rule preserves \(\pi\), then starting with \(\pi_0=\pi\) would keep the chain distributed as \(\pi\); otherwise, the chain must move toward the target through its transitions. This is why a starting point is not itself evidence that posterior exploration has occurred.

The value of using multiple distinct starts is practical: it makes it harder for all chains to inherit the same unexamined choice of variance scale. It does not guarantee that the chains will explore the same posterior region, nor does apparent agreement prove that they have explored all relevant regions. A useful design deliberately varies the variance components while keeping each state plausible and valid.

Initial Values for Augmented Models

Some GLMM samplers augment the state with latent variables. These variables must be initialized consistently with both the observed data and the augmentation scheme. For example, in a binary probit model with latent response \(Z_i\), the observed outcome is \(Y_i=1\) when \(Z_i>0\) and \(Y_i=0\) when \(Z_i<0\). Thus a starting latent value on the wrong side of zero is not a valid state for that augmentation. In other samplers, a particular latent variable may be drawn conditionally as part of the first update; follow the sampler’s specified state definition.

Practical Construction and Failure Modes

A careful initialization can be organized into a short sequence. First list every stored parameter and auxiliary variable. Then choose scales using the model’s prior and the units of the linear predictor. Assign different variance-component values across chains, transform them into the sampler’s coordinates, and check every matrix and latent-variable constraint. Finally, evaluate whether the log posterior or required conditional quantities are finite at each state, if the implementation permits that check.

Several tempting approaches can fail. Initializing every chain at the same posterior mode can make all chains begin in one local region and conceal sensitivity to other variance scales. Starting all variance components at zero is invalid for strictly positive standard-deviation parameters and can be incompatible with some parameterizations. Starting each chain with huge random effects may create extreme logits or near-deterministic fitted probabilities. Finally, assigning independent random effects while claiming to start from a correlated model can be internally inconsistent if the sampler expects a joint state constructed under the current covariance parameters.

A prior draw is one possible source of valid starts, but prior draws can be far from the data-supported region, especially with diffuse priors. A data-informed point, such as a rough fitted value, can be more computationally stable but may place all chains in one narrow region if used identically. A practical compromise is to begin from sensible scales informed by the model and data, then vary the variance and correlation parameters across chains while retaining valid associated effects.

Initialization is only one part of MCMC computation. A valid and dispersed set of starts does not establish that the chain has reached a stationary regime, that it mixes adequately, or that posterior summaries are precise. Those conclusions require examination of the resulting draws with appropriate diagnostics. Here the limitation is important: initialization makes later assessment more informative, but cannot replace it.

Common Mistake

Wrong reasoning: “All chains start at the same posterior mode, so their later agreement shows that the variance components are well explored.” Agreement can result from the shared initialization; it does not establish broad exploration. Correction: Use distinct, plausible starting values for the variance components and associated parameters, then assess the sampled chains using suitable diagnostics. Likewise, do not reject a covariance matrix merely because it is not positive definite: positive semidefiniteness is the general validity condition, while nonsingularity is an additional requirement only when the model or algorithm imposes it.