Sparse Individual Data and Random-Effects Failures

Why Sparse Individual Data Matter

A population model can contain many records and still provide little information about some random-effects parameters. The key is not only the total number of observations, but also how those observations are distributed across individuals and across time or covariate conditions. If most individuals contribute one concentration, the data may show between-record variation without revealing whether it arose from differences between individuals or from residual error. If individuals have repeated observations but only over a narrow time range, random slopes may be poorly informed.

This distinction matters during model debugging. A variance estimate near zero, a very large standard error, or a covariance step that fails may reflect weak information in the sampling design. Such symptoms do not by themselves prove that the optimizer or covariance calculation is defective. Conversely, sparse data do not excuse numerical errors: the model and computation still need to be checked.

Model and Information in Repeated Observations

Consider a simple random-intercept model for a response \(Y_{ij}\) from individual \(i\) at observation \(j\):

Here \(\mu\) is the population mean, \(\eta_i\) is an individual random effect, and \(\epsilon_{ij}\) is residual error. The standard assumptions for this illustration are that random effects are independent across individuals, residual errors are independent across observations and individuals, and random effects and residual errors are mutually independent. The model assumes a common random-effect variance \(\omega^2\) and residual variance \(\sigma^2\). Both variances are nonnegative.

For an individual with \(m\) observations, the mean of those observations is \(\bar{Y}_i=\mu+\eta_i+\bar{\epsilon}_i\). Its variance is \(\omega^2+\sigma^2/m\). Repeated observations reduce the residual contribution to the individual mean, while the random-intercept contribution remains. This creates information for separating the two sources of variation. With only one observation, that separation is generally unavailable in this model.

Why One Observation Can Leave Variances Confounded

The limitation can be shown directly. If each individual contributes one observation, the model gives \(Y_i=\mu+\eta_i+\epsilon_i\). Since the two independent terms \(\eta_i\) and \(\epsilon_i\) are normally distributed, their sum is normal with variance equal to the sum of their variances:

Thus, the distribution of the observed data identifies the total variance \(\omega^2+\sigma^2\), not its two components separately. For any two nonnegative variance pairs with the same sum, the implied distribution of \(Y_i\) is identical. The likelihood cannot distinguish those pairs under this model. Additional individuals can estimate the total variance more precisely, but do not by themselves resolve this exact confounding.

With two observations per individual, the marginal covariance matrix is instead:

The shared random intercept contributes to the covariance between observations; independent residual errors do not. This repeated-measures pattern can help distinguish \(\omega^2\) from \(\sigma^2\), provided there are enough independent individuals and the model assumptions are reasonable. The information remains finite-sample and can be weak, especially when there are few individuals or when the observations do not provide a useful contrast.

Repeated Measurements Help, but Do Not Guarantee Stability

For \(m\) repeated measurements in the random-intercept model, a simple moment relationship is \(\operatorname{Var}(\bar{Y}_i)=\omega^2+\sigma^2/m\). Within-individual variation also informs \(\sigma^2\), because the shared \(\eta_i\) cancels when measurements from the same individual are differenced. These facts explain why repeated records can separate the two variance components. They do not mean that any small dataset will estimate them precisely: both the number of individuals and the pattern of their repeated observations matter.

Random Slopes Need Informative Within-Individual Spread

Sparse sampling is especially consequential when a model includes random slopes. For example, let response depend linearly on time, with an individual-specific intercept and slope:

Here \(b_{0i}\) and \(b_{1i}\) are individual deviations, with a population covariance matrix describing their variances and covariance. For a given individual, the design matrix has rows \((1,t_{ij})\). Under independent residual errors with variance \(\sigma^2\), the information matrix for that individual's intercept and slope, conditional on the residual model, is \(X_i^\mathsf{T}X_i/\sigma^2\). The determinant of the unscaled design cross-product is:

To see this, the entries of \(X_i^\mathsf{T}X_i\) are \(m\), \(\sum_j t_{ij}\), and \(\sum_j t_{ij}^2\). Its determinant is \(m\sum_j t_{ij}^2-(\sum_j t_{ij})^2\), which equals \(m\sum_j(t_{ij}-\bar{t}_i)^2\). If all the times are identical, this determinant is zero: the individual intercept and slope cannot be separated from that individual's records. A narrow time range makes the determinant small and slope information weak.

Diagnostics and Practical Comparisons

When a random-effects fit is unstable, inspect the design before concluding that a covariance calculation has failed. Useful summaries include the number of observations per individual, the distribution of those counts, the time or covariate range within individuals, and how many individuals contribute information to each part of the design. A large total record count can conceal a large group with only one observation each.

Weak support may appear as a variance estimate at or near zero, a covariance estimate that changes substantially across starts, a correlation near its allowable limit, broad uncertainty, or a likelihood profile that changes little over a range of parameter values. A failed covariance step can also arise when the fitted information matrix is poorly conditioned. The earlier material on covariance eigenstructure explains how to examine that numerical symptom; sparse design is one possible reason the likelihood contains little curvature in a parameter direction.

The main alternatives address different limitations:

• Use a simpler random-effects structure. If the data do not support a random slope or an intercept-slope covariance, a model with fewer random-effects parameters may be more estimable. This changes the model assumptions and should be scientifically justified; it does not prove the omitted variation is absent.
• Improve the sampling design. Additional observations at informative times or covariate values can help distinguish individual effects from residual error. Adding many records at nearly identical conditions may contribute less than adding measurements that broaden within-individual support.
• Use a different residual model only when justified. Changing the residual variance structure can alter how variation is partitioned. It should reflect a plausible observation process, rather than being used solely to force a preferred random-effects estimate.
• Compare population likelihood with individual-level summaries cautiously. Estimating an effect separately for each individual and then treating those estimates as observed random effects can be unreliable when records are sparse. Empirical Bayes estimates are informed by the population distribution and are commonly shrunk toward the population mean; their observed spread is therefore not a direct estimate of the random-effects variance.

No universal observation-per-individual threshold determines whether a variance or covariance is estimable. The answer depends on the model, the number of independent individuals, the sampling schedule, residual error, and the information contributed by covariates and other parts of the design. Even with a structurally identifiable model, a particular dataset can yield imprecise estimates.

Common Mistake

A useful debugging sequence is to summarize individual-level support, connect each random effect to the observations that can inform it, and then examine whether estimates are stable and scientifically plausible. Keep the distinction clear between a parameter that the design cannot separate, one that is technically identifiable but imprecise, and one that is supported but difficult for the numerical procedure to estimate.